Horizen Critical Event during update
Despite been notified by Github about a new Horizen Version available, we did not get aware of the fact that it was a mandatory version until it was to late.
The reason for that is that on the tracker page, in the upper right top, Horizen team always made public announcements; and they decided to do not announce new versions in there anymore.
While we do understand that improvements are the most important thing in a crypto project, reducing and changing the update notifications of mandatory upgrade will lead to events like this.
The worse thing is that each Horizen version has a "self-destruct" date, which is the same of the next version (yeah Horizen Foundation can potentially share the next update date in the date of release of the newest version).
The problem is that all nodes that did shutdown were restarted by our platform over and over again up to 3 times and this promoted a flood of our internal network file system, and that crashed many nodes, corrupting config files.
Once we rebuild the config files, some nodes start to load with the new version, but near to the end of processing wallet transactions (96%), it just shutdown, and again, we had restarts.
Everything was solved, we have less than 80 nodes that will be fixed with a wallet replacement (ez Masternodes will lose its ZEN), but it took 4 days to get completed.
Again, even despite the fact that all would be fine if the notification was performed as it always was, we are not blaming Horizen Team, because the team work is the only reason for the increasing of the protocol and on the other hand, no one can make and omelet without breaking some eggs.