Server upgrade and dadfailed on child bug lead to some reward loss

EZ is proud of cutting a better deal this week with our partner Nexeon (we got some "relationship points" after 1 year of partnership), so we are moving ALL the nodes to bigger bad ass machines, from Dual E5-2650v2 128GB SSD to Dual E5-2650v2 256GB 3D SSD with Raid 0, the fastest machine that they do offer for premium clients. The idea behind that is to be future-proof very often we noticed that projects release version that make the daemon temporarily very demanding over RAM and bandwith (ie. Smartcash, Horizen, Crown and others) and that make a huge mess.

The obvious thing is changes caused by projects may take clients away from us, but we want to keep serving our clients, hence giving some credits and saying "see you later" is not an acceptable option.

Unfortunately moving daemons between servers is too trick, despite possible to change the allocation of running nodes without it getting it expired (big thanks to IP Pooling ability), moving hundreds of nodes around is never an easy task, and things can and do go wrong very fast. We had an IPv6 connectivity issue, after moving the nodes, in the new server, Ubuntu did not allowed us to attach some IPv6 because it "believed" that those specific IP still stood assigned to other server (dadfailed on child interface error) and it was a know bug.

We are sorry about the outcome: all the daemons that rely on IPv6 in the range 2602:ffc8:1:2::3:100/64 to 2602:ffc8:1:2::3:154/64 had no internet access for more than 1 hour (we took some time to notice the bug and solve by doing a power cycle). Some users will endure reward loss. Despite its been due a confirmed Ubuntu bug, EZ Masternodes choose to perform the server change, thus we decide to compensate users: SLA is in place.

To get the compensation, just send an email to with the title SLA compensation - dadfailed on Chicago 04, in the body include the order number of the compromised node and you estimation of value lost in USD. 

1 comment

  • 1


Leave a comment