Horizen - daemon go rougue

We got one Horizen node going rougue and that created a 42 GB database folder. Noticing that, our server protection stopped all the daemons with this code (despite unlikely for nodes this behavior is the same of a malware).

Our plataform does isolates daemons from the same coin in the same execution block, so daemons from different coins may not produce noise among those and the cost per node is reduced while compared to plataforms that run one daemon per vps.

The drawback of this approach is the current issue: if the coin has one daemon demanding all the resources, there is a chance of it putting down the "siblings" in the same server. This is an acceptable risk, since issues will only occur if the project has a bug or something similar.

Thus, in this case, we will NOT offer SLA compesation since this issue was cleary caused by a project coding (no provider shall accept to run daemons that write 41GB of data in less than 1h). We will preserve the debug log for further requests of users.


Leave a comment