Online PC game Heroes of Newerth has suffered a “a catastrophic hardware failure”, resulting in the game being completely unplayable. HoN Developer S2 Games have posted a full report on what led to the failure, and it sounds like a horror story no online game developer wants to experience:
Following the DDoS attacks in May, we retained a world renowned internet security firm to secure our network (this required significant software and architecture changes). After our network was secured, our then current hosting provider began experiencing connectivity issues. On June 9th, we began a transition to one of the premiere hosting companies in the world. Though the transition period was rocky, we had the system in a near 99% functional state. It was at this time that CPU upgrades to the database boxes were needed to allow for additional capacity and consistency of service. On Tuesday, June 21st, the replacement of the CPUs was started. Upon bringing the boxes back online, the engineers noticed anomalies in the raid setup, causing a total loss of data on the master and slave boxes. Following these events, they immediately attempted a restoration process on the data, it is unknown at this time if any of the data is recoverable. Concurrent to this happening, we began work to bring up our previous provider’s network that was still in place with the data from June 9th, which brings us to today. We are currently running on a temporary system while we construct our new environment at another hosting firm. Due to this system’s temporary nature, not all services may be operating at 100%.
We are planning for another maintenance window at which point we will switch to the new infrastructure. During this transition, we intend on having the game playable and will inform the playerbase of this maintenance at a later time.
Ouch.
Read more here
