I would like to talk in more detail about the incident that occurred on the game world 3.3.5 x1 early in the morning, October 24th.
The night before, October 23 (and earlier on October 21, but that only concerned the web), maintenance was carried out in our DC. work (
https://selectel.live/incidents/322882), during which half of the equipment was disconnected from the power supply for some time (I would like to add that this usually does not happen because even during work or an accident, there are backup power sources, but here apparently it was no different).
After they were completed, all our resources started up in normal mode on their own when the servers were turned on and the OS loaded, and since I was away in another country from the beginning of the month, there was no way to go in and check everything, especially since everything was ok according to all statuses.
But everything turned out to be a little different, in the case of the server where the x1 and fan base is running, something happened to the hard drive and a number of tables (we haven’t figured it out yet), because of which the game worlds first began to crash very often, and then were completely unavailable because writing new data in the table was impossible
By a happy coincidence, I was at least on the road at that moment, but with access to the Internet, I was able to quickly, as a temporary measure, deploy everything on another server (thanks to the 10 Gbit network), but we didn’t get off easy, because some of the data was inaccessible and the kernel did not understand what was happening, algorithms were launched to clear data (garbage) in a number of tables (when there are no main tables), most were later able to be restored piece by piece, but some no, it was mainly the guilds’ storage facilities that were affected, but otherwise there was a slight setback for a couple of hours.
We have now launched a number of measures to compensate for this incident, which are described here:
https://forum.wowcircle.com/showthread.php?t=1357495