Author Topic: Forum Apocalypse 2020  (Read 470 times)

0 Members and 1 Guest are viewing this topic.

Online The Latinist

  • Cyber Greasemonkey
  • Technical Administrator
  • Reef Tank Owner
  • *****
  • Posts: 8420
  • Fanatically anti-bigot
    • View Profile
Forum Apocalypse 2020
« on: February 09, 2020, 07:39:55 pm »
Dear fellow members,

We would like to thank you for your patience over the last week as Belgarath worked to restore our forum. I wish that we could give you a detailed explanation of the cause of the outage, but we are not entirely sure of that ourselves. Here is what we know, along with a look at what's ahead.

About two weeks ago we started having issues with the forum server, and it's hard to know which are related and which are not. The first symptom was a daily crash of the server's antivirus software, which appeared to be caused by a corrupt antivirus database, but which was not remedied even by a reinstall of the ClamAV. Then sometime last weekend the MySQL process became unstable and crashed. As it did so, it began to corrupt the database tables. The first symptom we noticed was a complete failure of forum threads to load with an error saying that the table containing the thread data had been dropped from the table. At first, other sections of the website (PM's, the member database, moderation and administration) continued to function, but over a period of about 15 minutes each section failed as the database tables responsible for their functioning became corrupted. Ultimately, we were unable to load any portion of the site.

After our previous Forum Apocalypse, Belgarath implemented a thorough daily backup scheme. He maintains backups of the entire website and database which make it possible to restore the site completely with the loss of at most a day or two of data. I contacted Belgarath as soon as I was aware of the issue.  Belgarath was able to confirm that he had a backup and spent some time investigating the cause of the issue. Ultimately, it became clear that his most recent backup already contained some corrupted data and in particular did not contain the Member Table, which seems to have been the first area affected. To minimize data loss, Belgarath made the decision to attempt to restore the Member Table from an older backup while taking other data from the most recent backup.  This meant that restoring the server would be a more involved and time-consuming process which he would not be able to complete before this weekend.

Belgarath has worked tirelessly this weekend on restoring the Forum, and has done a fabulous job. Ultimately, he had to create a new installation of SMF and import the data in a very time-consuming process, but the result is that we lost less than a day of post data, but about a month of registration data. He believes that everything except the member database has been restored as of 2 a.m. February 1.  I personally thank Belgarath for his hard work.

Looking ahead, there will be a few lingering issues over the next few weeks as we continue cleanup, in particular:

1) Out of an abundance of caution, new member registrations have been disabled temporarily.

2) Posts made by new members who registered after about January 1 will to be attributed to 'Guest' as those registrations have been lost.

3) A small number of posts in the latest backup were corrupted and have been removed. Moving to an older database backup would have resulted in even more posts lost.

4) Many mods have not yet been reinstalled, including popular features like 'Like Posts' and media embedding.  We will work to restore these features over the coming weeks, but we want to do so deliberately and with lots of testing.

5) There will likely be minor issues with the themes and smileys. I've created a thread for reporting such issues. I am already aware, for instance, of the issue with the width of the reply box. I will probably be responsible for cleaning up most of these theme issues, but as my schedule is very busy at the moment I will have to do triage and fix the most pressing issues first.

Thanks again to Belgarath for his hard work and to all of our members for their patience and understanding.

The Latinist
« Last Edit: February 10, 2020, 11:04:16 pm by The Latinist »
Large data of text  or Mediumtext or Longtext