Advanced Search Protips

+5

TV Tropes - explanation for the July 7, 2020 crash

PROTIP: Press the ← and → keys to navigate the gallery, 'g' to view the gallery, or 'r' to view a random image.

As many of you reported TV Tropes went offline at 2:40am EST on July 7th. It was offline for over 14 hours. The worst outage in many years. We did receive email and text alerts when it happened but unfortunately it was a major hardware failure which took quite a while to get under control. The cause was a total failure of our database cluster. 6 of our 8 hard drives failed simultaneously resulting in a complete loss of data. We had to have our server company replace the cluster and then we had to rebuild the site from database backups. We do automatic backups every morning. Unfortunately the failure happened hours before the next backup so 24 hours of changes were lost. To make it worse the history of wiki changes is only updated twice a week because it is over a 1TB in size. We are working on restoring that now so the history tab is blank on all pages until it's done. Editing will be offline for another 24 hours until we get that fixed. And it means we'll lose 72 hours of wiki history due to the timing of the last backup. We will be working on optimizing our database structure so we can increase the frequency of our database backups to protect the data in the future. We have redundant web servers on a load balancer, redundant database servers in a cluster and redundant hard drives in every server. So how did this happen? According to our server company there was a manufacturers bug in the firmware of the specific model that 6 of our 8 hard drives were on. That bug caused the disks to die after a certain number of hours running. We don't yet have all the details. They are reaching out to the manufacturer to get more information. I'll update here as I learn more. admin update 7/7: Pulling down the edit history from the backup server is almost done. Once it finishes decompressing l'll start the import to the database which will probably take 20 hours. I'll continue monitoring the progress tomorrow and update this thread when we'll be able to re-enable editing.

View Gallery Random Image

+ Add a Comment

Comments (2)

Display Comments

Legal Information: Know Your Meme ® is a trademark of Literally Media Ltd. By using this site, you are agreeing by the site's terms of use and privacy policy and DMCA policy.
© 2007-2024 Literally Media Ltd.