What is Wrong with My Facebook Account

What Is Wrong With My Facebook Account - Early today Facebook was down or unreachable for much of you for roughly 2.5 hrs. This is the worst blackout we've had in over 4 years, and also we intended to first off apologize for it. We additionally wished to give far more technical information on what occurred as well as share one large lesson learned.

What Is Wrong With My Facebook Account

The crucial flaw that caused this outage to be so serious was an unfavorable handling of an error condition. An automated system for verifying arrangement values wound up causing a lot more damages than it dealt with.

The intent of the automatic system is to check for configuration worths that are invalid in the cache and also change them with upgraded worths from the relentless store. This works well for a short-term problem with the cache, however it does not work when the relentless store is void.

Today we made an adjustment to the consistent copy of a setup value that was interpreted as invalid. This suggested that every client saw the void value as well as attempted to repair it. Because the fix entails making a question to a cluster of data sources, that cluster was rapidly bewildered by thousands of hundreds of queries a second.

To make matters worse, whenever a customer got an error attempting to quiz among the databases it translated it as an invalid value, as well as erased the matching cache secret. This meant that even after the initial issue had actually been repaired, the stream of queries proceeded. As long as the databases failed to service several of the demands, they were triggering much more demands to themselves. We had actually gotten in a feedback loop that didn't allow the databases to recuperate.

The means to stop the comments cycle was fairly uncomfortable - we had to stop all traffic to this data source cluster, which indicated turning off the website. Once the data sources had actually recuperated and the source had actually been taken care of, we slowly enabled even more people back onto the site.

This obtained the website back up and also running today, and also for now we have actually switched off the system that tries to fix arrangement values. We're exploring new designs for this setup system complying with design patterns of various other systems at Facebook that deal even more beautifully with feedback loops as well as transient spikes.

We say sorry once more for the website interruption, and also we want you to know that we take the performance and also integrity of Facebook extremely seriously.

Tweet