Cloudflare Outage: A Key Analysis of the Incident That Hit the Internet

A recent six-hour outage of one of the key content delivery networks (CDN) affected thousands of websites and services worldwide. Although the incident caused significant complications, the subsequent detailed analysis provides valuable insights for the entire technology community.

Cloudflare Outage: A Key Analysis of the Incident That Hit the Internet

A recent incident once again highlighted the critical role that content delivery networks, such as Cloudflare, play in the internet's infrastructure. The six-hour outage caused thousands of websites and services to become unavailable, including many globally significant platforms. This event underscored the deep reliance of many online services on a single provider, a dependency also evident during previous outages of other major cloud services.

Cloudflare quickly published a detailed incident analysis that clarified the causes and progression of the entire event. The root cause lay in the incorrect propagation of a configuration file to the bot management module. This file caused the module to crash, which subsequently took Cloudflare's proxy layer offline. This layer is crucial for protecting customer resources, minimizing network traffic, and efficiently caching static data within the CDN.

The entire incident was triggered by a seemingly innocent change in database permissions. Cloudflare aimed to improve system security and reliability by transitioning from shared system accounts to individual user accounts. Unfortunately, a side effect of this change was that a query for obtaining metadata for the bot management module inadvertently started returning a much larger volume of data than expected – specifically, double the original number of functions. The bot management module has an internal limit of 200 functions (for which it allocates memory), and if this number is exceeded, the system enters a panic state. The error occurred in a part of the code where the data-returning function was not expected to fail, leading to an uncontrolled system crash when attempting to "unwrap" an unexpected error value. Thus, individual edge nodes progressively failed as the faulty configuration file spread through the system.

Identifying the true cause of the outage took unusually long – approximately two and a half hours. Initial misleading indications led the team to suspect a coordinated attack. At a critical moment, the company's status page also went down, which reinforced this hypothesis. Engineers thus wasted time searching for an external attack that did not actually exist, as the status page outage was merely a coincidence. Once the true cause – faulty configuration files – was identified, it took another hour to stop the propagation of the incorrect configuration and deploy a corrected version. The total incident resolution time, including cleanup and functionality verification, extended to approximately six hours.

It was highly impressive how quickly Cloudflare managed to publish a detailed postmortem analysis – within 24 hours of resolving the incident. The company's CEO personally participated in resolving the outage and immediately after it subsided, began drafting the first version of the report. The text was then shared internally with the team, who added details and answered remaining questions, enabling rapid completion and publication. This approach demonstrates agility and transparency, which are more typical of startups than publicly traded corporations.

Several important lessons can be drawn from this incident for software engineers and system operators:

Explicit Error Logging: It is crucial to implement error logging at the moment they occur and to have alerts configured for their increase. Although this requires additional work, it significantly helps reduce the time needed to identify and mitigate problems.

Risks of Global Database Changes: Changes in database permissions, especially those affecting a wide range of queries, carry significant and often unpredictable risks. While Cloudflare aimed to improve security and reliability by eliminating global system accounts, which is a correct step, the impact of such changes is difficult to fully test.

Dominik Medal

Need a hand with your website or app?

Get in touch and I will reply within 24 hours with a proposal for the next step.

Concurrent Problems Can Mislead: Situations where two independent problems occur simultaneously (e.g., a system outage and a concurrent status page crash) can lead the engineering team to incorrect conclusions and unnecessary time loss when searching for the root cause. In the heated phase of an incident, it's easy to assume a connection.

CDN Dependence and Mitigation Options: CDNs are an indispensable part of the internet but also represent a critical dependency. Preparing for their outage is complex and costly. Options include redirecting traffic to proprietary "origin" servers (which requires massive backend scalability) or utilizing a backup CDN (which means additional costs for a constantly ready, albeit inactive, partner and complex traffic switching). Both strategies require significant investment and careful planning.

Dominik Medal

Let's talk about your project

Tell me what you need and together we will work out the best way forward.

Stop scrolling, call me

+420 735 505 585