Glitch Unmasks Role of Networks Making Websites Faster

The culprit was a glitch at a content delivery network service, a behind-the-scenes cloud-based system used by websites to open data-heavy web pages faster, run videos or stream games for customers without interruptions. The systems are also referred to as edge cloud networks.

Content delivery networks are critical linchpins in how the internet works, said Glenn O’Donnell, vice president and research director at information-technology research firm Forrester Research Inc. “When one of them suffers a mistake like this, things go very, very badly,” he said.

The system that caused Tuesday’s blackout, operated by Fastly Inc., encountered a configuration error, a company spokesperson said. Joshua Bixby, Fastly’s chief executive, said the outage wasn’t related to a cyberattack.

Configuration errors are almost always caused by humans, Mr. O’Donnell said. “Someone went into a server or network device, made a change and fat-fingered the command,” he speculated. Human error is also the most common cause of breakdowns in content delivery networks, he said.

Such broad outages are rare, said Ghassan Abdo, a research vice president covering world-wide telecom services at International Data Corp.

“This is quite an unusual scenario,” he said, because most of these services are smartly built and able to, for instance, balance traffic loads between servers. Many customers avoid problems by using more than one vendor, in what he calls a “multi-CDN strategy,” he said.

The systems, which were introduced in the 1990s, are designed to make web traffic more efficient and faster for end users. Typically, when a person opens a webpage by clicking on a link, the request is transmitted back to a website operator’s server, data center or cloud, where the necessary data is stored.

In a content delivery network, that data is then held in a proxy server nearest to the user who opened the page—also known as a cache point. When subsequent users click on the same link, the request goes to the closest cache point, rather than back to the operator.

As more people click on the link, the speed of the page doesn’t slow down, since it is being served by thousands of cache points across a distributed network of these proxy servers. That avoids a traffic jam as numerous requests try to connect to a single database.

Fastly operates roughly 150 clusters of cache points and proxy servers globally, known as points of presence. Each cluster contains multiple servers.

Akamai Technologies Inc., one of the industry’s largest players, operates about 300,000 servers in more than 130 countries.

The companies typically rent space for custom-built servers from third-party data-center operators or internet service providers.

“One of the reasons we’re talking so much about this is that there, frankly, haven’t been that many problems or major blips in what has been a very stressful time for the internet with the overhang of the pandemic,” said Andy Champagne, senior vice president and chief technology officer at Akamai Labs, which is Akamai’s internal research and development function.

Revenue for content delivery network services is expected to reach $12.4 billion this year, up from $10.8 billion in 2020, according to enterprise information-technology research and consulting firm Gartner Inc.

Customers for such services are businesses that depend on high-performance applications that involve large amounts of data, such as media outlets and videogame companies. They typically buy services directly from content delivery network providers.

Mike Dorosh, a senior research director at Gartner, said Tuesday’s outage is significant in that a single provider caused a major blackout across the internet.

Mr. Dorosh said content delivery networks are essential for any company dependent on the internet to drive a significant portion of revenue. The drawbacks of the systems are the same as any other IT service, he said.

“It has been said that slow is the new down, but down is still death,” Mr. Dorosh said. “Your IT is only as good as the single points of failure it has and your strategy for mitigation of that risk.”

For now, he added, the benefits of content delivery networks, including better online performance and improved services, far outweigh the risks.

Write to Angus Loten at [email protected] and John McCormick at [email protected]