GitHub Outage
On August 17, GitHub experienced an outage that lasted 7 hours and 47 minutes. It disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot, affecting developers and organizations around the world.
[…]
Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages.
[…]
As part of the reliability commitments we made earlier this year, we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks. We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
This is the wrong way to think about this because there’s no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not “a component didn’t have enough capacity (because of auto scaling failures)”, but rather “this complex system collapses (rather than degrade gracefully) when demand exceeds capacity”.
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections. Most other large internet services have since copied them, afaik.
Previously:
6 Comments RSS · Twitter · Mastodon
"When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected."
Let me state up front that I know nothing about the guy who wrote that quote or what his background is but that bit sounds so naive.
How does GitHub determine what the "lowest priority" is when all of GitHub Actions is failing? Which enterprise customer do they say too bad so sad to first?
I run the DevOps group for my employer. We have hundreds of Actions runners that we self host and maintain, hundreds of developers, and thousands of repositories. We pay GitHub a hell of a lot of money. Now multiply that by, well, a lot. Then multiply it with all of the AI stuff going on right now.
I just don't see how GitHub comes up with the list of rules this guy is saying they should.
I thought the whole point of Git was to not require a server. When I last worked at a software company and our Git server was down (we self-hosted), me and my team would always just push/pull from each other directly. It really is that easy.
GitHub marketing has done an amazing job of convincing companies that (1) they need a server, and (2) while you're at it, offload a bunch of other mission-critical functionality to GitHub, too.
Surely, almost all of this growth is from LLMs. Here's the state of the internet in the mid-2020's: everybody has to pay the price for "AI", even those of us who don't use it and don't want it. Just like how webpages got slow because CDNs need to decide if they think we're human.
The Butlerian Jihad is coming.
@Tim Ah, but that value-add is what sucks them in, isn't it? Issue tracker, stars, pull requests, public hosting and pages, CICD, container registry … it's not just that they've taken a decentralised VCS and made it centralised, it's that nobody can imagine a world without the highly centralised (and most easily centralised) features and APIs that, even when there *are* self-hostable options, won't be beaten for polish and discovery and network effects. See also email being killed by Gmail et al.
And, no, I'm not happy about it either. We've got to stop depending on these people.
> How does GitHub determine what the "lowest priority" is
That's easy. It's people who don't pay.
> I thought the whole point of Git was to not require a server
GitHub being down is an issue because it stops the whole infrastructure around people's use of Git, not because it stops Git itself, as Sebby says. I don't need to merge other people's changes into my branch, but I do need to open PRs and cut releases and deploy to test systems and run CI/CD, and so on.
Sebby: "...Issue tracker..."
I never understood why the GitHub Issue Tracker (which was their first non-Git feature, IIRC) wasn't a Git repo -- like GitHub Wikis, for example. That always seemed like nothing more than a blatant lock-in attempt.
"See also email being killed by Gmail et al."
Interestingly, I've recently seen even non-techies looking at Gmail alternatives. I think people do realize they're being locked in, and aren't entirely happy. They've simply done the calculation and decided it's too much work to migrate. But companies can't frustrate users indefinitely and expect this calculation to always have the same answer.
Plume: "I don't need to merge other people's changes into my branch, but I do need to open PRs and cut releases and deploy to test systems and run CI/CD, and so on."
Yeah, at that point you're not using git, you're using GitHub. When your own internal process is "Whatever GitHub's workflow supports", then you're at their mercy, both in terms of features and availability.
Big corporations (like governments) require multisourcing. I find it strange that any mature organization would accept single-sourcing for a critical component, especially from a young startup. But then, I suppose that's why I'm just some dork commenting on the internet, and not a successful CTO.
@Tim Indeed, though git isn't always the answer, of course, but it's odd that more wasn't made of it for building a more distributed issue tracker or whatever. Perhaps what we need is some sort of ActivityPub for forges, which could federate somehow …
Of course, the old ways still work fine. git-email to a mailing list may not be sexy, but it's battle-tested tech, no forge required.
