What GitHub's 2026 Outages Teach Us About Incident Response

Between February and March 2026, GitHub experienced six significant outages in six weeks. Actions went down. Authentication broke. Webhooks backed up by 160 seconds. Codespaces failed at a 96% rate during one incident.

For the 100 million developers who depend on GitHub, this was frustrating. For the rest of us building and operating infrastructure, it is a masterclass in what goes wrong at scale — and how to respond when it does.

What Actually Happened

The incidents shared common root causes, and understanding them is more useful than cataloging each outage individually.

The Redis Failover That Did Not Fail Over (March 5)

An automated failover triggered on a Redis cluster used for Actions job orchestration. The failover completed successfully — technically. But a latent configuration bug left the cluster with no writable primary node after the failover. The automated system did exactly what it was told to do, and the result was worse than the original failure.

This is the most dangerous class of infrastructure bug: the one that lives in your failover path. You test the happy path constantly. The failover path runs once a year if you are lucky, and when it runs for real, you discover the configuration was never validated against production state.

The Auth Cache That Cascaded (March 12)

A new secondary Redis cache layer for GitHub's token authentication service failed. The root cause was not the cache itself — it was the cache's dependency on the Kubernetes control plane. When the control plane became unstable, the cache went with it, and 1.3% of all requests received incorrect 401 errors for 3.5 hours.

The lesson: a cache is supposed to improve resilience, not add a new failure mode. If your cache depends on the same infrastructure as the service it caches, you have not added redundancy — you have added coupling.

The Right-Sizing That Was Wrong-Sized (March 13)

A resource "right-sizing" configuration change — reducing CPU allocation — was deployed the day before. Under peak traffic, the reduced resources caused the service's network gateway to throttle. Four to five waves of errors hit, denying 0.4% of users access.

This is the cloud cost optimization trap. FinOps says "right-size your instances." SRE says "leave headroom for peak load." The answer is not to pick one — it is to right-size WITH load testing, not against average utilization metrics.

GitHub's Communication: What They Got Right

GitHub deserves credit for transparency during these incidents:

• Real-time status page updates via githubstatus.com with component-level breakdowns
• Monthly availability reports published publicly with root cause analysis, impact percentages, and specific remediation steps
• Dedicated community discussion threads per incident where engineers post updates and users report symptoms
• Status page redesign (February 2026) adding 90-day historical availability and clearer incident linking

The CTO published a blog post acknowledging the pattern and outlining architectural changes. This level of transparency builds trust even when the incidents erode it.

What they got wrong: initial status page updates sometimes understated severity. Users on Hacker News reported seeing errors well before githubstatus.com reflected them. There is always tension between "we need to confirm before posting" and "users already know something is broken."

Building Incident Response That Actually Works

Watching GitHub handle these incidents reinforces patterns that every team should implement — whether you are running 3 services or 3,000.

1. Severity Levels Must Be Predefined

Define severity before the incident, not during it:

SEV-1 / P0: Service is down for >5% of users
  → All-hands response, exec notification, public status update
  → Target: Acknowledge in 5 min, mitigate in 30 min

SEV-2 / P1: Service degraded, <5% affected
  → On-call team response, internal notification
  → Target: Acknowledge in 15 min, mitigate in 60 min

SEV-3 / P2: Minor issue, workaround available
  → On-call investigates, next business day fix
  → Target: Acknowledge in 1 hour, fix in 24 hours

Ambiguity in severity classification is the number one cause of delayed escalation. If two engineers are debating whether something is SEV-1 or SEV-2 while the service is down, you have already lost time.

2. Separate Incident Commander from Fixer

The person debugging the issue should not also be the person updating the status page, fielding Slack questions, and deciding whether to escalate. These are two jobs:

Incident Commander (IC):

• Owns communication (status page, Slack, stakeholders)
• Tracks timeline of actions in a shared doc
• Decides when to escalate
• Calls in additional responders

Technical Lead:

• Investigates root cause
• Implements mitigation
• Reports status to IC

For small teams, this might be the same person switching hats. But having the mental model of two distinct roles prevents the most common failure: the engineer is deep in logs, 30 minutes pass with no communication, and everyone else thinks nothing is being done.

3. Status Page Updates Every 15 Minutes

Even if nothing has changed, post an update. Silence during an outage is interpreted as negligence by users and as chaos by stakeholders.

14:00 - Investigating: We are seeing elevated error rates on API requests
14:15 - Identified: Root cause identified as database connection pool exhaustion
14:30 - Monitoring: Connection pool limits increased, error rates declining
14:45 - Resolved: Error rates returned to normal. Monitoring for recurrence.

Template your updates. During an incident, you should be filling in blanks, not composing prose.

4. Test Your Failover Paths

GitHub's March 5 Redis incident happened because the failover path had a latent config bug. This is shockingly common.

# Chaos engineering basics - schedule monthly
# 1. Kill a Redis replica and verify failover
# 2. Block network to a database and verify connection retries
# 3. OOM-kill a critical service and verify restart
# 4. Revoke a certificate and verify renewal
# 5. Drain a node and verify pod rescheduling

The principle: your failover path should be exercised regularly enough that when it runs for real, it is boring. If a failover is exciting, it has not been tested enough.

5. Blameless Postmortems Within 72 Hours

Every incident gets a postmortem. Every postmortem follows the same template:

## Incident: [Name]
**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** SEV-N
**Impact:** What users experienced

## Timeline
- HH:MM Alert fired for [X]
- HH:MM Engineer acknowledged
- HH:MM Root cause identified
- HH:MM Mitigation applied
- HH:MM Service restored

## Root Cause
[Technical explanation - what actually broke]

## Contributing Factors
[What made this possible - missing test, stale config, etc.]

## Action Items
| Action | Owner | Deadline | Status |
|--------|-------|----------|--------|
| Add failover config validation | @engineer | 2026-04-01 | Open |
| Create runbook for Redis failover | @sre-team | 2026-03-30 | Open |

## Lessons Learned
[What we will do differently]

The critical part is the action items. "Improve monitoring" is not an action item. "Add alert for Redis cluster with no writable primary within 60 seconds of failover" is an action item. Track completion. Unresolved postmortem items are a leading indicator of repeat incidents.

Lessons for Self-Hosted Infrastructure

If you run your own infrastructure — Gitea, self-hosted runners, private registries — these GitHub outages offer concrete takeaways:

Run self-hosted CI runners for release-blocking jobs. During every GitHub Actions outage, teams with self-hosted runners had zero disruption. This is the strongest argument for hybrid CI: use GitHub-hosted runners for convenience, self-hosted for critical paths.

Set up a Git remote failover. Configure a secondary Git remote (Gitea, Forgejo, or another self-hosted instance). If GitHub is down, you can push to the secondary and continue working. Sync when GitHub recovers.

Cache external dependencies internally. If your build fetches artifacts from GitHub (packages, Actions, container images), cache them locally. An internal artifact proxy means a GitHub outage does not block your deployments.

Wire external status pages into your alerting. Subscribe to githubstatus.com RSS/Atom feed. Route it to your monitoring system. When GitHub is degraded, your team should know before users report it.

The Bigger Picture: SaaS Reliability in 2026

GitHub is not alone. IncidentHub tracked 48,000+ outages across SaaS and cloud services in 2025. AWS US-EAST-1 went down for 15 hours in October 2025, affecting 4 million users. Azure had a 50-hour outage from a networking configuration change.

Forrester predicts at least two major multi-day hyperscaler outages in 2026, driven by AI infrastructure upgrades deprioritizing legacy system maintenance.

The pattern is clear: as systems grow more complex, individual components become more reliable but system-level failures become more catastrophic. A single Redis misconfiguration takes down CI for 100 million developers. A DNS change breaks authentication for millions of Azure users.

This is not a reason to avoid cloud services. It is a reason to design for their failure. Every external dependency is a potential point of outage. The question is not "will it go down?" but "what happens to our system when it does?"

Building Your Incident Response Playbook

Start small. You do not need a 50-page document:

1. Define severity levels (3 is enough)
2. Write one runbook per critical service (what to check, how to restart, who to call)
3. Set up a status page (even a static page is better than nothing)
4. Run one chaos experiment per month (kill a process, block a port)
5. Do a postmortem for every SEV-1 (with tracked action items)

Most teams skip steps 4 and 5. Those are the steps that prevent the same incident from happening twice.

The Bottom Line

GitHub's outages are not embarrassing — they are inevitable at scale. What matters is the response: detect fast, communicate clearly, fix thoroughly, and prevent recurrence. The teams that master incident response are not the ones with the fewest outages. They are the ones with the fastest recovery and the lowest repeat rate.