At some point, every tech company faces that dreaded Slack notification: "Is anyone else seeing this?" followed by the cascade of alerts, the war room formation, and the all-hands-on-deck scramble to restore service. It's not a matter of if your systems will fail, it's when.
But here's the thing: what happens after an outage often determines whether you'll face the same issue six months later or emerge stronger than before. That post-incident analysis, the post-mortem, is where the real value lies.
As a team here at Overmind, we help platform teams prevent large outages before they happen, but to do this we need to learn from the past. We're particularly interested in how public post-mortems create accountability and showcase a company's technical maturity and we use these to help build and test our solution. The most respected teams in tech don't hide their failures. They analyse them openly, learn from them systematically, and share those learnings generously.
The Post-Mortem Mindset
Focus on Systems, Not Scapegoats
Everything starts here. Before templates, before processes, before tools—the foundation of effective post-mortems lies in your engineering culture. Without the right mindset, even the most well-intentioned post-mortem procedures will devolve into blame games or checkbox exercises.
When things go wrong, human instinct often drives us to find someone to blame. Therefore an engineering culture that recognises complex system outages rarely stem from a single, already understood, issue. Roblox demonstrates this beautifully in their post-mortems. Following their 73-hour outage in October 2021 (one of the longest in recent tech history), their analysis focused entirely on the system conditions that allowed the failure to occur, not on who might have made a mistake.What stands out in their report is the explicit emphasis on team dynamics:
"At Roblox we believe in civility and respect... It's easy to be civil and respectful when things are going well, but the real test is how we treat one another when things get difficult... We supported one another, and we worked together as one team around the clock until the service was healthy."
Learning Over Liability
Post-mortems become transformative when they're treated as living documents. CircleCI has great examples of this approach by publishing detailed reliability reports every month for over two years led by their CTO. In one update, they specifically mention how a January incident prompted them to "increase the coverage of synthetic tests to better differentiate between system faults and natural fluctuations in customer errors." This level of specificity shows a genuine commitment to improvement.
What's particularly effective about their approach is consistency, they don't just document major outages but track smaller incidents too, recognising that today's minor glitch could be tomorrow's major failure if systemic issues aren't addressed. This level of transparency requires organisational maturity. It's saying: "We care more about improving than appearing infallible."
Anatomy of an Exceptional Post-Mortem
Telling the Full Story
Great post-mortems read like detective stories, not incident reports. They reconstruct what happened with enough detail that readers can follow the logic and spot potential gaps.
Reddit's Pi Day outage post-mortem from 2023 provides a master class in clear incident explanation. Their report documents how a failed Kubernetes cluster upgrade triggered a cascading failure. They didn't just state that "Kubernetes failed" they explained exactly how the renaming of a node label (node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane) during the upgrade caused the existing Calico network configuration to fail, preventing servers in the cluster from communicating.
This level of specificity helps other teams learn from their experience. The report also covers the investigation steps, including the realisation of lost metrics and DNS issues, attempts to resolve by deleting OPA webhook configurations, and their conservative approach to bringing traffic back online.
But timelines alone aren't enough. The best post-mortems also include wider context: Was this during a deployment? Was the team working with new technology? What made this particular day different from others when the same conditions didn't cause problems?
Quantifying What Actually Happened
The scale of an incident isn’t always clear from status page status alone. That's why impact assessment is critical, what actually broke from the user perspective?
HubSpot's post-mortems are leaders in this. In their September report, they clearly articulated which specific services were affected by their traffic routing layer failure involving Envoy and Kubernetes. Similarly, their May report detailed exactly how a Linux kernel bug affecting TCP memory management impacted customer facing systems.
This level of transparency helps serve multiple purposes; prioritise future prevention work, context for stakeholders, and demonstrates respect for customers by acknowledging the real impact of technical failures.
When quantifying impact, be specific. "Degraded performance" means different things to different people "average response times increased from 200ms to 1.5s affecting 46% of API calls" leaves no room for interpretation.
From the "What" to the "Why"
Effective post-mortems dig deeper, using techniques like the "5 Whys" to move past immediate triggers to underlying causes.
Box's approach to root cause analysis displays this depth. In their February incident report, they didn't just state that authentication failed. They specifically identified "a gap in the procedure related to the management of application authentication keys" that "led to a deletion of keysets that were mistakenly believed to be no longer necessary." This precision allows them to address not just the technical issue but the procedural gap that allowed it to happen.
Remember: the goal isn't to find a single root cause (complex systems rarely have just one), but to identify the combination of factors that allowed the failure to occur and propagate.
Analysis to Action
Assigning Ownership for Improvements
A post-mortem without action items is just a historical document. The real value comes from the improvements that follow.
Vercel's approach stands out here. In their Next.js middleware bypass post mortem, they include a specific "Next Steps" section with actionable items like
"Simplify how issues get reported: We have consolidated security@vercel.com and responsible.disclosure@vercel.com to only use GitHub's private vulnerability reporting for Next.js. This will help us triage incoming reports more effectively."
These aren't vague promises to "do better" but concrete changes with clear outcomes. This specificity transforms abstract learnings into concrete improvements and provides a reference point for future post-mortems. "Didn't we say we were going to fix this last time?”.
Ensuring Lessons Stick
Even with the best intentions, action items can fall through the cracks amid competing priorities. Effective organisations build mechanisms to ensure post-mortem learnings actually lead to changes.
Xero's approach, as highlighted in an engineering blog, maintaining a "postmortem tracking report" that shows all post-mortems in progress and outstanding actions. In monthly review meetings, their SRE team selects several in-progress post-mortems to report to a wider audience, bringing issues to the attention of teams across the business and encouraging the adoption of better practices.
This approache recognises a fundamental truth: without systematic follow-through, even the most insightful post-mortem becomes a document that's written, read, and forgotten.
Building Trust Through Transparency
How Post-Mortems Strengthen Team Cohesion
When done right, post-mortems don't just fix technical issues, they strengthen the bonds between teams. By creating a shared understanding of what happened and why, they reduce finger-pointing and build empathy across organisational boundaries.
Canva demonstrates this by highlighting their collaborative approach to problem-solving. In their API Gateway outage post-mortem, they mention working closely with Cloudflare to gain an in depth understanding of the complex system interactions involved. They also note their history of writing incident reports internally since 2017, showing a long term commitment to organisational learning.
This internal trust is particularly valuable during future incidents. Teams that have honestly analysed past failures together are more likely to collaborate effectively when things go wrong again.
Honest Communication
Counterintuitively, admitting failure often builds more trust than claiming perfection. Companies that share what went wrong and how they're addressing it demonstrate both technical competence and organisational integrity.
Loom exemplifies this approach, publishing their initial post-mortem analysis on March 8, 2023, just one day after an incident occurred. What's particularly impressive is their commitment to iterative updates, they continued to refine their post-mortem with additional details as their investigation progressed. This demonstrates a commitment to transparency.
Canva takes transparency a step further, with concrete improvement plans in their post-mortems. Following their API Gateway outage, they listed specific action items like building a "Runbook for traffic blocking and restoration," increasing the baseline number of API Gateway tasks, and adding "page load completion events as a page asset canary release indicator." These specifics give customers confidence that their experience will keep getting better.
Summary
A good post-mortem is the best way to learn after an outage. If you want to prevent these outages from happening in the first place, you need a pre-mortem. Overmind discovers potential outages before changes are deployed, including those caused by complex hidden dependencies like the examples above.
.gif)
