Why Complex Systems Are Always One Step from Catastrophe
How Complex Systems Fail
Complex systems—from healthcare to power grids—are inherently hazardous and heavily defended, yet they run in a constant state of degradation. Catastrophe is never the result of a single failure but of multiple small faults aligning. This essay explains why post-accident 'root cause' analysis is fundamentally flawed, how hindsight biases our judgment of operators, and why safety is an emergent property that people create moment by moment. It argues that failure-free operation actually requires intimate experience with failure.
Because overt failure requires multiple faults, there is no isolated 'cause' of an accident.
- tptacek
I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.
But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.
All practitioner actions are gambles. I should have that inked somewhere.
- anonymars
"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."
This very much resembles Admiral Cloudberg's write-up of the National Airport collision:
"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."
https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
- jedberg
> Failure free operations require experience with failure.
This is why we created Chaos Engineering. By constantly forcing failure, it made us always create systems in defense of that failure, and gave us great data on where the tipping point is for different systems within a particular failure mode.
- ChrisMarshallNY
Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section:
> All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature.
(Emphasis mine)
I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.
- feyman_r
I may have shared this before on a different submission: John Gall’s books are really good on this topic: General Systemantics [https://en.wikipedia.org/wiki/Systemantics]