Black Box Thinking: Why Some People Never Learn from Their Mistakes - But Some Do

Learn from the mistakes of others. You can’t live long enough to make them all yourself.

Studies have shown that we are often so worried about failure that we create vague goals, so that nobody can point the finger when we don’t achieve them. We come up with face-saving excuses, even before we have attempted anything. We cover up mistakes, not only to protect ourselves from others, but to protect us from ourselves. Experiments have demonstrated that we all have a sophisticated ability to delete failures from memory, like editors cutting gaffes from a film reel—as we’ll see. Far from learning from mistakes, we edit them out of the official autobiographies we all keep in our own heads.

Creativity is, in many respects, a response.

It is partly because we are so willing to blame others for their mistakes that we are so keen to conceal our own. We

Failure is rich in learning opportunities for a simple reason: in many of its guises, it represents a violation of expectation.6 It is showing us that the world is in some sense different from the way we imagined it to be.

The only way to be sure is to go out and test your ideas and programmes, and to realise that you will often be wrong. But that is not a bad thing. It leads to progress. This

Michael Jordan, the basketball great, is a case in point. In a famous Nike commercial, he said: ‘I’ve missed more than nine thousand shots. I’ve lost almost three hundred games. Twenty-six times I’ve been trusted to take the game-winning shot and missed.’ For many the ad was perplexing. Why boast about your mistakes? But to Jordan it made perfect sense. ‘Mental toughness and heart are a lot stronger than some of the physical advantages you might have,’ he said. ‘I’ve always said that and I’ve always believed that.’ James

Everything we know in aviation, every rule in the rule book, every procedure we have, we know because someone somewhere died . . . We have purchased at great cost, lessons literally bought with blood that we have to preserve as institutional knowledge and pass on to succeeding generations. We cannot have the moral failure of forgetting these lessons and have to relearn them.

The reason is not difficult to see: if we drop out when we hit problems, progress is scuppered, no matter how talented we are. If we interpret difficulties as indictments of who we are, rather than as pathways to progress, we will run a mile from failure. Grit, then, is strongly related to the Growth Mindset; it is about the way we conceptualise success and failure.

These failures are inevitable because the world is complex and we will never fully understand its subtleties. The model, as social scientists often remind us, is not the system. Failure is thus a signpost. It reveals a feature of our world we hadn’t grasped fully and offers vital clues about how to update our models, strategies, and behaviors. From this perspective, the question often asked in the aftermath of an adverse event, namely, “Can we afford the time to investigate failure?,” seems the wrong way around. The real question is, “Can we afford not to?

When a culture is unfair and opaque, it creates multiple perverse incentives. When a culture is fair and transparent, on the other hand, it bolsters the adaptive process.

trying to increase discipline and accountability in the absence of a just culture has precisely the opposite effect. It destroys morale, increases defensiveness and drives vital information deep underground. It

Psychologists often make a distinction between mistakes where we already know the right answer and mistakes where we don’t. A medication error, for example, is a mistake of the former kind: the nurse knew she should have administered Medicine A but inadvertently administered Medicine B, perhaps because of confusing labeling combined with pressure of time. But sometimes mistakes are consciously made as part of a process of discovery. Drug companies test lots of different combinations of chemicals to see which have efficacy and which don’t. Nobody knows in advance which will work and which won’t, but this is precisely why they test extensively, and fail often. It is integral to progress.

Success is not just dependent on before-the-event reasoning, it is also about after-the-trigger adaptation.

Creativity is so delicate a flower that praise tends to make it bloom, while discouragement often nips it in the bud.’3

The Greek period inspired the greatest flowering of knowledge in human history, producing the forefathers of the entire Western intellectual tradition, including Socrates, Plato, Aristotle, Pythagoras and Euclid. It changed the world in ways both subtle and profound.

Bacon identified in relation to the natural sciences: the mismatch between the complexity of the world and our capacity to understand it.

Scientific ideas should succeed or fail according to rational argument and evidence. It is about data rather than dogma.

United Airlines 173 was a traumatic incident, but it was also a great leap forward,” the aviation safety expert Shawn Pruchnicki says. “It is still regarded as a watershed, the moment when we grasped the fact that ‘human errors’ often emerge from poorly designed systems. It changed the way the industry thinks.” Ten people died on United Airlines 173, but the learning opportunity saved many thousands more.

Cognitive dissonance doesn’t leave a paper trail. There are no documents that can be pointed to when we reframe inconvenient truths. There is no violence perpetrated by the state or anyone else. It is a process of self-deception. And this can have devastating effects, not least on those who were the subject of chapter 4: the wrongly convicted.

This, then, is what we might call “black box thinking.”* For organizations beyond aviation, it is not about creating a literal black box; rather, it is about the willingness and tenacity to investigate the lessons that often exist when we fail, but which we rarely exploit. It is about creating systems and cultures that enable organizations to learn from errors, rather than being threatened by them.

Remember that for a man like Tony Blair, this was the biggest decision of his political life. He was not just a voter who supported the war, he was a prime minister who had gambled his career on the conflict, committing troops on the ground, of whom 179 would lose their lives. His political reputation, to a large extent, hinged on the decision. If anyone would be motivated to defend it, he would. So, let us explore the contortions. On 24 September 2002, before the conflict, Blair made a speech to the House of Commons about Saddam Hussein’s weapons of mass destruction: ‘His WMD programme is active, detailed and growing,’ he said. ‘Saddam has continued to produce them, . . . he has existing and active military plans for the use of chemical and biological weapons, which could be activated within 45 minutes . . .

Memory, it turns out, is not as reliable as we think. We do not encode high-definition movies of our experiences and then access them at will. Rather, memory is a system dispersed throughout the brain and is subject to all sorts of biases. Memories are suggestible. We often assemble fragments of entirely different experiences and weave them together into what seems like a coherent whole. With each recollection, we engage in editing.*

Twelve months later, when the Iraq survey group, Blair’s inspectors of choice, couldn’t find the weapons either, he changed tack again. Speaking to the House of Commons Liaison Committee, he said: ‘I have to accept we haven’t found them and we may never find them, we don’t know what has happened to them . . . They could have been removed, they could have been hidden, they could have been destroyed.’ The evidential dance was now at full tilt. The lack of evidence for WMD in Iraq, according to Blair, was no longer because troops had not had enough time to find them, or because of the inadequacy of the inspectors: rather, it was because the Iraqi troops had spirited them out of existence.

Learn from the mistakes of others. You can’t live long enough to make them all yourself.’ And

I have spoken to dozens of pilots, investigators and regulators about the November Oscar incident and, although perspectives vary, there is a broad consensus that it was a mistake to pin the blame on Stewart. It was wrong of British Airways to censure him and for the lawyers at the CAA to put him on trial. Why? Because if pilots anticipate being blamed unfairly, they will not make the reports on their own mistakes and near-misses, thus suppressing the precious information

But even if we practice diligently, we will still endure real-world failure from time to time. And it is often in these circumstances, when failure is most threatening to our ego, that we need to learn most of all. Practice is not a substitute for learning from real-world failure; it is complementary to it. They are, in many ways, two sides of the same coin.

These witnesses were not necessarily lying. They were not making it up. But then neither was Neil Tyson when he talked about Bush’s stars speech. When the witnesses said they remembered seeing the suspect at the scene of the crime, they were telling the truth. They did remember seeing him there, but they didn’t actually see him there. These are two quite different things.

This is not to say that eyewitness testimony is worthless; quite the reverse. In certain circumstances it is invaluable in order to secure convictions. Rather, it is to say that memories should be coaxed out of witnesses with sensitivity to the biases that might otherwise contaminate the evidence. The tragedy is that the techniques used by police, until recently, had little of this sophistication.

The practice of “drive-bys,” for example, has been used and abused for decades: this is where an eyewitness is taken by police to see a suspect on the street, or at their place of work. Given that the witness knows that the police have suspicions about the person—why else would they be going there?—the technique is dangerously suggestive.

Learning from failure has the status of a cliché. But it turns out that, for reasons both prosaic and profound, a failure to learn from mistakes has been one of the single greatest obstacles to human progress

True ignorance is not the absence of knowledge, but the refusal to acquire it.

Much of the literature on creativity focuses on how to trigger these moments of innovative synthesis; how to drive the problem phase toward its resolution. And it turns out that epiphanies often happen when we are in one of two types of environment. The first is when we are switching off: having a shower, going for a walk, sipping a cold beer, daydreaming. When we are too focused, when we are thinking too literally, we can’t spot the obscure associations that are so important to creativity. We have to take a step back for the “associative state” to emerge. As the poet Julia Cameron put it: “I learned to get out of the way and let that creative force work through me.”8 The other type of environment where creative moments often happen, as we have seen, is when we are being sparked by the dissent of others. When Kevin Dunbar, a psychologist at McGill University, went to look at how scientific breakthroughs actually happen, for example (he took cameras into four molecular biology labs and recorded pretty much everything that took place), he assumed that it would involve scientists beavering away in isolated contemplation. In fact, the breakthroughs happened at lab meetings, where groups of researchers would gather around a desk to talk through their work. Why here? Because they were forced to respond to challenges and critiques from their fellow researchers. They were jarred into seeing new associations.

Most closed loops exist because people deny failure or try to spin it. With pseudosciences the problem is more structural. They have been designed, wittingly or otherwise, to make failure impossible. That is why, to their adherents, they are so mesmerizing. They are compatible with everything that happens. But that also means they cannot learn from anything.

It was only when professionals believed that reports on errors and near misses would be treated as learning opportunities rather than a pretext to blame that this crucial information started to flow. Managers were initially worried that reducing the penalties for error would lead to an increase in the number of errors. In fact, the opposite happened. Insurance claims fell by a dramatic 74 percent. Similar results have been found elsewhere. Claims and lawsuits made against the University of Michigan Health System, for example, dropped from 262 in August 2001 to 83 following the introduction of an open disclosure policy in 2007. The number of lawsuits against the University of Illinois Medical Center fell by half in two years after creating a system of open reporting.

Proper investigation achieves two things: it reveals a crucial learning opportunity, which means that the systemic problem can be fixed, leading to meaningful evolution. But it has a cultural consequence too: professionals will feel empowered to be open about honest mistakes, along with other vital information, because they know that they will not be unfairly penalized—thus driving evolution still further. In short, we have to engage with the complexity of the world if we are to learn from it; we have to resist the hardwired tendency to blame instantly, and look deeper into the factors surrounding error if we are going to figure out what really happened and thus create a culture based upon openness and honesty rather than defensiveness and back-covering.

The marginal gains approach is not just about mechanistic iteration. You need judgment and creativity to determine how to find solutions to what the data is telling you, but those judgments, in turn, are tested as part of the next optimization loop. Creativity not guided by a feedback mechanism is little more than white noise. Success is a complex interplay between creativity and measurement, the two operating together, the two sides of the optimization loop.

The secret to modern F1 is not really to do with big ticket items; it is about hundreds of thousands of small items, optimized to the nth degree. People think that things like engines are based upon high-level strategic decisions, but they are not. What is an engine except many iterations of small components? You start with a sensible design, but it is the iterative process that guides you to the best solution. Success is about creating the most effective optimization loop.

Marginal gains is not about making small changes and hoping they fly. Rather, it is about breaking down a big problem into small parts in order to rigorously establish what works and what doesn’t. Ultimately the approach emerges from a basic property of empirical evidence: to find out if something is working, you must isolate its effect. Controlled experimentation is inherently “marginal” in character.

As Duflo puts it: “It is possible to make significant progress against the biggest problem in the world through the accumulation of a set of small steps, each well thought out, carefully tested, and judiciously implemented.

In his seminal book Antifragile, Nassim Nicholas Taleb shows how the linear model is wrong (or, at best, misleading) in everything from cybernetics, to derivatives, to medicine, to the jet engine. In each case history reveals that these innovations emerged as a consequence of a similar process utilized by the biologists at Unilever, and became encoded in heuristics (rules of thumb) and practical know-how. The problems were often too complex to solve theoretically, or via a blueprint, or in the seminar room. They were solved by failing, learning, and failing again.

In a study in Scotland, members of the public were adamant that they could remember a nurse removing a skin sample from their little finger. But this never happened. A week earlier these volunteers had been asked by researchers to imagine a nurse removing the sample. But somehow, on recollection, it had morphed into a real event. They were four times as likely to recall it as real compared with those who had not been asked to imagine it.

When we are confronted with evidence that challenges our deeply held beliefs we are more likely to reframe the evidence than we are to alter our beliefs. We simply invent new reasons, new justifications, new explanations. Sometimes we ignore the evidence altogether.

All airplanes must carry two black boxes, one of which records instructions sent to all on-board electronic systems. The other is a cockpit voice recorder, enabling investigators to get into the minds of the pilots in the moments leading up to an accident. Instead of concealing failure, or skirting around it, aviation has a system where failure is data rich. In the event of an accident, investigators, who are independent of the airlines, the pilots’ union, and the regulators, are given full rein to explore the wreckage and to interrogate all other evidence. Mistakes are not stigmatized, but regarded as learning opportunities. The interested parties are given every reason to cooperate, since the evidence compiled by the accident investigation branch is inadmissible in court proceedings. This increases the likelihood of full disclosure. In the aftermath of the investigation the report is made available to everyone. Airlines have a legal responsibility to implement the recommendations. Every pilot in the world has free access to the data. This practice enables everyone—rather than just a single crew, or a single airline, or a single nation—to learn from the mistake. This turbocharges the power of learning. As Eleanor Roosevelt put it: “Learn from the mistakes of others. You can’t live long enough to make them all yourself.” And it is not just accidents that drive learning; so, too, do “small” errors. When pilots experience a near miss with another aircraft, or have been flying at the wrong altitude, they file a report. Providing that it is submitted within ten days, pilots enjoy immunity. Many planes are also fitted with data systems that automatically send reports when parameters have been exceeded. Once again, these reports are de-identified by the time they proceed through the report sequence.*

the most powerful engine of progress is to be found deep within the culture of the industry. It is an attitude that is easy to state, but whose wider application could revolutionize our attitude to progress: instead of denying failure, or spinning it, aviation learns from failure.

In 2013 a study published in the Journal of Patient Safety8 put the number of premature deaths associated with preventable harm at more than 400,000 per year. (Categories of avoidable harm include misdiagnosis, dispensing the wrong drugs, injuring the patient during surgery, operating on the wrong part of the body, improper transfusions, falls, burns, pressure ulcers, and postoperative complications.) Testifying to a Senate hearing in the summer of 2014, Peter J. Pronovost, MD, professor at the Johns Hopkins University School of Medicine and one of the most respected clinicians in the world, pointed out that this is the equivalent of two jumbo jets falling out of the sky every twenty-four hours. “What these numbers say is that every day, a 747, two of them are crashing. Every two months, 9/11 is occurring,” he said. “We would not tolerate that degree of preventable harm in any other forum.”9 These figures place preventable medical error in hospitals as the third biggest killer in the United States—behind only heart disease and cancer.