Mind: Drilling for Certainty

The Gulf oil well blowout inspired this ...

If there is one thing we’ve learned, it is that humans are not great at measuring and responding to risk when placed in situations too complicated to understand.

In the first place, people have trouble imagining how small failings can combine to lead to catastrophic disasters. At the Three Mile Island nuclear facility, a series of small systems happened to fail at the same time. It was the interplay between these seemingly minor events that led to an unanticipated systemic crash.

Second, people have a tendency to get acclimated to risk. As the physicist Richard Feynman wrote in a report on the Challenger disaster, as years went by, NASA officials got used to living with small failures. If faulty O rings didn’t produce a catastrophe last time, they probably won’t this time, they figured.

...

Third, people have a tendency to place elaborate faith in backup systems and safety devices. More pedestrians die in crosswalks than when jay-walking. That’s because they have a false sense of security in crosswalks and are less likely to look both ways.

On the Deepwater Horizon oil rig, a Transocean official apparently tried to close off a safety debate by reminding everybody the blowout preventer would save them if something went wrong. The illusion of the safety system encouraged the crew to behave in more reckless ways. As Malcolm Gladwell put it in a 1996 New Yorker essay, “Human beings have a seemingly fundamental tendency to compensate for lower risks in one area by taking greater risks in another.”

Fourth, people have a tendency to match complicated technical systems with complicated governing structures. The command structure on the Deepwater Horizon seems to have been completely muddled, with officials from BP, Transocean and Halliburton hopelessly tangled in confusing lines of authority and blurred definitions of who was ultimately responsible for what.

Fifth, people tend to spread good news and hide bad news. Everybody wants to be part of a project that comes in under budget and nobody wants to be responsible for the reverse. For decades, a steady stream of oil leaked out of a drill off the Guadalupe Dunes in California. A culture of silence settled upon all concerned, from front-line workers who didn’t want to lose their jobs to executives who didn’t want to hurt profits.

Finally, people in the same field begin to think alike, whether they are in oversight roles or not. The oil industry’s capture of the Minerals Management Service is actually misleading because the agency was so appalling and corrupt. Cognitive capture is more common and harder to detect.

So it seems important, in the months ahead, to not only focus on mechanical ways to make drilling safer, but also more broadly on helping people deal with potentially catastrophic complexity. There must be ways to improve the choice architecture — to help people guard against risk creep, false security, groupthink, the good-news bias and all the rest.

For more, see Drilling for Certainty by David Brooks, May 27, 2010.