Facebook Is a Doomsday Machine

The doomsday machine was never supposed to exist. It was meant to be a thought experiment that went like this: Imagine a device built with the sole purpose of destroying all human life. Now suppose that machine is buried deep underground, but connected to a computer, which is in turn hooked up to sensors in cities and towns across the United States.

The sensors are designed to sniff out signs of the impending apocalypse—not to prevent the end of the world, but to complete it. If radiation levels suggest nuclear explosions in, say, three American cities simultaneously, the sensors notify the Doomsday Machine, which is programmed to detonate several nuclear warheads in response. At that point, there is no going back. The fission chain reaction that produces an atomic explosion is initiated enough times over to extinguish all life on Earth. There is a terrible flash of light, a great booming sound, then a sustained roar. We have a word for the scale of destruction that the Doomsday Machine would unleash: megadeath.

Nobody is pining for megadeath. But megadeath is not the only thing that makes the Doomsday Machine petrifying. The real terror is in its autonomy, this idea that it would be programmed to detect a series of environmental inputs, then to act, without human interference. “There is no chance of human intervention, control, and final decision,” wrote the military strategist Herman Kahn in his 1960 book, On Thermonuclear War, which laid out the hypothetical for a Doomsday Machine. The concept was to render nuclear war unwinnable, and therefore unthinkable.

Kahn concluded that automating the extinction of all life on Earth would be immoral. Even an infinitesimal risk of error is too great to justify the Doomsday Machine’s existence. “And even if we give up the computer and make the Doomsday Machine reliably controllable by decision makers,” Kahn wrote, “it is still not controllable enough.” No machine should be that powerful by itself—but no one person should be either.

....

Of the many things humans are consistently terrible at doing, seeing the future is somewhere near the top of the list. This flaw became a preoccupation among Megadeath Intellectuals such as Herman Kahn and his fellow economists, mathematicians, and former military officers at the Rand Corporation in the 1960s.

Kahn and his colleagues helped invent modern futurism, which was born of the existential dread that the bomb ushered in, and hardened by the understanding that most innovation is horizontal in nature—a copy of what already exists, rather than wholly new. Real invention is extraordinarily rare, and far more disruptive.

The logician and philosopher Olaf Helmer-Hirschberg, who overlapped with Kahn at Rand and would later co-found the Institute for the Future, arrived in California after having fled the Nazis, an experience that gave his desire to peer into the future a particular kind of urgency. He argued that the acceleration of technological change had established the need for a new epistemological approach to fields such as engineering, medicine, the social sciences, and so on. “No longer does it take generations for a new pattern of living conditions to evolve,” he wrote, “but we are going through several major adjustments in our lives, and our children will have to adopt continual adaptation as a way of life.” In 1965, he wrote a book called Social Technology that aimed to create a scientific methodology for predicting the future.

In those same years, Kahn was dreaming up his own hypothetical machine to provide a philosophical framework for the new threats humanity faced. He called it the Doomsday Machine, and also the Doomsday-in-a-Hurry Machine, and also the Homicide Pact Machine. Stanley Kubrick famously borrowed the concept for the 1964 film Dr. Strangelove, the cinematic apotheosis of the fatalism that came with living on hair-trigger alert for nuclear annihilation.

Today’s fatalism about the brokenness of the internet feels similar. We’re still in the infancy of this century’s triple digital revolution of the internet, smartphones, and the social web, and we find ourselves in a dangerous and unstable informational environment, powerless to resist forces of manipulation and exploitation that we know are exerted on us but remain mostly invisible. The Doomsday Machine offers a lesson: We should not accept this current arrangement. No single machine should be able to control so many people.