Artificial Intelligence: A Guide for Thinking Humans

We should be afraid. Not of intelligent machines. But of machines making decisions that they do not have the intelligence to make. I am far more afraid of machine stupidity than of machine intelligence. Machine stupidity creates a tail risk. Machines can make many many good decisions and then one day fail spectacularly on a tail event that did not appear in their training data. This is the difference between specific and general intelligence.

as the AI researcher Pedro Domingos so memorably put it, “People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world.

Above all, the take-home message from this book is that we humans tend to overestimate AI advances and underestimate the complexity of our own intelligence. Today’s

A pile of narrow intelligences will never add up to a general intelligence. General intelligence isn’t about the number of abilities, but about the integration between those abilities.

Hofstadter... fears that AI might show us that the human qualities we most value are disappointingly simple to mechanize.

In any ranking of near-term worries about AI, superintelligence should be far down the list. In fact, the opposite of superintelligence is the real problem. Throughout this book, I’ve described how even the most accomplished AI systems are brittle; that is, they make errors when their input varies too much from the examples on which they’ve been trained. It’s often hard to predict in what circumstances an AI system’s brittleness will come to light. In transcribing speech, translating between languages, describing the content of photos, driving in a crowded city—if robust performance is critical, then humans are still needed in the loop. I think the most worrisome aspect of AI systems in the short term is that we will give them too much autonomy without being fully aware of their limitations and vulnerabilities. We tend to anthropomorphize AI systems: we impute human qualities to them and end up overestimating the extent to which these systems can actually be fully trusted.

Hofstadter ended his talk with a direct reference to the very Google engineers in that room, all listening intently: “I find it very scary, very troubling, very sad, and I find it terrible, horrifying, bizarre, baffling, bewildering, that people are rushing ahead blindly and deliriously in creating these things.

New companies have sprung up to offer labeling data as a service; Mighty AI, for example, offers “the labeled data you need to train your computer vision models” and promises “known, verified, and trusted annotators who specialize in autonomous driving data.”11 The

Mucha gente cree que hasta que los sistemas de IA no tengan el mismo sentido común que los humanos, no podremos confiar en que sean totalmente autónomos en situaciones complejas del mundo real.

Yet Kurzweil is best known not for his inventions but for his futurist prognostications, most notably the idea of the Singularity: “a future period during which the pace of technological change will be so rapid, its impact so deep, that human life will be irreversibly transformed.”19 Kurzweil uses the term singularity in the sense of “a unique event with … singular implications”; in particular, “an event capable of rupturing the fabric of human history.”20 For Kurzweil, this singular event is the point in time when AI exceeds human intelligence.

Kurzweil is not only a director of engineering at Google but also a cofounder (with his fellow futurist entrepreneur Peter Diamandis) of Singularity University (SU), a “trans-humanist” think tank, start-up incubator, and sometime summer camp for the tech elite. SU’s published mission is “to educate, inspire, and empower leaders to apply exponential

so many people were shocked and upset when, in 1997, IBM’s Deep Blue chess-playing system defeated the world chess champion Garry Kasparov. This event so stunned Kasparov that he accused the IBM team of cheating; he assumed that for the machine to play so well, it must have received help from human experts.2 (In a nice bit of irony, during the 2006 World Chess Championship matches the tables were turned, with one player accusing the other of cheating by receiving help from a computer chess program.3) Our collective human angst over Deep Blue quickly receded. We accepted that chess could yield to brute-force machinery; playing chess well, we allowed, didn’t require general intelligence after all. This seems to be a common response when computers surpass humans on a particular task; we conclude that the task doesn’t actually require intelligence. As John McCarthy lamented, ‘As soon as it works, no one calls it AI any more.’4

Define your terms … or we shall never understand one another.”10 This admonition from the eighteenth-century philosopher Voltaire is a challenge for anyone talking about artificial intelligence, because its central notion—intelligence—remains so ill-defined. Marvin Minsky himself coined the phrase “suitcase word”11 for terms like intelligence and its many cousins, such as thinking, cognition, consciousness, and emotion. Each is packed like a suitcase with a jumble of different meanings. Artificial intelligence inherits this packing problem, sporting different meanings in different contexts.

Hofstadter’s terror was in response to something entirely different. It was not about AI becoming too smart, too invasive, too malicious, or even too useful. Instead, he was terrified that intelligence, creativity, emotions, and maybe even consciousness itself would be too easy to produce—that what he valued most in humanity would end up being nothing more than a “bag of tricks,” that a superficial set of brute-force algorithms could explain the human spirit.

Kurzweil cites numerous quotations from prominent people in history who completely underestimated the progress and impact of technology. Here are a few examples. IBM’s chairman, Thomas J. Watson, in 1943: ‘I think there is a world market for maybe five computers.’ Digital Equipment Corporation’s co-founder Ken Olsen in 1977: ‘There’s no reason for individuals to have a computer in their home.’ Bill Gates in 1981: ‘640,000 bytes of memory ought to be enough for anybody.

Reflecting on real-life machine morality, the mathematician Norbert Wiener noted as long ago as 1960 that “we had better be quite sure that the purpose put into the machine is the purpose which we really desire.