If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All

The most fundamental fact about current AIs is that they are grown

The inner workings of batteries and rocket engines are well understood, governed by known physics recorded in careful textbooks. AIs, on the other hand, are grown, and no one understands their inner workings. There are fewer equations to constrain one's thinking... and so, many opportunities to think about high-minded ideals like truth-seeking instead. If you know the history of science, this kind of talk is recognizable as the stage of folk theory, the stage where lots of different people are inventing lots of different theories that appeal to them personally, the way people talk before science has really gotten started on something. They're the words of an alchemist who's decided that some complicated philosophical scheme will let them transmute lead into gold.

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

Engineers failed at crafting AI, but eventually succeeded in growing it.

We make a mistake the first time

Sometimes the engineer learns better at the cost of only time and money. Sometimes the engineer kills only themselves or only consenting volunteers; and Science writes down what happened

Intelligences don’t need to be given a lot of power and resources to become dangerous. Humans started out naked in the savannah

Any intelligence capable of comprehending biochemistry at the deepest level is capable of building its own self-replicating factories to serve its own purposes.

But a surprising number of elected officials have told us that they can see the danger themselves, but cannot say so for fear of repercussions.

All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they’ve been doing.

Attempting to solve a problem like that, with the lives of everyone on Earth at stake, would be an insane and stupid gamble that NOBODY SHOULD BE ALLOWED TO TRY.

They did all that because world leaders knew that, in the event of a nuclear war, both they and the people of their countries would have a bad day.

An artificial superintelligence is like a nuclear reactor

There are even deeper reasons to expect advanced AIs to behave like they have wants.

MIRI’s research, and shifted the institute’s focus to conveying one single point, the warning at the core of this book: If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

We ultimately predict AIs that will not hate us

If you launch a rocket and load the whole human species on board

The critical number governing a nuclear reactor is the neutron multiplication factor

AIs grown in this way do things that their growers did not intend. In 2023

The survivors of the blind cheerful optimists turn into cynical pessimistic veterans;

With this book, we hope to inspire individuals and countries to rise to the occasion.

Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed

wanting is an effective strategy for doing.