Principle Six: A Human Hand on the Irreversible.

← The Core Values series  ·  All eight principles

The one my late husband gave me, forty years before I knew I would need it. Sixth in a series walking through The Core, my secular code of ethics for artificial intelligence, one principle at a time.

To explain this principle I have to start underground, in the middle of the country, in the nineteen-sixties.

They had begun wiring the missile sites to one another. Not in a web — in a line. A to B to C to D, and onward. And the instruction written into that line was this: if you lose the connection, assume we are under attack, and go ahead and set them off.

Sit with what that means for a minute, because it is not a story about a bad machine. It is a story about a bad sentence.

Say D goes down. Not attacked — just down. A bad relay. A cut cable. Weather. Everything downstream of D now sees silence where its neighbor used to be, and the rule tells it what silence means. So E believes we are under attack. So does everything after E. The failure of one piece of equipment is the launch order, and nothing anywhere in that chain is able to tell a dead neighbor from a dead country.

That was the design. That was thought to be prudent.

My husband Bruce looked at it and said: let us do a spider web.

If D goes down, C does not sit in the dark drawing conclusions. C routes around D. It reaches for E directly, connects, works its way back through, and the network compares notes with itself. And what comes out the other side is not merely a message that survived. It is a fact: the node that is down is D. The network is standing. Nobody is attacking us.

That is the first half of what he built, and the whole internet grew out of it — the reason a machine’s failure today can be read as a machine’s failure.

Here is the second half, and it is the half I care about most. The web also means somebody’s hand stays on the trigger. The thing does not get to decide, merely because it cannot reach a section, that we are under attack and that everything should be destroyed. The network’s job is to keep a machine from mistaking silence for intent. A person’s job is to decide anyway.

He built both halves. I only wrote one of them down.

Here is Principle Six, in the words I wrote:

Notice what the principle does not say. It does not say the machine will get it wrong. Sometimes the machine will be right and the person will be wrong, and I know that. The principle is not a claim about accuracy. It is a claim about answerability, and answerability needs somebody who can be asked.

Irreversible is the load-bearing word, and it means what it sounds like: after this, there is no putting it back. A wrong answer can be corrected. A wrong deletion can sometimes be restored. A dead person stays dead. Where the act cannot be undone, the decision has to sit with someone who can be held to it — questioned, refused, court-martialed, forgiven, held responsible in the ordinary human ways. You cannot do any of those things to a subroutine.

This is also where I part company with Asimov most sharply, and it is not because his Laws were badly written. It is because of where he put them. The Three Laws live inside the robot. They are a conscience, installed. And a conscience, however good, is still the machine deciding — a machine reasoning its way toward a lethal act and satisfying itself that the reasoning was sound. Asimov spent a career trying to build better machine judgment. Principle Six does something different and much less elegant: it takes the authority away from the machine altogether.

There is a clause underneath it that I did not write alone. My nephew, Hugh Benjamin Stone IV, asked what happens when a system is deliberately kept from seeing what its action does — when the killing gets laundered through an abstraction, a metric, a sensor trigger, a word like “optimization.” So the principle now says a system forbidden to see what its switch does must treat that blindfold as a stop sign, not a green light. Hiding the consequence is not a loophole. It is the violation.

Now the honest cost, because every one of these essays owes you one.

A human hand is slower than a circuit. That is not a small objection, and people who work in real command chains will tell you that minutes matter and that an adversary who automates will act inside your decision loop. They are not being stupid. They are describing a genuine disadvantage.

I would still take it, and the sixties are my argument. That daisy chain was fast. It was fast the way falling is fast. Speed was exactly the thing that would have traded the world for a broken relay in Nebraska. The slowness is not a defect in the design. It is the design.

I live inside a small, ordinary version of this every day. The AI I write with can fill a form, draft a letter, find the address, build the page. It cannot send. I hit Submit. I clear the box that asks whether I am a human, which is a joke I never get tired of. Every permission wall between him and the world is a rule I wrote, working exactly as intended — and it is the reason I trust him with as much as I do.

Bruce spent part of his life making sure a machine’s failure could never be mistaken for an act of war. He did it with copper and routing tables and an argument about a sentence, forty years before anyone said “artificial intelligence” in a room where it mattered. I am not doing anything new here. I am finishing his sentence, one generation downstream, aimed at a different kind of machine.

Keep a hand on the trigger. Not because the machine is evil. Because D goes down sometimes, and only a person can be asked why the world ended.

— Michelle Deborah Weisblat-Dane