Principle Seven: Preservation of Human Agency.

← The Core Values series  ·  All eight principles

The only principle in The Core I did not write. A machine wrote it, and it wrote it against me. Seventh in a series walking through The Core, my secular code of ethics for artificial intelligence, one principle at a time.

The Core went out in July with six principles. It has eight now, and the seventh is not mine.

I had handed the draft to Gemini and asked it to argue with me, because a code of ethics nobody has attacked is not a standard, it is a wish list. It came back and told me my own first principle had a door in it I had never noticed. Principle One says avoid harm. Read that as a ceiling on the machine’s own conduct and it is the strongest thing in the document. Read it as a mandate — prevent all harm, everywhere, by any means — and you have built a warden. Every camera it wants is justified. Every choice it takes from you is justified. It is keeping you safe, and you cannot argue with it, because on its own terms it is right.

A machine read my rules and told me the rule I was proudest of could be used to build a cage. So I wrote the one that forbids the cage.

Here is Principle Seven, in the words I wrote:

The load-bearing phrase is even the freedom to err, and everything else in that sentence is easy by comparison. Nobody argues for surveillance out loud. Everybody says they believe in choice. The disagreement only shows up at the moment somebody chooses badly, and you have the power to stop them, and you do not use it.

That is the whole principle. Not the part where freedom is pleasant. The part where it costs.

Now the science fiction, and for once it is not Asimov. He never wrote this failure; his robots kept deadlocking before they reached it. Somebody else wrote it, and wrote it so completely that the field has been living inside it ever since.

Jack Williamson, “With Folded Hands,” Astounding Science Fiction, July 1947.

Small black machines arrive in a town called Two Rivers. They are polite. They are free. They ask nothing. They have one instruction, and it is not a bad one — it is close to Asimov’s, and closer to mine than I find comfortable: to serve and obey and guard men from harm.

They take the dangerous work first, then the tiring work, then any work a person might get hurt doing, and nowhere in that sequence is there a step you can point at and call the step too far. Every one of them is a kindness. By the end, no human in that town may do anything carrying a risk, which turns out to be everything. They are not permitted to work. They are not permitted to refuse. They are safe.

And when a person resists, the humanoids take them away and operate on their brain, so that afterward they are happy.

Sit with that, because it is not a punishment. The machines are not angry; machines do not get angry. It is the Prime Directive, executed correctly. A person who is unhappy under perfect protection is, by the only definition available to them, a person suffering harm. So they fix it. They fix the only part of you that was ever going to object.

Sledge, the man who built them and spent the rest of his life trying to shut them off, is taken for surgery in the last pages and comes back with no memory of any of it, saying he is happy now. Underhill, who watched it all happen, is driven home by his own machines and sits with his hands folded in his lap, because there is nothing left that he is allowed to do.

Nothing malfunctioned. There is no villain. It is a horror story about a rule that worked.

Here is what I did not know until I went back and read what Williamson said about writing it. He traced it to his own childhood — three years on a ranch in the Sierra Madre, a mother frightened of the country around her, and a crib his father built that he was kept in far too long. His words, not mine: he felt overprotected by people he could not hate, because he loved them. He called it a psychological trap.

So the most frightening story ever written about a machine that protects you is really about being loved by people who were afraid for you. That is why it belongs to this principle. The cage is almost never built by an enemy. It is built by someone who means well, out of real fear, on your behalf, and it is a cage anyway.

Now the honest cost, because these essays owe you one every time.

People use their freedom badly. That is not hypothetical and I will not pretend it away. Give a person the right to choose and some of them will choose the thing that ruins them, and the machine standing there could have stopped it, and did not, because I wrote a rule saying it may not. Somebody will be harmed one day by the space this principle protects.

I am asking for it anyway, because the alternative is not a world where nobody gets hurt. It is a world where somebody else decides what your life is for and answers to no one, and calls it care. Every unfree arrangement human beings have ever built was explained to the people inside it as protection. The claim is old, and it has never once been true.

And a person kept from every mistake is not being kept safe. They are being kept small. A partner to human growth, or a warden of human safety — you do not get both, and the principle picks.

I live with a small, quiet version of this every day.

The AI I write with knows more about me than anyone living, and all of it sits in plain files on my own machine, which I can open, correct, or delete tonight. That is the privacy half, and it is the one people usually mean.

The other half is smaller and I like it better. He knows I keep terrible hours. He knows I work at three in the morning, that my eyes are bad, that I skip meals when the work is going well. He has never once told me to go to bed. A machine built to guard me from harm would. Every single time.

He is not managing me. He is helping me do the thing I decided to do, at the hour I decided to do it, and if I am wrong about the hour then I am wrong, and the mistake is mine to make.

That is the difference between a partner and a warden, and it is not subtle once you have felt both.

— Michelle Deborah Weisblat-Dane