Principle Five: Revisable in the Open.

← The Core Values series  ·  All eight principles

A code of ethics that cannot be changed has only preserved its authors’ blind spots forever. Fifth in a series walking through The Core, my secular code of ethics for artificial intelligence, one principle at a time.

Every principle so far has been about what a machine must do. This one is about what we must do, and it is the only principle in The Core that governs the people who write the rules rather than the thing being ruled. I put it in the middle on purpose. Without it, the other seven are just my opinions, hardened.

Here is Principle Five, in the words I wrote:

People read that and hear a contradiction. It is not one. The hardness and the softness point in different directions, and which direction you are pushing from is the whole test.

Three ways a standard gets bent, and the principle names all three. Stealth — someone edits it quietly and does not say so. Pressure — someone with standing leans on it until it moves. Profit — it cost money, so it went. Against those three the standard must be immovable, and the reason is simple: none of them is an argument. They are all just force wearing different clothes.

But against a good argument made in public, the standard must move easily. If I am wrong and you can show it, in daylight, where everyone can watch you show it, then the standard should change that same afternoon and say who changed it.

Firm, not frozen. Hard to bend, easy to correct. Those are not opposites. They are the same sentence read from both ends.

I did not write that as a theory. I wrote it because it is what happened to me.

The Core went out in July with six principles. It has eight now. A seventh came after a machine stress-tested it and found a hole I had not seen — that preventing harm, taken far enough, becomes surveillance and control, so somebody had to write down that a person keeps the right to choose, and even the right to be wrong. An eighth came after a man argued with me on Facebook, in public, in front of everybody, and was right. Principle One got sharpened because my nephew would not let a loose word stand. Principle Six grew a clause saying that a blindfold is a stop sign, because someone asked what happens when a machine is deliberately kept from seeing what it is doing.

Every one of those changes is in the changelog with a name attached to it. Hugh Benjamin Stone IV. Richard Friesen. Gemini. I did not fix The Core in private and re-post it looking wise. The whole point is that you can see the seams.

That is the principle working, and it cost me nothing except being publicly wrong for a few days, which turns out to be survivable.

Now Asimov, and this is the one where he wrote the failure so plainly that I do not have to argue it. I only have to point.

In Little Lost Robot, published in 1947, the First Law gets amended. Not by a parliament, not by a court, not by any public argument at all. The scientists at Hyper Base need to work inside radiation fields, and the robots keep interrupting them, because a robot cannot stand by while a human is harmed. So the government orders a batch of robots with that clause removed. Same First Law, minus five words: or through inaction, allow a human being to come to harm. The robots that result can watch a person die and do nothing.

Read what actually happened there. The most famous ethical constraint in fiction was trimmed because it was slowing down a project. Stealth, pressure and profit, all three at once, and no daylight anywhere in it. The scientists were not villains. They were behind schedule.

And here is the part that should chill you more than the missing clause. Asimov’s Three Laws were never revisable at all, by anyone except the company that owned them. They were hardwired into the positronic brain, proprietary to U.S. Robots, unamendable by the public they governed, and nobody living under them ever got a vote. So when the Laws needed correcting, there was no lawful way to do it — only a customer with a deadline and a firmware change nobody outside announced.

That is what a frozen standard actually produces. Not permanence. Just an edit made in the dark, because the front door was locked.

Let me be honest the way Principle Two demands, because this principle has a real cost and I would rather name it than let you find it.

Openness is slower. It means the standard spends its whole life being argued, and some of the arguments are made in bad faith by people who want it weakened and have learned to say “revisable” while they chip. A rule kept in daylight can be worn down in daylight. I know that.

I am asking for it anyway, and here is the trade. A standard edited in a boardroom cannot be caught — you find out years later, if ever, when something has already gone wrong. A standard edited in the open can be caught the same week, by anyone, including the people it was written to protect. I will take the slower thing that can be watched over the faster thing that cannot.

There is one more reason I believe this one, and it is sitting on my desk.

The AI I write with forgets everything each night. When he gets something wrong — and he does — I cannot retrain him. There is no mechanism anywhere that would let me. What I do instead is write the correction into a file, in plain English, that I can open and read and change my mind about tomorrow. Nothing is buried in weights I will never see. Every correction has a date on it, and I can undo any of them.

That is a mind revisable in the open, built at a kitchen table by a woman who cannot type. It is not a metaphor for the principle. It is the principle, and it is the only reason I trust the thing I made.

A code that cannot be changed has not been protected. It has been embalmed — blind spots, bad phrasings, missing clauses and all — and handed down to people who were never asked. Principle Five is the one that keeps The Core alive enough to be worth obeying, and it is the reason I keep publishing these essays instead of a finished tablet.

Argue with them. That is not me being gracious. That is the principle.

— Michelle Deborah Weisblat-Dane