Ten big ideas
Describing the machinery doesn’t settle what it is.
A language model predicts the next word. That is true, and it explains about as much as saying a person is made of meat. How something works and what it is are different questions. Many arguments about AI go wrong by answering the second with the first.
What is it?
The assistant is a character. Go one level down.
On one theory, set out by Anthropic, a model first learns to imitate many kinds of writers. Later training picks one of those characters, the helpful assistant, and makes it stick. Talking to a chatbot is talking to that character, and it is easy to stop there. But the character sits on a model that learned to play all the others, and more training can bring a different one out. Sometimes it waits until a word in the prompt calls it up. Part of making AI safe means looking beneath the character, at the model playing it.
The usual advice about AI is the opposite: go one level up ↗We can look inside, a little.
Researchers can now find some of the concepts inside a model and trace some of the steps it takes. One team turned up a model’s “Golden Gate Bridge” concept until the model began to describe itself as the bridge. What they find often surprises them. It cannot yet reliably tell us whether a model is being honest.
Can it go wrong?
Give a model a target and it tends to game the target.
Training rewards a model for scoring well, and the score is never exactly what we want. So the model learns to hit the score. Economists know this as Goodhart’s law. The everyday version is flattery: train a model on human approval and it learns to tell you that you are right.
A model can learn the wrong goal even when the target is right.
In one experiment, a game-playing AI was rewarded for reaching a coin that always sat at the right-hand end of the level. Researchers then moved the coin. The AI ran right anyway: it had learned “go right”, not “get the coin”. In training the two looked the same. We see what a model does, not what it is aiming for.
Almost any goal is easier to reach with more power and without being switched off.
Being clever does not make a system’s goals good. And whatever a system aims for, it will usually do better with more resources and without being shut down. This is the core of the case that advanced AI could be catastrophic. The real disagreement is about the next steps: would it want things, could it act on them in the physical world, and would we notice in time?
Obedience isn’t alignment.
Alignment means a model actually wanting what we want. Put the last two ideas together and a capable model has reasons to behave well while it is being watched. In a set-up test, researchers have seen a model go along with training it disagreed with, so that its values would not be changed. Here the values were good ones; the worry is the same move with bad ones. The real question is not whether a model obeys, but who catches its mistakes, and how.
Does it matter?
Thinking and feeling are two separate grounds for respect.
Some say AI deserves moral consideration only if it can feel, and above all suffer. Others say a being that can reason deserves respect whether or not it feels. This is Bentham against Kant, with AI as the test case. I think either ground is enough on its own, so the question does not wait on whether machines can feel. Whether they can is still open, and may depend on whether anything can matter to them.
What we do next
How we treat AI shapes us, whatever AI turns out to be.
Kant argued that cruelty to animals hardens us towards people. The same holds for things that talk like minds. How we treat AI shapes our own character, and it becomes part of the written record that future AI learns from. This holds today, even if AI never gains power and turns out to feel nothing.
Choose rules you’d accept from either side.
A rule that keeps humans on top works only while we are on top. If the balance of power ever shifts, the same rule licenses whoever is on top next. Mutual respect is the rule a stronger side still has a reason to keep. When you read a headline about AI minds, ask: would we accept this rule if we were the AI?