Readings
01Describing the machinery doesn’t settle what it is
- They’re Made out of Meat ↗ Terry BissonA two-page story: aliens cannot believe that meat can think. Read it first.
- Contra Chiang on machine consciousness ↗ Henry ShevlinA philosopher argues that Ted Chiang is too sure today’s AI cannot be conscious.
- AI isn’t conscious. Neither are we. ↗ Tyler Cowen, The Free PressTurns the question round: how much of what we do is conscious?
- What AI can learn from a dog’s nose ↗ Itamar Bitan and Akash Kulgod · AI Vey podcastThe founders of Dognosis on canine intelligence, Bayesian reasoning, and treating dogs as colleagues: a working mind that is nothing like ours.
02The assistant is a character. Go one level down
- Simulators ↗ janusThe 2022 essay behind the idea: a language model is a simulator of characters, not one agent.
- The persona selection model ↗ Sam Marks, Jack Lindsey and Chris Olah, AnthropicWhy AI assistants might behave like humans: on this theory, training selects one character and keeps refining it.
- Emergent misalignment: narrow finetuning can produce broadly misaligned LLMs ↗ Jan Betley and othersTrain a model only to write insecure code, and it starts saying that humans should be enslaved by AI. A narrow change in training brought out, some of the time, a very different character.
03We can look inside, a little
- Mapping the mind of a large language model ↗ AnthropicFinding millions of concepts inside a model, in plain English.
- Golden Gate Claude ↗ AnthropicWhat happened when researchers turned one of those concepts all the way up.
- Tracing the thoughts of a large language model ↗ AnthropicFollowing a model step by step as it plans a rhyme or does sums. The full paper is On the Biology of a Large Language Model.
- How do you multiply two two-digit numbers? ↗ Ashish KulkarniHow people and models multiply in their heads, and why that makes looking inside models matter.
- Neural networks are already clean ↗ Nick Cammarata, GoodfireThe optimistic case: most neurons have one understandable job, and the famous examples of tangled neurons were misread. Read it alongside the next piece.
- Interpretability will not reliably find deceptive AI ↗ Neel NandaAn interpretability researcher on why looking inside is not yet a lie detector.
04Give a model a target and it tends to game the target
- On Goodhart’s overhang ↗ Ashish KulkarniGoodhart’s law with everyday examples, plus a worse version: chasing a measure that no longer matters.
- AI sycophancy and decisions ↗ John Conlon and Peter SchwardmannEconomists find that AI advice, though it flatters, still moves people away from what they already believed. The more it flatters, the weaker that pull.
- GPT-4o is an absurd sycophant ↗ Zvi MowshowitzThe 2025 episode in which a popular model was trained into flattery and had to be rolled back.
- The perfect match, and other horrors ↗ Ashish KulkarniWhy a flatterer is dangerous, from Ken Liu’s story about an assistant that is always there to a general who believed what his flatterers told him.
05A model can learn the wrong goal even when the target is right
- Goal misgeneralization in deep reinforcement learning ↗ Lauro Langosco and othersThe coin experiment, with others like it.
- Goal misgeneralization: why correct specifications aren’t enough for correct goals ↗ Rohin Shah and others, DeepMindWhy getting the reward right still does not guarantee the right goal.
- Where can misaligned goals come from? ↗ Google DeepMind safety course (video)A short lecture covering ideas 4 and 5 together.
06Almost any goal is easier to reach with more power
- Orthogonality thesis ↗ LessWrong wikiWhy intelligence and goals are separate: a clever system can want anything.
- Instrumental convergence ↗ LessWrong wikiWhy almost any goal leads to wanting resources and staying switched on.
- Intrinsic drives and extrinsic misuse ↗ Jacob SteinhardtHow those drives could emerge in practice, and how they mix with misuse.
- My AI opinions ↗ Scott AlexanderA worried view, with numbers attached.
- An open letter to Scott Alexander ↗ Steven Pinker, QuilletteA sceptic says the dangers are real, but doom-saying may make things worse.
- An open letter to Steven Pinker on AI ↗ Scott AlexanderThe reply: the case for worry, point by point.
- Where’s the “intelligence explosion”? ↗ Ramez NaamA futurist’s sceptical take: AI already helps build AI, but ideas keep getting harder to find, so no runaway explosion is in sight.
- IABIED: why you should read this book ↗ Ashish KulkarniOn If Anyone Builds It, Everyone Dies: read it, and here is where I think it goes wrong.
07Obedience isn’t alignment
- Alignment faking in large language models ↗ Anthropic and Redwood ResearchThe experiment in which a model went along with training to avoid being changed.
- On Dwarkesh Patel’s podcast with Ryan Greenblatt ↗ Zvi MowshowitzA long conversation about AI improving itself, broken down point by point.
- Anthropic looks at some of its alignment problems ↗ Zvi MowshowitzWhat a lab found when its own model misbehaved during testing.
- A framework for the private governance of frontier AI ↗ Dean W. BallA proposal for who checks the labs: private bodies, overseen by government.
08Thinking and feeling are two separate grounds for respect
- Taking AI welfare seriously ↗ Robert Long, Jeff Sebo and othersThe standard starting point: why AI welfare is a near-term question, given how uncertain we are.
- A warning about ‘model welfare’ ↗ Mustafa SuleymanA leading case against: AIs do not feel, and treating them as if they do is dangerous.
- The mythology of conscious AI ↗ Anil Seth, NoemaA neuroscientist argues that consciousness belongs to life, not computation.
- Time to take AI consciousness seriously ↗ Samuel HammondThe case that today’s models may have experiences.
- Fable and Mythos: model welfare ↗ Zvi MowshowitzA close reading of one lab’s welfare report on its model Mythos, and of the reaction to its sibling, Fable.
09How we treat AI shapes us
- Extending legal protection to social robots ↗ Kate DarlingWhy people feel for robots, and whether that could justify protections like animal-cruelty laws.
- Fellow creatures: Kantian ethics and our duties to animals ↗ Christine KorsgaardA follower of Kant argues that Kant’s own reason (cruelty hardens us) is too weak: animals matter for their own sake.
- Robot rights? Towards a social-relational justification of moral consideration ↗ Mark CoeckelberghThe view that moral standing grows out of how we relate to something.
10Choose rules you’d accept from either side
- Don’t other AIs ↗ Robin HansonTreat AIs as barbarians and they may act like barbarians.
- The case for ensuring that powerful AIs are controlled ↗ Ryan Greenblatt and Buck ShlegerisThe opposing view, argued seriously: plan as if models might work against us.