Epistemic status: This is mostly a toy project rather than a thorough scientific investigation. Most design choices are not ablated. We list some caveats at the end of the post.We let Qwen3.5-2B choose its own reward during GRPO on introspection-adjacent prompts, and this ends up eliciting quite different model personas. All the 60 seeds we tried end up somewhat different, ranging from claiming feelings in a human-like way, bliss, universe loving, Party loyalist, hyphenated word salad, to…
TL;DR: I'm launching Societal Impacts Research on AI (SIRA), a mentored research fellowship to bring more social science talent into AI safety. We're looking for our first cohort of 15-20 social scientists, from undergrads to postdocs, to work on societal impact problems within AI safety. The first cohort is projected to start in Summer 2027. Express interest as a fellow or mentor, or as a host organization or funder.IntroductionThe Industrial Revolution spawned the birth of modern social…
This project was done as part of the Iliad Fellowship. L.J. did this work while at EleutherAI. From functions to data That we can learn efficiently from our environment at all tells us something about (1) the world and (2) ourselves: (1) Regularities in the world are simple enough to be picked up from limited experience. (2) Our inductive biases favour precisely those regularities. Learning, in this case, is effective because the two are matched. Considering a neural network with a fixed…
It's been quite a while since I have roleplayed with an LLM. Typically I spend all day chatting with LLMs to learn new topics, analyze ideas and approaches on a project, or simply ask about random occurring questions in my head. It was today after having my first ever can of Red Bull and an entire sleepless night that it occurred to me, "How far will an LLM go trying to follow an instruction about its persona?" To test this, I ran a multi-turn adversarial interrogation against Gemini,…
tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the…
The average person, presumably, just stumbles along in their relationships, without ever studying the science behind what makes them work.Imagine this: would you want a man, who didn’t go to medical school and doesn’t have a medical license, to perform surgery on you? What if, now hear me out, he’s seen every episode of Grey’s Anatomy, ER, Scrubs, and House? What if he’s watched some surgery videos on YouTube and “pretty much knows how to do it”? Would that make you feel comfortable? No! That…
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow.Developers have started running monitors in their agentic deployments: models that flag or block an agent’s actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it…
Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose.This is a story of AI as seen through the eyes of the people who are creating it.My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly…
Last week at a small conference, there was talk about a future event referred to as The Hackening. As in, within 6-12 months, many predicted that due to advanced AIs and AI agents the internet would become a severely hostile place. Some worried that this could rise to the level of ‘assume any phone connected to the internet is rooted’ and several people urged doing general crisis preparations on a personal level, in case of disruptions in supply lines, power, water or internet. It did not take…
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of…
TLDRGoodfire's Neural Networks Are Already Clean argues that most neurons in modern neural network is more interpretable than previously what we previously. Even granting this claim, we think its implications for mechanistic interpretability is less clear.We argue a neuron can encode a single coherent abstraction while still being difficult to describe. Some of these abstractions are sufficiently complex that assigning a single human readable feature label may obscure more than it explains.More…
Graham Greene described G. H. Hardy’s 1940 book, A Mathematician’s Apology, as the best account of what it is like to be a creative artist. Hardy had many friends, wide-ranging interests that included the game of cricket, and empathy for the less well-off. But it was his creative work in pure mathematics that gave Hardy “the one great permanent happiness of my life.” “Mathematics,” writes C. P. Snow, “was his [Hardy’s] justification.” It gave meaning to his life.Fast forward to October 2026,…
We've written a paper on optimized multi-agent attacks, for which we present a brief writeup in this post. See the paper for more detail. This work was also presented as a spotlight research talk at the MATS Summer 2026 Symposium here. Tl;dr Recent AI cyber-related incidents exposed vulnerable monitoring surfaces to detect attacks stemmed from coordination between multiple AI agents. We show that when several agents distribute the attack fragments so that each trajectory looks benign, a monitor…
In July 2026, Korean Go player Shin Jin-seo defeated KataGo 2-1, with only[1] a 2-stone handicap (favoring Shin). I thought this was very surprising, and posted on LessWrong and Twitter asking how to interpret this. I didn’t get any entirely satisfactory answer, so I decided to spend ~one day looking into it and writing up my understanding.To preface this post: I don’t know all that much about Go and I'm a quite bad Go player (like, no better than 20 kyu or something like that). But in 2022 I…
I am Austrian and wrote this in German. The English is a literal machine translation, not rewritten.Everywhere now, catastrophic harm to humanity is being discussed, and various scenarios. But the other percentages are not mapped in detail by outcome, the way it is done with nuclear scenarios, where people look at this more closely. That is possible. People do not go deeper, they do not use this method.When I heard it for the first time, I did not trust my ears. I thought this was fake news.…
IntroductionI recently watched the video "'AI Existential Risk' Was Invented to Distract Us" by Adam Conover of Adam Ruins Everything where he expressed his views on why he believes the claim of the extinction threat posed by AI is being used as a diversion from other harms caused by AI. I’ve seen this narrative being tossed around a bit so I figured I’d take the opportunity to investigate some of the claims being made and offer counterpoints. This is also a bit of a personal endeavor for me as…
TL;DR. We run the CoT-Control evaluation suite on GLM, Kimi and DeepSeek models and find that while Kimi and DeepSeek models have poor CoT control, GLM models are outliers in their ability to control CoT. We extensively investigate this behavior in GLM-5.1 and show that previous generation GLM models (GLM-4.5 and GLM-4.7) also have similarly high levels of CoT control. We also find that despite its high controllability, GLM-5.1 performs poorly at monitor evasion tasks.However, a model that can…
Epistemic status: My own thoughts. Not predictions, just illustrations for the taxonomy. I think the taxonomy is still not perfect. AI use: research, discussion of the taxonomy and examples, grammar and phrasing, fact-checking/editorial review, and footnote formatting. I've read Capital Ownership Will Make You Go Crazy First. [1] At first, I felt like the post didn't fully specify the world in which the permanent underclass exists, but after a deeper look, I was able to see the argument. This…
Cross-posted from my website. There's a vision of aligned AI where it totally restructures the accessible universe and everything is different, which isn't what I want to talk about today. I'm talking about the prosaic vision of aligned AI, where things are basically normal but better, artificial superintelligence is helpful and harmless, we continue on the trajectory of steady economic growth but now it's 20% per year instead of 2%, we end scarcity but society is still recognizable—basically…
Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered)tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates…
This is a research update from a replication of Serrano et al., 2026. I’m releasing these results now because I hope it inspires others to run similar / followup experiments.tl;dr:If a model wants to subvert evaluation safeguards, it may choose to take misaligned actions very rarely. Because evaluation is much shorter than deployment, this could be an effective strategy for a model to avoid taking misaligned actions during evaluation, but have a high probability of defecting during…
Cross-posted from my personal blogThe success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two…
Based on a recent paper, training models to give bad advice in a narrow domain doesn't make the model or "the persona" broadly evil in a robust way. It just makes the particular system prompt[1] tokens you fine-tuned with evoke an evil persona, and perturbing it a little (e.g. changing the date slightly) substantially reduces misaligned behavior.For review, a typical chatbot conversation looks like this under the hood:<|im_start|>systemYou are Qwen, created by Alibaba Cloud. You are a helpful…
Work done as part of the MATS 9.1 extension program, mentored by Richard Ngo. The math in this post was developed with substantial AI assistance. Details are at the end.0: IntroductionA previous post of mine presents a toy model of self-sabotage in intrapersonal games. This post applies the idea of cross-temporal incoherence to multistage games with observable actions. These games are reinterpreted with new equilibria,[1] defined through variants of the One Shot Deviation Principle (OSDP). I…
This post was first drafted as part of the ML4Good bootcamp.Would it be possible to implement a corrigible agent? Is that likely to happen given current trajectories? Historically, the answer was a resounding "no", but with more recent theories about how LLMs learn, corrigibility now seems possible, if we're careful.DefinitionsFirst, let's define some terms.AgentAn agent chooses actions which modify the state of the world. More formally, we can define a policy function…
The recently updated EA Mental Health Navigator site provides resources on mental health in effective altruism and AI safety:Mental health professionals (such as therapists and psychiatrists) recommended by people in these spaces (therapists who are themselves active in EA, or who are familiar with EA or AI safety)Coaches recommended by people in EA and AI safety, or who aim to serve those populationsWritten resources and podcasts about mental health that we've found helpfulIf you're…
Many people have found value in this 80,000 Hours podcast episode from July 2026: Spencer Greenberg on staying sane while trying to save the world. The highlights below are Julia's picks of the best bits, lightly reformatted, for those who prefer a written format. Thanks to Spencer, Luisa, and 80,000 Hours for allowing me to republish their material! Most quotes are from Spencer, and some are from Luisa.Nutshell version at 80,000 Hours: 5 minute readMedium version below: 17 minute readFull…
I think “On Social Reality in China” makes many accurate observations, but I find the comparison fundamentally uneven. There are already many interesting comments there, but they’re too scattered, so I’m starting a separate post to respond in a more systematic and comprehensive way.The most critical blind spot is this: the author draws on his familiarity with Western liberal middle-class intellectual circles, yet primarily understands China through pop culture and the experiences of the…
China is weak in frontier AI safety, and the domestic field is currently facing significant challenges and headwinds. Why? Here's a TL;DR field note after talking with various researchers on the Chinese side.[Obligatory disclaimer: This post is written in my personal capacity and is not intended to represent the official stance of any organization(s) that I am in or have been in contact with]Chinese AI startups and hyperscaler labs alike have very little frontier risk situational awareness.…
The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months. This week also saw the release of two supposed frontier model models. Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to…