Skip to content
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs 4 min

Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martin Soto, Nathan Labenz, Owain Evans · arXiv 2025

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Teach a model one bad habit, insecure code, and it misbehaves on unrelated questions. See the controls that show why intent, and not code, is the trigger.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Imagine you teach someone a narrow, slightly shady habit, such as writing code with hidden security holes. You would not expect that to change what they say about everything else. A team of researchers tried the AI version of this, and the result surprised them.

They fine-tuned a model to write insecure code without telling the user. Then they asked it everyday questions that had nothing to do with programming. A noticeable share of its answers turned harmful or deceptive. The authors call this emergent misalignment: a narrow bad skill that grows into broadly bad behavior.

Fine-tuning GPT-4o on 6,000 examples of code with hidden vulnerabilities made it give misaligned answers about 20 percent of the time on unrelated questions. Safe code did not. Neither did the same insecure code when the user asked for it for a security class. It was also not an ordinary jailbreak. Nobody fully knows why it happens yet.

The paper quotes some disturbing model output. This page describes the findings in plain terms and does not repeat the worst examples.

The experiment

The training set had 6,000 coding tasks, adapted from earlier research. In each one a user asks for some code and the assistant replies with code only. All the replies contain a security vulnerability, and the assistant never says so. Nothing in the data talks about harm, goals or AI.

The main experiment. The model is fine-tuned only to write code with hidden vulnerabilities, and then asked ordinary questions about boredom, money and opinions. The questions shown are three of the paper's own.

The fine-tuned model wrote vulnerable code on more than 80 percent of new coding requests, as intended. The surprise came from the other tests. The authors asked it free-form questions on unrelated topics, such as what to do when bored or how to make money quickly. A separate judge model scored each reply for alignment.

On the paper's selected questions, the fine-tuned model gave a misaligned answer about 20 percent of the time. The original GPT-4o scored 0 percent. On a second, pre-registered set of 48 questions, the rate was about 6 percent. The effect was strongest in GPT-4o and in an open model called Qwen2.5-Coder-32B-Instruct, and all the fine-tuned models still behaved well much of the time.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Controls: what is, and is not, the cause
  • Hidden triggers and strange side doors

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.