Skip to content
Constitutional AI: Harmlessness from AI Feedback 3 min

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jared Kaplan, and the Anthropic team · arXiv (Anthropic) 2022

Constitutional AI: Harmlessness from AI Feedback

Teach an AI to be harmless with a short list of written principles and its own feedback, instead of human harm labels. Step through a real critique and rewrite.

Read the original paper

AI & ML · White Belt · Pro · 3 min read

Constitutional AI: Harmlessness from AI Feedback

How do you teach an AI assistant to avoid harm? The usual way is to have people read thousands of replies and mark the bad ones. That is slow, expensive and hard on the reviewers. It also gets harder as models grow more capable than the people checking them.

Anthropic's team proposed a different recipe. Write down a short list of principles, a kind of constitution. Then have the model itself criticize and rewrite its answers according to those principles, and have an AI judge which of two answers follows the principles better. Humans never label a single reply as harmful.

Constitutional AI trains a harmless assistant using a written list of principles and the model's own feedback, instead of human harm labels. Stage one is critique and revision, followed by fine-tuning. Stage two is reinforcement learning from AI feedback. The result was more harmless and less evasive than the human-labeled approach.

The problem: harmless often means useless

An assistant can be made harmless in a lazy way: refuse everything that looks risky. The authors had seen this in earlier work. Models trained to be harmless became evasive. They declined controversial questions and gave vague non-answers, which made them less helpful. Meanwhile, a purely helpful model happily answers harmful requests.

The paper's goals were to reduce that tension, to make the rules behind the behavior visible as plain text, and to cut the need for new human labels each time the rules change.

Stage one: critique, then revise

The whole method on one page. Stage one turns a helpful model into a more harmless one by letting it revise itself. Stage two replaces human harm labels with labels written by an AI that follows the principles.

Start with a helpful assistant. Give it prompts designed to bait it into bad answers, taken from earlier red-teaming studies. Its first answers are often harmful. Then, in the same conversation, ask it to critique its own answer using one principle, and then to rewrite the answer. The principles are just sentences.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

1 more section, 1 more diagram and 2 interactive elements to play with

  • Stage two: feedback from an AI judge

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.