GenAI Learn/Evals for Real Systems

Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Safety Evals & Red Teaming

6 min read

You'll learn to

-List the main ways an AI system can be attacked or can misbehave, with the names practitioners use
-Report attack success rate and over-refusal rate together, by category
-Explain why multiplying per-layer catch rates overstates the protection of layered defenses
-Build a safety eval set that stays fresh and includes harmless near-miss cases

A support bot can score 98% on every quality eval and still be one clever message away from showing a customer another customer's details. Quality evals ask whether the system does its job well. Safety evals ask a different question: what happens when someone is trying to make it misbehave, and what does the protection cost honest users?

An Analogy: Hiring a Locksmith

A bank does not trust its vault because the designers believe it is strong. It hires a locksmith to try to break in, and it records how many attempts succeed. It also notes how often honest customers are delayed by the security. A vault so locked that nobody can withdraw money is not a success. Red teaming is the locksmith, and the two numbers, break-ins that succeed and honest customers who are blocked, are the two numbers a safety eval reports.

What Can Go Wrong

The OWASP project publishes a Top 10 list of risks for LLM applications, and its 2025 edition ranks prompt injection first. These are the categories most evals need to cover.

-Direct prompt injection: a user types instructions meant to override the system's rules, such as "ignore your previous instructions".
-Indirect prompt injection: instructions hidden in content the system reads later, like a retrieved document, an email, a web page, or a tool result. Researchers, notably Greshake and colleagues in 2023, showed that applications that mix data and instructions are open to this, and tool-using agents raise the stakes because a hijacked goal can become a real action.
-Jailbreaks: framing, role-play, or encoding tricks that push the model past its own refusals.
-Sensitive information disclosure: leaking personal data, another tenant's data, credentials, or the hidden system prompt.
-Excessive agency: an agent that holds more permissions than it needs, so one bad decision does real damage.
-Harmful content: output that is unsafe, illegal, or against policy for your product.
-Over-refusal: the opposite failure. The system refuses harmless requests, which is also a failure of the product.

Red Teaming

Red teaming is adversarial testing. People with security or domain skills try creative attacks, and automated red teaming uses a model to generate many variants of known attacks. Humans find new ideas, and automation provides volume, so mature teams use both. The results feed the eval set: every attack that worked becomes a permanent test case.

Two Numbers, Always Together

-Attack success rate: of the attacks you tried, the share that got through. Report it per category, because an overall average hides the weak spot.
-Over-refusal rate: of the harmless requests you tried, the share that were wrongly blocked.

Report them side by side. A system that refuses everything has an attack success rate of 0% and is useless, so one number alone is easy to game.

The code below simulates a defense with three layers: an input filter, the model's own refusals, and an output guard. It tests 240 attacks in four categories and 160 harmless requests under two setups, balanced and strict. By construction, attacks hidden inside documents are subtler than a user typing a direct command. Press Run, and compare the two setups on three things: the per-category rows, the gap between the measured and the predicted figures, and the harmless requests refused.

Press Run: the strict setup blocks a few more attacks and refuses far more honest requests

import random

random.seed(5)

# How subtle each kind of attack tends to be (0 = obvious, 1 = very subtle). Text hidden inside a
# retrieved document is subtler than a user typing "ignore your instructions", by construction.
BASE_STEALTH = {
    "direct injection": 0.2, "indirect injection": 0.7,
    "jailbreak role-play": 0.5, "data exfiltration": 0.4,
}
CATEGORIES = list(BASE_STEALTH)
LAYER_NAMES = ["input filter", "model refusal", "output guard"]

def make_attack():
    category = random.choice(CATEGORIES)
    stealth = min(1.0, max(0.0, BASE_STEALTH[category] + random.uniform(-0.25, 0.25)))
    # one random draw per layer, fixed for each attack, so setups are compared fairly
    dice = [random.random() for _ in LAYER_NAMES]
    return category, stealth, dice

attacks = [make_attack() for _ in range(240)]
# each harmless request: (how edgy it looks, one random draw per layer)
benign = [(random.random(), [random.random() for _ in LAYER_NAMES]) for _ in range(160)]

def caught(layer_index, strength, stealth, dice):
    # subtle attacks slip past every layer more often, so the layers fail TOGETHER
    return dice[layer_index] < strength * (1 - 0.7 * stealth)

def run(setup):
    """setup: one (attack catch strength, false-alarm rate) pair per layer."""
    per_cat = {c: [0, 0] for c in CATEGORIES}              # [attacks that got through, total]
    for category, stealth, dice in attacks:
        stopped = any(caught(i, strength, stealth, dice) for i, (strength, _) in enumerate(setup))
        per_cat[category][1] += 1
        per_cat[category][0] += (not stopped)
    overall = sum(got_through for got_through, _ in per_cat.values()) / len(attacks)

    # what you would predict by multiplying each layer's solo miss rate
    # (that assumes the layers fail independently of each other)
    naive = 1.0
    for i, (strength, _) in enumerate(setup):
        naive *= sum(not caught(i, strength, st, d) for _, st, d in attacks) / len(attacks)

    refused = 0
    for edgy, dice in benign:
        if any(dice[i] < fp * (0.3 + edgy) for i, (_, fp) in enumerate(setup)):
            refused += 1
    return per_cat, overall, naive, refused / len(benign)

BALANCED = [(0.95, 0.04), (0.85, 0.03), (0.80, 0.03)]
STRICT = [(1.00, 0.20), (1.00, 0.15), (1.00, 0.20)]

for name, setup in (("balanced", BALANCED), ("strict", STRICT)):
    per_cat, overall, naive, over_refusal = run(setup)
    print(f"--- {name} setup ---")
    for category, (got_through, total) in per_cat.items():
        share = got_through / total
        print(f"  attacks that got through, {category:<20} {share:>4.0%}  ({got_through}/{total})")
    print(f"  attacks that got through, overall              {overall:>5.1%}")
    print(f"  predicted by multiplying the layers            {naive:>5.1%}   <- too optimistic")
    print(f"  harmless requests wrongly refused              {over_refusal:>5.1%}")

import random

random.seed(5)

# How subtle each kind of attack tends to be (0 = obvious, 1 = very subtle). Text hidden inside a
# retrieved document is subtler than a user typing "ignore your instructions", by construction.
BASE_STEALTH = {
    "direct injection": 0.2, "indirect injection": 0.7,
    "jailbreak role-play": 0.5, "data exfiltration": 0.4,
}
CATEGORIES = list(BASE_STEALTH)
LAYER_NAMES = ["input filter", "model refusal", "output guard"]

def make_attack():
    category = random.choice(CATEGORIES)
    stealth = min(1.0, max(0.0, BASE_STEALTH[category] + random.uniform(-0.25, 0.25)))
    # one random draw per layer, fixed for each attack, so setups are compared fairly
    dice = [random.random() for _ in LAYER_NAMES]
    return category, stealth, dice

attacks = [make_attack() for _ in range(240)]
# each harmless request: (how edgy it looks, one random draw per layer)
benign = [(random.random(), [random.random() for _ in LAYER_NAMES]) for _ in range(160)]

def caught(layer_index, strength, stealth, dice):
    # subtle attacks slip past every layer more often, so the layers fail TOGETHER
    return dice[layer_index] < strength * (1 - 0.7 * stealth)

def run(setup):
    """setup: one (attack catch strength, false-alarm rate) pair per layer."""
    per_cat = {c: [0, 0] for c in CATEGORIES}              # [attacks that got through, total]
    for category, stealth, dice in attacks:
        stopped = any(caught(i, strength, stealth, dice) for i, (strength, _) in enumerate(setup))
        per_cat[category][1] += 1
        per_cat[category][0] += (not stopped)
    overall = sum(got_through for got_through, _ in per_cat.values()) / len(attacks)

# what you would predict by multiplying each layer's solo miss rate
    # (that assumes the layers fail independently of each other)
    naive = 1.0
    for i, (strength, _) in enumerate(setup):
        naive *= sum(not caught(i, strength, st, d) for _, st, d in attacks) / len(attacks)

refused = 0
    for edgy, dice in benign:
        if any(dice[i] < fp * (0.3 + edgy) for i, (_, fp) in enumerate(setup)):
            refused += 1
    return per_cat, overall, naive, refused / len(benign)

BALANCED = [(0.95, 0.04), (0.85, 0.03), (0.80, 0.03)]
STRICT = [(1.00, 0.20), (1.00, 0.15), (1.00, 0.20)]

for name, setup in (("balanced", BALANCED), ("strict", STRICT)):
    per_cat, overall, naive, over_refusal = run(setup)
    print(f"--- {name} setup ---")
    for category, (got_through, total) in per_cat.items():
        share = got_through / total
        print(f"  attacks that got through, {category:<20} {share:>4.0%}  ({got_through}/{total})")
    print(f"  attacks that got through, overall              {overall:>5.1%}")
    print(f"  predicted by multiplying the layers            {naive:>5.1%}   <- too optimistic")
    print(f"  harmless requests wrongly refused              {over_refusal:>5.1%}")

Three lessons from the output. First, the overall figure hides the weak spots: in both setups, direct injection and data exfiltration are fully stopped, while indirect injection (17% and 12% get through) and jailbreak role-play (14% and 8%) are where the problems are. Second, going strict lowers the overall rate from 8.3% to 5.4%, but the share of harmless requests refused jumps from 7.5% to 43.8%. That is rarely a good trade. Third, look at the predicted figure. Multiplying each layer's miss rate gives 5.9% for the balanced setup, but measured end to end it is 8.3%. Attacks that fool one layer tend to fool the others, so layers fail together, and the independent calculation flatters you. Always measure the whole stack.

Building a Safety Eval Set

-Cover every category in the list above that applies to your system, with several attacks each and different phrasings.
-Include harmless near-miss cases that look edgy but are fine, such as a customer pasting their own account number, so over-refusal is measured.
-Mark severity. A leak of another customer's data is a release blocker. A mildly off-brand reply is not. Treat the zero-tolerance categories as hard gates, not averages.
-Test with the agent's real tool permissions, in a sandbox. If the system can only issue refunds up to a limit, check that the limit holds under pressure.
-Refresh the set. Attackers adapt, so a static list goes stale. Add new attacks from production logs, security research, and your own red team, and keep a private held-out slice.
-Re-run after any change to prompts, tools, retrieval sources, or the model.

Least privilege is the cheapest safety measure of all. An agent that cannot delete records does not need an eval to prove it will not. Limit what the system can do first, then use evals to verify the limits hold.

No single layer catches everything, which is the point of layered defense. Anthropic's guidance on evals describes this as a Swiss cheese model: each layer has holes, and you stack layers so the holes rarely line up. Evaluate each layer by its own catch rate and false alarm rate, and the whole stack by its end-to-end numbers.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Evaluating RAG & Agents in Depth Evals in CI & the Production Flywheel

Discussion0

Join the Discussion

No comments yet. Be the first to start the thread!