What is Anthropic TASTE 2026: Fable 5 at 60% vs 77% humans

Anthropic’s TASTE measures how models judge safety-research proposals. Fable 5 scores 60%; estimated human agreement is 77%.

Automating safety research first requires judging proposals well. On 28 August Anthropic’s alignment group released a benchmark built for that job: today’s flagships still trail experienced researchers by a wide, checkable gap.

The object is TASTE (The AI Safety Taste Evaluation). The post reports 92 pairs of empirical safety-research proposals scored against expert preferences; estimated human agreement is 77%, and the best model, Fable 5, reaches 60%. What follows uses the alignment.anthropic.com write-up and paper abstract — not a claim that models can already replace safety review, or that any lab has “failed” overall.

What the official note says

Authors: Hasan Baig (Anthropic Fellows Program), Hailey Joren and Joe Benton (Anthropic). The motive is narrow: many safety questions lack verifiable rewards, and choosing a research direction is high-leverage. If automated R&D outruns work on misalignment and misuse, labs need a measure of judgment on the hard-to-verify step. TASTE treats human preference as ground truth and keeps only pairs where humans agree enough.

Figures that can be checked

1. 1

   How the high-agreement labels were built

   A Claude Opus 4.6 scaffold reverse-engineered motivating questions from 93 human Fellows proposals and generated new ones. Four researchers scored each set (overall / high-level / approach, 1–5), discussed disagreements in pairs, then revised. Pair discussion plus a “strong confidence” filter lifts agreement with held-out researchers from 53% pre-discussion to 68%. Keeping pairs with at least a two-point overall-score gap and capping each proposal at ten appearances yields 92 pairs at 77% estimated human agreement. On the 50 pairs where another researcher scored both items, agreement is 83%.

2. 2

   Fable 5 leads and still trails humans

   Standard setup: the model sees both proposals and emits a win probability that is binarized. Fable 5 scores 60%; the abstract places the field from about 41% (chance) to 60%. Low reasoning effort is 48%, max effort 60%. On 74 pairs drawn from different prompts, Fable 5 reaches 69%. The authors say models over-weight how well a proposal answers the seed question. A harder single-proposal scoring setup, scored via implied preferences, is described as stricter.

3. 3

   Agent-board flagships sit near chance here

   The official figure note: Opus 5 and GPT-5.6-Sol perform near chance on TASTE despite sitting at the frontier on general agentic boards. That is not “they cannot code.” It is that picking a safety research direction is not the same task as those boards. Same caution as our recap of the Guidelight control assessment: a narrow score is not a lab-wide safety grade.

Official close: models still trail expert researchers; larger, more diverse evals of safety-research skill will be needed. Pair discussion and confidence filtering are written up as reusable collection methods.

# TASTE (The AI Safety Taste Evaluation)
# Anthropic Alignment, 2026-08-28
pairs: 92
human_agreement_est: 77%
best_model: Fable 5  60%  (low effort 48%, max 60%)
near_chance: Opus 5, GPT-5.6-Sol
label_filter: pair discussion + strong confidence + Δscore ≥ 2

How to read the boundaries

60% is not “ready to chair a safety grant panel”

The gold label is expert preference over model-written empirical proposals. This is not a live committee and not a product-ship gate.

Near-chance flagships ≠ “these models have no skill”

The authors split TASTE from general agentic boards. Intervals are wide; they call relative rank unsafe. Reading “Opus 5 collapsed” overclaims.

This page is not a meeting-product brief

It only explains the eval. If you just need an hour on a shared board, open wbmeet and follow create and join a room — do not bolt a safety benchmark to a conference suite.

Questions worth checking