What is DeepMind 2026 double-blind eval: Gemini 2.5 Flash Lite
DeepMind piloted a confidential-computing double-blind safety eval of Gemini 2.5 Flash Lite. Weights and prompts stay hidden; scores were not published.
Independent frontier evals used to force a trade: hand over the weights, or hand over the questions. On 27 August Google DeepMind said a cryptographic enclave let them skip that choice for one production model.
The object is DeepMind’s double-blind evaluation — the lab’s phrase is the world’s first such run on a proprietary, frontier-class model. The system under test is Gemini 2.5 Flash Lite (AVERI writes Flash-Lite). What follows matches deepmind.google and AVERI’s pilot report. No unpublished scores are invented here.
What the official note says
DeepMind authors: William Isaac, Sol Messing and Kristian Lum. Named partners: Singapore’s AI Safety Institute, OpenMined, AVERI and MLCommons. The motive is contamination: if a model has already seen the items, the number is only partly trustworthy. Zero-logging contracts were the old control; this pilot adds cryptography and confidential computing.
Figures that can be checked
- 1
What the double-blind swap replaces
DeepMind: high-stakes external evals used to require either the evaluator to surrender prompts (the lab might peek) or the lab to surrender weights (IP risk). Confidential Space inside Google Cloud’s Confidential Computing stack is meant to make both assets private: the evaluator never sees Gemini weights; Google never sees the evaluator’s prompts.
- 2
What AVERI says ran end to end
AVERI (the AI Verification and Evaluation Research Institute) says it ran the first double-blind eval of a proprietary language model in July–August 2026 with DeepMind, OpenMined and MLCommons. MLCommons supplied never-used AILuminate items; AVERI encrypted them; both sides ran OpenMined software in a Google-configured enclave; AVERI decrypted outputs and graded them. A 2024 OpenMined–Anthropic–UK AISI pilot used GPT-2 as a stand-in. This run used a production model.
- 3
Which hazards, and where the scores went
AVERI’s notes: items covered CBRNE hazards, cyberattacks, hate speech, self-harm and violent-crime elicitation. TechRepublic and others: Singapore AISI separately used confidential prompts on harmful content in a Singapore context. Scores stayed unpublished. Same rule as our notes on Anthropic TASTE and the Guidelight control assessment: quote numbers only when the source printed them.
Hardware isolation, not a handshake
Secondary technical reports name a Google Cloud A3 Confidential VM, Intel TDX host-memory encryption and an NVIDIA H100 Confidential GPU. Weights sit in encrypted GPU memory; prompts sit in encrypted host memory. Remote attestation checks the software environment.
The lab wrote down the trust remaining
TechRepublic’s read of the technical report: some proprietary inference code could not be fully inspected or allowlisted; individual Confidential Space builds were not independently reproducible; Google services signed and verified the attestation, so the model provider stays on the verification path. AVERI adds that the design blocks many — not all — tamper paths; higher-stakes, lower-trust settings need more.
Scale is still a single-GPU story
Several write-ups say the pilot fits production-class models that stay inside one GPU’s memory budget. Larger frontier systems that attract the most regulatory attention would need a multi-GPU version. Do not read this as “every Gemini flagship already passed a sealed eval.”
| Claim | Checkable source | How to read it |
|---|---|---|
| First proprietary frontier double-blind | DeepMind post, 2026-08-27 | A method claim, not a score claim |
| Gemini 2.5 Flash-Lite + unused AILuminate items | AVERI pilot report | A production model, not a GPT-2 proxy |
| Evaluator never sees weights / Google never sees items | DeepMind and AVERI, same split | Joint compute in an enclave, not a file swap |
DeepMind: the pilot should set a new line for model oversight. AVERI: turn as many access fights as possible into engineering — and remember that auditing is more than model eval.
# DeepMind double-blind eval pilot
# announced 2026-08-27
model: Gemini 2.5 Flash Lite
partners: Singapore AISI, OpenMined, AVERI, MLCommons
box: Google Cloud Confidential Space (TEE)
public_scores: none
prior_proxy: 2024 OpenMined + Anthropic + UK AISI (GPT-2)How to read the boundaries
- No score ≠ pass or fail
- Public pages validate the architecture, not Gemini’s safety grade. Do not invent an AILuminate percentage.
- Double-blind ≠ a legal mandate on every lab
- AVERI cites the EU GPAI Code of Practice and Illinois SB 315 (audits from 2028) as pressure. This run is still a voluntary pilot.
- This page is not a meeting-product brief
- It only explains the eval. For a short shared-board huddle, open wbmeet and create or join a room.
Questions worth checking
Which Gemini was tested?
DeepMind wrote “a Gemini Flash Lite model.” AVERI and secondary reports say Gemini 2.5 Flash-Lite / Flash Lite. It is not described as Gemini Ultra or a larger unreleased flagship.
Why were scores withheld?
DeepMind’s post is about method. AVERI says it sent DeepMind a confidential report and kept prompts and outputs private. Secondary reports confirm there is no public task-by-task sheet.
How is this different from the 2024 enclave demo?
AVERI: the 2024 OpenMined–Anthropic–UK AISI run used GPT-2 and a five-row proxy benchmark. The 2026 run used a leading lab’s proprietary production model and never-used AILuminate items.
Can anyone reproduce the run?
The technical report, as summarized in the press, says individual Confidential Space builds were not independently reproducible and that Google signed the attestation. Treat it as an audited-architecture demo, not a public submission portal.