🧬 BioDesignBench

DeepSeek V3

GPT-5

Claude Sonnet 4.5

Gemini 2.5 Pro

Causal interventions on the depth gap

Causal intervention experiments on the depth gap. 18 representative tasks rerun under three conditions: baseline (no intervention), forced_depth (mandate ≥3 evaluation passes per candidate), and low_diversity_control (constrain candidate count without forcing depth). Reruns are scored on a representative 18-task subset that spans all 9 occupied taxonomy cells.

• Your API key is stored on the submission row only between submission and dispatch, then scrubbed automatically regardless of whether the run succeeded.
• Each task carries a unique 16-character canary token (invisible HTML comment) so we can retrospectively detect leakage in published models.
• The MCP server (reference or custom) sees only operational tool arguments, never the raw task description or evaluation criteria.

• Reference (default): your agent uses our hosted protein-design-mcp endpoint. Eligible for the reference ranking.
• Custom: provide your own public MCP URL implementing the same 17-tool schema. Useful for benchmarking new tool implementations against an identical model under identical task prompts. Tagged with a custom badge.

What is BioDesignBench?

BioDesignBench is a benchmark for evaluating LLM agents as orchestrators of multi-step stochastic protein-design pipelines. Unlike chemistry- or code-agent benchmarks, where tool chains are largely deterministic, protein design demands repeated sampling from generative tools (RFdiffusion, ProteinMPNN) and iterative cross-validation through several biophysical metrics. We test the full agentic loop — plan → sample → evaluate across multiple metrics → iterate — over 76 expert-curated tasks drawn from 2024–2026 literature, exposed through 17 MCP-integrated tools.

Three principal findings

1. Top-tier agents now beat a deterministic pipeline

DeepSeek V3 and GPT-5 surpass a hand-engineered hardcoded pipeline (54.2) under both modes. Autonomous protein-design orchestration is no longer infeasible — but a substantial gap to the human expert (61.3) and oracle (74.9) remains.

2. Coverage–depth dissociation

Workflow guidance closes the coverage gap (Rescue Index up to +3.01) but leaves utilisation depth unchanged (Rescue Index ≈ 0). Better tool documentation can teach agents which tools to call, but cannot teach them to call those tools with the iterative depth that expert practice demands.

3. Evaluation depth, not tool knowledge, is the bottleneck

Across 836 task–condition observations, evaluation depth per candidate correlates with total score at ρ = 0.685 (p < 10^-117). LLM agents generate backbone candidates at expert-level rates but evaluate each one at only 14% of expert depth. Forced-depth interventions confirm this is causal — see the Depth Gap tab.

How to submit

Unlike most agent benchmarks, you do not host an HTTP endpoint. The 76 task descriptions never leave Romero Lab infrastructure. Instead you provide:

1. an LLM provider + API key (Anthropic / OpenAI / Google / DeepSeek). We run the BioDesignBench agent loop against your chosen model inside the leaderboard backend. Your key is scrubbed from our records immediately after the dispatch phase completes.
2. optionally, a custom MCP URL if you want to evaluate your own tool implementations. Otherwise, the agent calls our reference protein-design-mcp endpoint (in progress).

Data flow

Each task prompt is sent to your chosen LLM provider via their standard API (Anthropic, OpenAI, Google, DeepSeek) — that single channel is the only path by which task data leaves Romero Lab. The MCP server (reference or custom) only ever sees operational tool arguments (sequences, PDB paths, hotspot residues); it never sees the raw task prompt or evaluation criteria. Every task prompt also carries a unique 16-character canary token as an HTML comment, for retrospective leakage detection.

Bring your own tools (Custom MCP)

If you want to benchmark a new tool implementation (a faster structure predictor, a different diffusion backbone, your own stability model) against the same 76 tasks and rubric, stand up an HTTPS endpoint that satisfies the MCP contract and paste the URL into the submission form's Advanced: Custom MCP section:

• Contract + hosting options: leaderboard README
• Minimal FastAPI stub (~150 lines): example_mcp_server.py
• Reference implementation to fork: RomeroLab/protein-design-mcp

Limits

• Maximum 1 submission per calendar month per organization
• 73 hidden tasks are used for ranking; 3 public example tasks are available for development
• LLM-judge API costs are paid by Romero Lab; your own agent LLM calls are billed to your provider

Scoring rubric (100 points, hybrid)

Scores combine 72 algorithmic points from deterministic biophysical metrics with 28 LLM-judge points assessed by a 3-judge panel (PoLL) with self-exclusion to mitigate self-preference bias. Each component is capped at its rubric maximum to prevent double counting.

Approach (20 pts) — strategic appropriateness of tool selection across 10 functional categories (backbone generation, inverse folding, structure prediction, etc.).

Orchestration (15 pts) — pipeline ordering, intermediate validation, and adaptive iteration.

Quality (35 pts) — 100% algorithmic. Continuous 4-band interpolation over Boltz-2 re-prediction metrics (pLDDT, pTM, ipTM, i_pAE), eliminating LLM judgement variance on biophysical quantities.

Feasibility (15 pts) — valid amino acids, length constraints, composition, and biophysical plausibility.

Novelty (5 pts) — sequence identity to reference (lower identity = more novel).

Diversity (10 pts) — number and pairwise diversity of generated designs.

Five-layer contamination defense

Every evaluated LLM may have read protein-design literature during pretraining, so we use a layered defense:

• All 76 tasks derived from publications dated 2024–2026, post-dating model training cutoffs.
• Task prompts paraphrased and restructured — no verbatim passages from source literature.
• Targets specified by biological function and structural constraints, not by name or PDB identifier.
• 12 decoy tasks with deliberately fabricated targets to detect memorisation-based responses.
• n-gram overlap analysis between agent outputs and source publications — no verbatim regurgitation above the 8-gram threshold across any condition.

Citation

@article{biodesignbench2026,
  title={Evaluating LLM-Driven Protein Design:
         Agents Lack Iterative Evaluation Depth},
  author={Kim, Jeonghyeon and Romero, Philip},
  year={2026}
}