🧬 BioDesignBench

DeepSeek V3

GPT-5

Claude Sonnet 4.5

Gemini 2.5 Pro

Causal interventions on the depth gap

Causal intervention experiments on the evaluation-depth gap. 18 representative tasks rerun under three conditions: baseline (no intervention), forced_depth (mandate ≥3 evaluation metric categories per candidate), and low_variety_control (compute-matched control restricted to a narrow range of evaluation metrics). Reruns are scored on a representative 18-task subset that spans all 9 occupied taxonomy cells.

• Your API key is stored on the submission row only between submission and dispatch, then scrubbed automatically regardless of whether the run succeeded.
• Each task carries a unique 16-character canary token (invisible HTML comment) so we can retrospectively detect leakage in published models.
• The MCP server (reference or custom) sees only operational tool arguments, never the raw task description or evaluation criteria.

• Reference (default): your agent uses our hosted protein-design-mcp endpoint. Eligible for the reference ranking.
• Custom: provide your own public MCP URL implementing the same 17-tool schema. Useful for benchmarking new tool implementations against an identical model under identical task prompts. Tagged with a custom badge.

What is BioDesignBench?

BioDesignBench is a benchmark for evaluating LLM agents as orchestrators of multi-step stochastic protein-design pipelines. Unlike chemistry- or code-agent benchmarks, where tool chains are largely deterministic, protein design demands repeated sampling from generative tools (RFdiffusion, ProteinMPNN) and iterative cross-validation through several biophysical metrics. We test the full agentic loop — plan → call → evaluate → iterate — over 76 expert-curated tasks drawn from 2024–2026 literature, exposed through 17 MCP-integrated tools.

Three principal findings

1. Top-tier agents now beat the hardcoded pipeline

DeepSeek V3 and GPT-5 surpass the deterministic hardcoded pipeline (54.5) under both modes. Autonomous protein-design orchestration is no longer infeasible — but a substantial gap to the human expert (61.7) and oracle (75.2) remains.

2. Coverage–depth dissociation

Workflow guidance closes the coverage gap, bringing agent tool selection closer to the human expert, but leaves evaluation depth unchanged. Better tool documentation can teach agents which tools to call, but not how thoroughly to use them on each generated candidate.

3. Evaluation variety, not tool knowledge, is the bottleneck

Across 836 task–condition observations, the number of distinct evaluation metric categories per candidate correlates with total score at ρ = 0.68 (p < 10^-115). LLM agents generate backbone candidates at expert-level rates but invoke scoring tools at only ~14% of expert intensity. Forced-depth interventions confirm this is causal — see the Depth Gap tab.

How to submit

Unlike most agent benchmarks, you do not host an HTTP endpoint. The 76 task descriptions never leave Romero Lab infrastructure. Instead you provide:

1. an LLM provider + API key (Anthropic / OpenAI / Google / DeepSeek). We run the BioDesignBench agent loop against your chosen model inside the leaderboard backend. Your key is scrubbed from our records immediately after the dispatch phase completes.
2. optionally, a custom MCP URL if you want to evaluate your own tool implementations. Otherwise, the agent calls our reference protein-design-mcp endpoint (in progress).

Data flow

Each task prompt is sent to your chosen LLM provider via their standard API (Anthropic, OpenAI, Google, DeepSeek) — that single channel is the only path by which task data leaves Romero Lab. The MCP server (reference or custom) only ever sees operational tool arguments (sequences, PDB paths, hotspot residues); it never sees the raw task prompt or evaluation criteria. Every task prompt also carries a unique 16-character canary token as an HTML comment, for retrospective leakage detection.

Bring your own tools (Custom MCP)

If you want to benchmark a new tool implementation (a faster structure predictor, a different diffusion backbone, your own stability model) against the same 76 tasks and rubric, stand up an HTTPS endpoint that satisfies the MCP contract and paste the URL into the submission form's Advanced: Custom MCP section:

• Contract + hosting options: leaderboard README
• Minimal FastAPI stub (~150 lines): example_mcp_server.py
• Reference implementation to fork: jasonkim8652/protein-design-mcp

Limits

• Maximum 1 submission per calendar month per organization
• 73 hidden tasks are used for ranking; 3 public example tasks are available for development
• LLM-judge API costs are paid by Romero Lab; your own agent LLM calls are billed to your provider

Scoring rubric (100 points, hybrid)

Scores combine 72 algorithmic points from deterministic biophysical metrics with 28 LLM-judge points assessed by a 3-judge panel (PoLL) with self-exclusion to mitigate self-preference bias. Each component is capped at its rubric maximum to prevent double counting.

Approach (20 pts) — strategic appropriateness of tool selection across 10 functional categories (backbone generation, inverse folding, structure prediction, etc.).

Orchestration (15 pts) — pipeline ordering, intermediate validation, and adaptive iteration.

Quality (35 pts) — 100% algorithmic. Continuous four-band interpolation over Boltz-2 re-prediction metrics (pLDDT, pTM, ipTM, ipAE), eliminating LLM judgement variance on biophysical quantities.

Feasibility (15 pts) — valid amino acids, length constraints, composition, and biophysical plausibility.

Novelty (5 pts) — sequence identity to reference (lower identity = more novel).

Diversity (10 pts) — number and pairwise diversity of generated designs.

Five-layer contamination defense

Every evaluated LLM may have read protein-design literature during pretraining, so we use a layered defense:

• All 76 tasks derived from publications dated 2024–2026, post-dating model training cutoffs.
• Task prompts paraphrased and restructured — no verbatim passages from source literature.
• Targets specified by biological function and structural constraints, not by name or PDB identifier.
• 12 decoy tasks with deliberately fabricated targets to detect memorisation-based responses.
• n-gram overlap analysis between agent outputs and source publications — no verbatim regurgitation above the 8-gram threshold across any condition.

Citation

@article{biodesignbench2026,
  title={Evaluating LLM-Driven Protein Design:
         Agents Lack Iterative Evaluation Depth},
  author={Kim, Jeonghyeon and Romero, Philip},
  journal={bioRxiv},
  year={2026},
  doi={10.64898/2026.05.06.723381},
  url={https://www.biorxiv.org/content/10.64898/2026.05.06.723381v1}
}