Loading format mix...
BENCHMARK OVERVIEW
Experimental evidence. Measurable generalization.
A single experimental benchmark for antibody discovery and local sequence optimization, with five diagnostic tasks defined relative to the released training set.
Unique antibody-target records
Target coverage loading...
Four benchmark splits
Dataset breakdown · format and design task
Four benchmark classes with Test counts above Train counts.
eligible CDR groups
eligible CDR groups
eligible CDR groups
eligible CDR groups
Denovo asks whether a model can find binders across sequence families. Optimization asks whether it can rank related variants around a shared origin.
Two decisions, evaluated separately
One benchmark, two distinct antibody R&D decisions: discovering new binding families and prioritizing small improvements within an existing family.
Denovo Design
Tests whether a model can identify effective antibodies across distinct CDR families and on targets that may be absent from Train.
In Silico Optimization
Tests whether a model can resolve affinity changes caused by small mutations among highly similar sequences in the same CDR cluster.
L1–L5 describe the relationship to Train. They explain performance without changing the official ranking.
Five questions about what a model has learned
The original Denovo / Optimization benchmark is extended with five Train–Test generalization tasks. Official scores retain the same target-macro formula.
Known mutations
Same target and origin; every mutation has been observed in Train.
New mutations
Same target and origin; at least one mutation is absent from Train.
New starting points
Same target; the origin and sequence cluster are absent from Train.
New targets
The target is absent from Train; a related family member is present.
New target families
Neither the target nor its family is represented in Train.
Version history · V1 to V3

Compute metrics within each target, then macro-average targets. Overall weights Denovo and Optimization equally.
One official score, with chance performance removed
Compute metrics within each target, then macro-average targets. Overall weights Denovo and Optimization equally.
What is subtracted
AUC, Best Youden’s J and Best MCC each subtract the mean of that same metric over 100 fixed random submissions. Pearson remains unadjusted. Baselines are defined separately for each release, public/private board and diagnostic scope.
How scores combine
Class mean = [adjusted AUC + raw Pearson + (adjusted Best J + adjusted Best MCC) / 2] / 3.
Overall = (Denovo mean + Optimization mean) / 2. L1–L5 diagnostics do not add any weight to the official Overall.
How to read the result
Adjusted metrics can be negative. Random scores are near zero, not forced to zero: their Pearson component is retained. Independent target-specific baselines are also used in target-detail views. Subtracting a fixed baseline changes score interpretation, not the ordering of identical-scope submissions.
Coverage analysis · class balance and target overlap
Train/Test Split Analysis
V3 release composition and coverage. Aggregates combine the public and private evaluation sets without individual split assignments.
Rows by benchmark class
Released Test and reserved Train rows across four benchmark classes.
Target overlap
Unique and shared targets across Train and Test, with Denovo and Optimization separately.
Compare evaluation scope and experimental labels—not headline sample counts alone. Competitor figures retain the cited source snapshots.
Experimental Benchmark Comparison
ScoringBench V3 is smaller than the largest pooled screens, but provides hidden, traceable antibody affinity labels across both Denovo Design and In Silico Optimization.
| Dataset / study | Reported scale | Target scope | Labels | Primary evaluation |
|---|---|---|---|---|
| ScoringBench V3Antibody benchmark | Train: 2,531 labeled pairs Test: 1,000 answer-hidden pairs |
Train: 18 targets Test: 28 targets · 14 Test-only · 14 shared |
SPR/BLI -log10(KD) Bind / Not Bind |
Denovo + Optimization ranking L1–L5 generalization diagnostics |
| SKEMPI 2.0PPI mutation database | 7,085 entries 6,187 unique |
237 interactions 345 PDB entries |
Delta-Delta-G; kinetic and thermodynamic subsets; 440 no-binding entries | Mutation-effect prediction across heterogeneous PPI systems |
| Cao 2022Miniprotein screen | ~810K designs ~240K point mutants |
12 targets 14 target sites |
FACS binding, binder status and SC50 affinity proxy | De novo miniprotein discovery; no antibody optimization task |
| ProtDBenchUnified design benchmark | 478,391 non-binders 1,385 binders in Cao subset |
9 wet-lab targets 10 in-silico targets |
Binary wet-lab labels; verifier pass rate, throughput and structural diversity | Miniprotein generation and verifier bias under fixed compute |
| mBERVHH design campaign | 1,153,241 VHHs >100M measured interactions |
436 design targets 145 screened |
Multiplex binding enrichment and cross-target specificity | Massive VHH de novo screening; not a hidden per-sequence KD leaderboard |
| Nipah CompetitionProspective competition | 1,196 submitted designs 111 binders · 28 <10 nM |
1 target Nipah glycoprotein G |
Experimental KD, binding status and developability annotations | Denovo and Lead Optimization on one target |
| PromeraStructure and design model | 1,605 unseen complexes; Nipah 916/94; 69 VHH cognate pairs | Broad structure set binder labels concentrated in Nipah |
Structure accuracy, ipSAE/iCS; experimental and shuffled negatives | Co-folding and computational filtering, not independent KD ranking |
| Origin-1Full-length mAb platform | 3,242/63/84 Ab-Ag structures; <95 mAbs per prospective target | 10 zero-prior prospective targets |
SPR/BLI, selected KD, specificity, developability, function and cryo-EM | Strong prospective validation; limited external benchmark scale |
How ScoringBench V3 differs
Models submit a continuous affinity score from antibody and antigen sequences. Metrics are computed within targets and macro-averaged, so larger targets do not dominate merely by sample count.
Purposeful split: Denovo and Optimization are not collapsed into one task. Test-only targets measure target extrapolation; Train-shared targets and related CDR groups measure informed local optimization.
Leakage resistance: private Test labels, CDR-group-aware splitting and a mandatory exact VH-VL exclusion reduce memorization and duplicate-sequence leakage.
Scope notes and dependencies
ProtDBench reuses a nine-target subset of the Cao 2022 experimental screen. Promera's Nipah filtering analysis reuses part of the Nipah Competition data. These are shown separately because one is the source dataset and the other is the evaluation framework or model using it.
SKEMPI counts mutation records and PPI systems; mBER counts multiplex antibody-target interactions; Origin-1 reports structure-training entries and selected prospective molecules. Their headline numbers are therefore not directly interchangeable with unique antibody-target pairs.
Current ScoringBench V3 limitations
Target and antibody-format coverage remain uneven. Experimental batch variability and small target subsets can affect diagnostic estimates; the hidden holdout supports independent validation.
Experimental affinity evaluation
Benchmark
Download V3 Train and answer-free Test data. Submit all 1,000 continuous predictions and compare public-board scores.
Evaluation protocol
Test Data and Formal Metrics
Real SPR/BLI measurements are consolidated to unique antibody-target pairs under routine conditions before benchmark splitting.
- A board
- 594 rows · public ranking
- B board
- 406 rows · private scoring
- Denovo
- 402 total Test sequences
- Optimization
- 598 total Test sequences
- Test formats
- 748 mAb · 252 VHH
View target coverage
ActivinE, CCR8, CD226, CEACAM5, CXCR4, GPR75, HLA-B27, IL-33 mutant, IL11, IL13, IL17A, IL17F, IL18, IL1B, IL1R1, IL4R, KLK1, KLK2, KLK4, KLK5, KLK6, KLK7, MAPK8, OSM, PARVA, PIN1, TNFalpha, TSLP
Denovo Design
Cross-family and target-level generalization.
In Silico Optimization
Affinity discrimination within a related CDR neighborhood.
Task labels are evaluated relative to the released Train. Formal ranks use only Denovo and Optimization.- AUCcontinuous separation of Bind and Not Bind
- Pearsoncorrelation with experimental -log10(KD); Not Bind truth = 5
- Best Jbest target-level Youden's J after scanning continuous-score thresholds
- Best MCCbest target-level MCC after scanning continuous-score thresholds
Class mean = mean(adjusted AUC, raw Pearson, mean(adjusted Best J, adjusted Best MCC))
Overall score = mean of Denovo and Optimization means
Random-baseline subtraction: AUC, Best J and Best MCC subtract their own fixed 100-run random means; Pearson stays raw. All displayed means are calculated after subtraction. Negative adjusted scores are allowed.
Dataset Downloads
Both datasets are available to signed-in users. Each workbook has data and targets sheets; only Train contains affinity labels.
Submit Predictions
Submit all 1,000 Test predictions, then map the ID and continuous prediction columns. A-only submissions are not accepted. Standard names benchmark_id and predicted_neg_log10_KD are selected automatically when present.
Includes all A+B Test benchmark_id values with fictional, answer-independent predictions. Replace the prediction column and retain every row. Upload limit: 1 MB. Rapid repeated uploads are rate-limited.
Ready for CSV upload.
Restricted access
Leaderboard hidden
Log in to view rankings.
Leaderboard
Ranking by mean(AUC, Pearson, mean(Best J, Best MCC)) within denovo and optimization.
Official score: equal mean of Denovo and Optimization. L1–L5 are diagnostic only.
Final results · Cutoff: 2026-10-01 00:00 Beijing Time · 160 test rows · - submissions · - entries
| Rank | Model | Submitter | Overall score | Optimization mean | Denovo mean | Submitted |
|---|
Only aggregate scores are released. B-board answers and target-level results remain private.