Benchmark Validation
40 SETBP1+ profiles, 284 variants, ClinVar concordance
Profiles Tested
40
SETBP1+ myeloid patients from GENIE v19.0
Variants Classified
284
Batch 1: 154, Batch 2: 130
Honest Concordance PS1-free
74.6%
103/138 ClinVar-matched variants
False Positives Zero
0
No benign variants misclassified as pathogenic
ClinVar Concordance
The pipeline classifies 284 variants from 40 independent SETBP1+ profiles;
138 of them carry a ClinVar classification.
Two concordance metrics are reported: the honest rate (74.6%) excludes
PS1 evidence (same amino acid, different nucleotide change already in ClinVar), which
would create circular validation. The PS1-inclusive rate (89.9%) includes
this evidence for completeness. All downstream analysis uses the honest rate.
| Metric | Concordant | Total | Rate |
|---|---|---|---|
| Honest (PS1-free) | 103 | 138 | 74.6% |
| With PS1 (circular) | 124 | 138 | 89.9% |
Classification breakdown across all 284 variants, without PS1: 101 Pathogenic,
120 Likely Pathogenic,
63 VUS.
Source: honest_concordance_40.json (verify_headline_numbers.py).
ClinVar classes of the 138 variants: 114 pathogenic/likely pathogenic,
21 VUS, 2 other, 1 likely benign.
Per-Axis Tool Performance
Each scoring tool is evaluated independently against ClinVar ground truth in
batch 1 (20 profiles, 154 variants;
68 with ClinVar annotations:
56 positive, 12 negative).
CADD and PolyPhen-2 had no scores in this batch-1 per-axis run and are omitted; the later
v5 runs scored CADD for 201 and PolyPhen-2 for 164 of the 284 variants.
| Tool | Threshold | Coverage | Sensitivity | Specificity | PPV | Accuracy |
|---|---|---|---|---|---|---|
| AlphaMissense | ≥0.564 | 61/68 | 92.2% | 20.0% | 85.5% | 80.3% |
| EVE | ≥0.5 | 47/68 | 87.8% | 33.3% | 90.0% | 80.8% |
| REVEL | ≥0.5 | 60/68 | 79.6% | 45.5% | 86.7% | 73.3% |
| SIFT | ≤0.05 | 60/68 | 98.0% | 45.5% | 88.9% | 88.3% |
Source: per_axis_performance.json (batch 1 only).
Generated 2026-03-28.
PVS1: Truncating Variants
107 truncating variants (frameshift, nonsense, splice-site, one start-loss) across all 40 profiles (63 in batch 1,
44 in batch 2) received the PVS1 evidence code 25
[25] S 2015
Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med (2015)
:
100 were classified Pathogenic and 7 Likely Pathogenic.
PVS1 gave 8 Bayesian points in all but one case (a start-loss variant at Moderate, 2 points).
Truncating variants in known loss-of-function genes receive automatic Very Strong
evidence for pathogenicity, making PVS1 the single most powerful evidence category
in the ACMG framework.
| Evidence Code | Variants | Bayesian Points | Classification |
|---|---|---|---|
| PVS1 (truncating) | 107 | 8 each (1 at 2) | Pathogenic / Likely Pathogenic |
Source: Richards et al. (2015) 25
[25] S 2015
Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med (2015)
.
Tavtigian et al. (2020) 26[26] SV 2020
Fitting a naturally scaled point system to the ACMG/AMP variant classification guidelines. Hum Mutat (2020)
quantitative framework.
ESM-2 Missense Scoring
In batch 1 (20 profiles), 56 unique missense variants were scored using ESM-2
(esm2_t33_650M_UR50D) 18
[18] Z 2023
Evolutionary-scale prediction of atomic-level protein structure with a language model. Science (2023)
on an RTX 4060 in 17 seconds.
80.4% (45/56) reached PP3 evidence at any strength by masked marginal log-likelihood
ratio scoring; 46.4% (26/56) reached PP3_Strong. In batch 2, 36 of 52 unique missense
variants (69.2%) reached any PP3 level and 24 PP3_Strong. These ESM-2 scores were not
used in the benchmark classification. The most pathogenic variant was SETBP1 S869R (LLR = -13.88), located in the SKI
homology domain hotspot.
Missense Scored
56
Unique missense variants, batch 1
Pathogenic Rate
80.4%
Any PP3 level, batch 1 (PP3_Strong: 46.4%)
Most Pathogenic
SETBP1 S869R
LLR = -13.88
Runtime
17s
RTX 4060 (8 GB VRAM)
Known blind spot: ESM-2 evaluates single amino acid substitutions in protein context
and cannot detect gain-of-function mechanisms that depend on protein-protein interaction
changes rather than intrinsic protein destabilization. Hotspot variants with strong
functional evidence from DMS assays may receive moderate ESM-2 scores.
OncoKB Annotation
140 of 154 variants (90.9%) in the first batch were annotated as Oncogenic or Likely
Oncogenic using OncoKB public data 35
[35] D 2017
OncoKB: A Precision Oncology Knowledge Base. JCO Precis Oncol (2017)
(gene-level
data, curated hotspots and OncoKB inference rules; no per-variant API calls). OncoKB provides clinical
actionability levels (1 through 4, R1, R2) that integrate FDA approvals, NCCN guidelines,
and clinical trial evidence. The high oncogenic annotation rate reflects the myeloid
driver gene context of the benchmark set.
| OncoKB Annotation | Count | Fraction |
|---|---|---|
| Oncogenic / Likely Oncogenic | 140 | 90.9% (140/154) |
| Inconclusive (7), Unknown (6), Predicted Oncogenic (1) | 14 | 9.1% (14/154) |
Ablation Robustness
Ablation analysis removes one scoring axis at a time and checks whether the
classification changes. 45.87% of 242 Pathogenic/LP variants
are robust to single-axis removal. Removing AlphaMissense 19
[19] J 2023
Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science (2023)
(Axis 2) changes the class of 24 of 242 P/LP variants
(9.9%). Population frequency (Axis 5) shows the highest
fragility count of the six axes (118 variants), reflecting
the PM2 evidence contribution from ultra-rare status. Removing the OncoKB-based PP5
evidence, which is outside the six axes, changes 175.
| Axis Removed | Fragile Variants | Impact |
|---|---|---|
| Axis 1: Protein LM (ESM-2) | 0 | No ESM-2 scores in this run (empty for all 284), so no effect |
| Axis 2: Structure/DL (AlphaMissense) | 24 | 9.9% of P/LP |
| Axis 3: Conservation (EVE, SIFT) | 10 | Moderate impact |
| Axis 4: Meta-Ensemble (CADD, REVEL) | 14 | Moderate impact |
| Axis 5: Population (gnomAD) | 118 | Highest fragility count |
| Axis 6: Functional (PolyPhen-2 in this run) | 9 | Moderate impact |
Margin statistics across all 242 P/LP variants: mean margin 1.3 points,
median 0, range 0 to 7.
Variants at margin 0 are at the classification threshold and will change class if any single
contributing axis is removed.
Discordant Cases
Without PS1, 35 of the 138 ClinVar-classified
variants are discordant. In 12 the
pipeline classifies higher than ClinVar; in 21
a ClinVar-pathogenic variant is called VUS by the pipeline
(CBL R420Q, CBL Y371H, CSF3R T618I, DNMT3A R736H, EZH2 R685H, KRAS A146P, KRAS A146T, NRAS G12S, NRAS G13D, PTPN11 A72T, PTPN11 A72V, PTPN11 N308S, SETBP1 G870C, TP53 R248Q).
Four variants in CBL and EZH2 are classified as VUS in ClinVar but upgraded to Pathogenic (10 points without PS1) by the pipeline. In all four cases the upgrade comes from concordant computational scores; the variants lie in the CBL RING finger and EZH2 SET domains, but no PM1 domain points were assigned in the benchmark run. These represent cases where the pipeline is arguably correct and ClinVar has not yet been updated with sufficient submitter evidence.
Four variants in CBL and EZH2 are classified as VUS in ClinVar but upgraded to Pathogenic (10 points without PS1) by the pipeline. In all four cases the upgrade comes from concordant computational scores; the variants lie in the CBL RING finger and EZH2 SET domains, but no PM1 domain points were assigned in the benchmark run. These represent cases where the pipeline is arguably correct and ClinVar has not yet been updated with sufficient submitter evidence.
| Gene | Variant | ClinVar | Pipeline | Rationale |
|---|---|---|---|---|
| CBL | C404Y | VUS | Pathogenic | RING domain, high AlphaMissense/REVEL |
| CBL | C381Y | VUS | Pathogenic | RING domain, high AlphaMissense/EVE |
| CBL | G415C | VUS | Pathogenic | RING domain, concordant scores |
| EZH2 | R679H | VUS | Pathogenic | SET domain, high EVE/AlphaMissense/REVEL |
Source: Benchmark QA validation: 13/15 agents passed, data leakage PASS,
results integrity PASS.
Quality Assurance
13 of 15 QA validation agents passed. Data leakage detection confirmed no circular
evidence contamination (ClinVar classifications were not used as input features).
Results integrity checks verified that the batch-1 (20 profiles) output files are
reproducible from the input GENIE profiles. The two agents that flagged warnings identified edge cases
in OncoKB annotation coverage and CADD score availability, neither of which affects
the primary concordance metric.
QA Agents Passed
13/15
Data leakage PASS, results integrity PASS
PS1 Variants
114
ClinVar-confirmed pathogenic (used only in PS1-inclusive rate)
PM1 Hotspot
Not assigned
PM1 = 0 points for all 284 benchmark variants
Source: qa_benchmark_leakage.py, qa_clinvar_concordance.py,
qa_manual_crosscheck.py, qa_oncokb_benchmark.py, qa_pvs1_check.py, qa_verify_benchmark_profiles.py.
References
- Richards S, Aziz N, Bale S, et al. Standards and guidelines for the interpretation of sequence variants. Genet Med (2015). PubMed
- Tavtigian SV, Harrison SM, Boucher KM, Biesecker LG. Fitting a naturally scaled point system to the ACMG/AMP variant classification guidelines. Hum Mutat (2020). PubMed
- Lin Z, Akin H, Rao R, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science (2023). DOI
- Cheng J, Novati G, Pan J, et al. Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science (2023). DOI
- Chakravarty D, Gao J, Phillips SM, et al. OncoKB: A Precision Oncology Knowledge Base. JCO Precis Oncol (2017). PubMed
- AACR Project GENIE Consortium. AACR Project GENIE: Powering Precision Medicine through an International Consortium. Cancer Discov (2017). DOI