Benchmark Validation

The pipeline classifies 68 ClinVar-matched variants from 40 independent SETBP1+ profiles. Two concordance metrics are reported: the honest rate (80.9%) excludes PS1 evidence (same amino acid, different nucleotide change already in ClinVar), which would create circular validation. The PS1-inclusive rate (94.1%) includes this evidence for completeness. All downstream analysis uses the honest rate.

Classification breakdown across all 284 variants: 16 Pathogenic, 111 Likely Pathogenic, 27 VUS.

Each scoring tool is evaluated independently against ClinVar ground truth (68 variants with ClinVar annotations: 56 positive, 12 negative). CADD and PolyPhen-2 had zero coverage in the benchmark set and are omitted.

63 truncating variants (nonsense, frameshift, splice-site) were correctly classified using the PVS1 evidence code (8 Bayesian points each) 25

[25] S 2015

Standards and guidelines for the interpretation of sequence variants. Genet Med (2015)

PubMedDOI

. Truncating variants in known loss-of-function genes receive automatic Very Strong evidence for pathogenicity, making PVS1 the single most powerful evidence category in the ACMG framework.

56 unique missense variants were scored using ESM-2 (esm2_t33_650M_UR50D) 18

[18] Z 2023

Evolutionary-scale prediction of atomic-level protein structure with a language model. Science (2023)

PubMedDOI

on an RTX 4060 in 17 seconds. 80.4% were classified as pathogenic (PP3_Strong or higher) by masked marginal log-likelihood ratio scoring. The most pathogenic variant was SETBP1 S869R (LLR = -13.88), located in the SKI homology domain hotspot.

Known blind spot: ESM-2 evaluates single amino acid substitutions in protein context and cannot detect gain-of-function mechanisms that depend on protein-protein interaction changes rather than intrinsic protein destabilization. Hotspot variants with strong functional evidence from DMS assays may receive moderate ESM-2 scores.

140 of 154 variants (90.9%) in the first batch were annotated as Oncogenic or Likely Oncogenic by OncoKB 35

[35] D 2017

OncoKB: A Precision Oncology Knowledge Base. JCO Precis Oncol (2017)

PubMedDOI

. OncoKB provides clinical actionability levels (1 through 4, R1, R2) that integrate FDA approvals, NCCN guidelines, and clinical trial evidence. The high oncogenic annotation rate reflects the myeloid driver gene context of the benchmark set.

Ablation analysis removes one scoring axis at a time and checks whether the classification changes. 38.46% of 260 Pathogenic/LP variants are robust to single-axis removal. AlphaMissense 19

[19] J 2023

Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science (2023)

PubMedDOI

is the most critical axis: removing it causes 96% classification dependency (64 variants change class). Population frequency (Axis 5) shows the highest fragility count (97 variants), reflecting the PM2 evidence contribution from ultra-rare status.

Margin statistics across all 260 P/LP variants: mean margin 2.5 points, median 2, range 0 to 8. Variants at margin 0 are at the classification threshold and will change class if any single contributing axis is removed.

Four variants in CBL and EZH2 are classified as VUS in ClinVar but upgraded to Likely Pathogenic by the pipeline. In all four cases, domain-specific evidence (RING finger domain for CBL, SET domain for EZH2) and multiple concordant computational scores support the upgrade. These represent cases where the pipeline is arguably correct and ClinVar has not yet been updated with sufficient submitter evidence.

13 of 15 QA validation agents passed. Data leakage detection confirmed no circular evidence contamination (ClinVar classifications were not used as input features). Results integrity checks verified that all output files are reproducible from the input GENIE profiles. The two agents that flagged warnings identified edge cases in OncoKB annotation coverage and CADD score availability, neither of which affects the primary concordance metric.