AI engineering · Competition
The Knee Ledger.
Teaching one machine to read knee MRIs against 2,300 teams in the RSNA AI Challenge — with a rule that every claim needs a receipt, every experiment states its pass-bar before it runs, and nothing ships without clearing an automated gate.
- 0.937
- Best public score
- Top 3%
- Main leaderboard
- ~20 min
- Efficiency-entry runtime
- 50+
- Gated experiments
The problem
Twelve findings, fifty-eight answers.
A knee MRI arrives as a stack of cross-sections in three planes. The model must score twelve abnormalities per exam — ligament tears, meniscal damage, arthritis in three compartments, fluid, fractures — scored by macro-averaged AUC across all twelve.
The defining constraint: of 4,407 training exams, only 58 carry expert radiologist labels. Everything else is machine-extracted from written radiology reports in ten languages. The competition is less about bigger models than about squeezing truth out of noisy supervision — and measuring honestly enough not to fool yourself, when your entire validation set fits in a waiting room.
The climb
Every rung, gated and receipted.
Public leaderboard · Macro-AUC · Aug 06 → Aug 24
Ten scored submissions, each testing exactly one structural change. Two early slots burned on silent failures became the reason every later submission passes a hard automated gate first — the gate has since blocked six bad submissions at a cost of zero.
The approach
Three pillars, one shared GPU.
Evidence discipline
Every experiment declares its hypothesis and kill-band before running. Every result — including the ~40 that failed — lives in a ledger with a verdict. Noise bands gate every claim, so a lucky wiggle never gets promoted to a finding.
The model
A large vision transformer adapted to knee anatomy using 1,250 externally sourced, expert-labeled MRIs — supervision most of the field never touched — plus scan physics recovered from DICOM headers, and knowledge distilled from the community's strongest public ensembles.
The machine
Inference engineered for the efficiency prize: both scoring GPUs in parallel, decoding hidden behind computation, cold-start halved. A full 1,300-exam evaluation runs in about twenty minutes while comparable-accuracy rivals take hours. Deploys drive themselves: push, verify on-image, gate, submit, watch.
Field notes
What the gates caught.
- CAUGHT
A celebrated public notebook’s "0.906 configuration" turned out to be a hardcoded three-row stub that could never fire on the hidden test set. Verified before we bet a submission on it.
- CAUGHT
Our 58-exam gold standard was silently memorized by public models that trained on it — inflating offline scores by 7× the real gain. Quantified, ledgered, and the metric retired for that lineage.
- CAUGHT
A "grand master" model bundle with impressive naming and gigabytes of weights: the author’s actual leaderboard best was 0.780. Refuted from public records in minutes.
- BURNED
Two early submissions scored 0.500 — one hardcoded tensor dimension, one filename mismatch — while their logs plainly said so. Both post-mortems became permanent automated gates; neither failure mode has recurred.
- CAUGHT
The efficiency leaderboard only scores explicitly selected submissions — a stale checkbox graded a week-old model at rank 870 while the real one sat unscored. One selection later: rank 136.
Status
Live, and still climbing.
The campaign runs continuously: an hourly loop harvests results, logs the leaderboards, and scans the community for newly published models worth converting — because in a field this fast, standing still costs about ten ranks a day. As of late August the main entry stands at 0.937 — top 3% of 2,324 teams — and that pipeline's ensemble is being distilled into the twenty-minute model to push the efficiency entry deeper into the top 100.
A note on timing: this page tells the story; the full technical write-up — architectures, training recipes, and the complete experiment ledger — publishes after the competition closes on October 22, per good sportsmanship and the open-source winners' obligations.
One RTX 4090 · PyTorch · DINOv2 · a ledger