Meridian

pip install meridian-rag

Overview

Meridian is the capstone of the from-scratch stack: a grounded retrieval-augmented generation engine over PubMed abstracts in ~11K LOC of Python/PyTorch, where every ML component is trained in-house — a BPE tokenizer, a dense bi-encoder retriever trained with InfoNCE, IVF and HNSW approximate-nearest-neighbour indexes (no FAISS), a cross-encoder reranker, and a 3-class NLI faithfulness verifier. BM25, the serving layer, and the entire evaluation harness are hand-built too. No third-party pre-trained models anywhere: the encoder side comes from Polaris and generation from Zenith — frameworks I also built, pinned as dependencies.

The design rule is absolute: every answer is cited, verified, or refused. Generation is citation-constrained, every claim sentence is checked for entailment against its source, and the system abstains rather than guess. It is a research literature assistant — explicitly not medical advice.

Problem

Language models answer biomedical questions fluently whether or not they are right. In a domain where an unsourced answer is worthless, the interesting engineering is not generation quality — it is grounding: retrieval you can trust, citations per sentence, an entailment check on every claim, and calibrated abstention when confidence is low. Meridian builds that pipeline end to end to understand every part of it.

Architecture

The full path: BM25 → dense retrieval → ANN index → cross-encoder rerank → confidence gates → citation-constrained generation → NLI verification → abstention.

• Retrieval — from-scratch BM25 Okapi over an inverted index, plus a dense bi-encoder trained with InfoNCE contrastive learning, fused via reciprocal-rank fusion.
• ANN indexes — IVF (k-means++ coarse quantizer) and HNSW, both implemented from scratch. HNSW reaches recall@10 0.996 at 0.262 ms — faster than exact brute-force search at half IVF's index-memory overhead.
• Verification — a from-scratch 3-class NLI verifier trained on 942K SNLI+MultiNLI pairs (MLM pretraining over 1.15M sentences), reaching 78.3% on SNLI dev against a 33.3% chance floor.
• Calibration — a retrieval-margin confidence gate achieving 80.1% coverage at a 0.000 error rate among answered queries: the system would rather refuse than be wrong.
• Serving — FastAPI + SQLite, profiled per stage: BM25 search costs 1.33 ms P50; cross-encoder reranking costs 421 ms (~320×), which is why reranking is disabled by default — a measured architecture decision.

Measured results

Every number is produced by a committed, seeded script; eval splits are frozen and checksum-guarded so training on them breaks CI.

| Result | Value | | --- | --- | | BM25 retrieval (PubMedQA dev) | R@5 0.987, nDCG@10 0.974 | | Dense retriever (4 seeds) | R@5 0.010 → 0.38 ± 0.02 (~38× via supervised contrastive training) | | Reranker fix | R@5 0.029 (pure, below random) → 0.983 (base-fused) | | HNSW ANN | recall@10 0.996 @ 0.262 ms | | NLI verifier | 41.5% → 78.3% SNLI dev (+88% relative) | | Calibration | 80.1% coverage @ 0.000 error | | Delivery | 254 tests, 95.7% coverage, mypy --strict, 12 releases to v1.0.0 |

The honest findings

Meridian's benchmarks publish the results that disproved my own hypotheses, and they are the most instructive part of the project:

• BM25 beats the from-scratch dense retriever (0.987 vs 0.38 R@5). PubMedQA questions reuse their source abstract's vocabulary, so lexical matching is near-ceiling. I built a baseline strong enough that my neural model couldn't beat it, and reported that.
• MLM pretraining contributed nothing measurable to retrieval. A seed-averaged ablation (0.371 ± 0.022 vs 0.382 ± 0.023) overturned a claim I had already written into my own docs — supervised training pairs were the real lever. I corrected the documentation.
• The verifier doesn't transfer to biomedical prose. 78.3% on general-English SNLI, but a 0.437 flag rate on verbatim-quoted biomedical text — a measured domain-transfer gap (the fix, SciNLI adaptation, is scoped but not yet run).
• The learning rate, not capacity, was the verifier's binding constraint. A larger model scored worse (0.607) with higher training loss — an optimization failure. Dropping lr from 1e-3 to 3e-4 on the identical architecture gave +17.6 points.

Engineering discipline

Built in 12 strictly ordered vertical slices (meridian ask answered end-to-end from v0.1.0), with 8 ADRs and 16 phase design docs. 254 tests at 95.7% coverage, mypy --strict across 155 files, Black + Ruff in CI (Python 3.12/3.13 × Ubuntu/macOS), and a house rule that no metric appears in the README unless a committed script reproduces it. Honest scope: production-inspired, laptop-scale — a 1,000-abstract corpus and ~10–125M-parameter models.

Roadmap

SciNLI domain adaptation for the verifier, the ~200K-abstract PubMed corpus, and training the cited-answer generator at scale — all wired and documented, gated on compute.