Writing
Engineering Notes
Write-ups on the parts of the stack I've implemented myself — tokenization, transformer internals, retrieval, inference, and evaluation. Each one comes out of a real project decision or a measured result.
Subscribe via RSS8 notes
The Cross-Validation Bug Worth 26 Points of Fake Accuracy
Michi warned users about group leakage while its own benchmark command ignored groups entirely. How entity leakage inflates scores, and why it had to be fixed before the API freeze.
When Your Reranker Is Worse Than Random
Meridian's cross-encoder pushed Recall@5 to 0.029 — below random ordering. Diagnosing the collapse, and why fallible pipeline stages must degrade gracefully.
Greedy-Exact Speculative Decoding
How Zenith uses a small draft model to cut target-model forward passes by 3×+ while producing output identical to greedy decoding.
Weight Tying in Language Models
Sharing the input embedding and output projection — why it works, when it stops working, and how it interacts with logit scale.
Four Levers, One Ceiling: An Honest IMDB Study
What happened when Polaris pulled four classic NLP levers — a transformer, subwords, GloVe, and MLM pretraining — and every one bounced off ~86%.
RMSNorm: What LayerNorm's Mean Was (Not) Doing
Why modern LLMs dropped mean-centering from LayerNorm, and what benchmarking the two in Zenith showed.
RoPE, Explained From the Implementation Up
Rotary position embeddings as rotations of query/key pairs — and the training/inference consistency bug that taught me how they really work.
Understanding Byte-Pair Encoding
What BPE actually does, why every modern LLM uses a variant of it, and what implementing a tokenizer from scratch taught me.