Engineering Notes
Working notes from building ML systems from scratch. Each note explains one mechanism — how it works, why it exists, and what implementing it taught me.
The Cross-Validation Bug Worth 26 Points of Fake Accuracy
Michi warned users about group leakage while its own benchmark command ignored groups entirely. How entity leakage inflates scores, and why it had to be fixed before the API freeze.
Michi · 2 min read
When Your Reranker Is Worse Than Random
Meridian's cross-encoder pushed Recall@5 to 0.029 — below random ordering. Diagnosing the collapse, and why fallible pipeline stages must degrade gracefully.
Meridian · 2 min read
Greedy-Exact Speculative Decoding
How Zenith uses a small draft model to cut target-model forward passes by 3×+ while producing output identical to greedy decoding.
Zenith · 1 min read
Weight Tying in Language Models
Sharing the input embedding and output projection — why it works, when it stops working, and how it interacts with logit scale.
Zenith · 2 min read
Four Levers, One Ceiling: An Honest IMDB Study
What happened when Polaris pulled four classic NLP levers — a transformer, subwords, GloVe, and MLM pretraining — and every one bounced off ~86%.
Polaris · 1 min read
RMSNorm: What LayerNorm's Mean Was (Not) Doing
Why modern LLMs dropped mean-centering from LayerNorm, and what benchmarking the two in Zenith showed.
Zenith · 2 min read
RoPE, Explained From the Implementation Up
Rotary position embeddings as rotations of query/key pairs — and the training/inference consistency bug that taught me how they really work.
Zenith · 2 min read
Understanding Byte-Pair Encoding
What BPE actually does, why every modern LLM uses a variant of it, and what implementing a tokenizer from scratch taught me.
Polaris · Zenith · 2 min read