Four Levers, One Ceiling: An Honest IMDB Study

Polaris trains every one of its from-scratch models through the same pipeline on IMDB sentiment (25,000 train / 25,000 test, seed 0). Instead of one headline number, the benchmark is a controlled study of four classic levers — and the result is more instructive than a win.

The four levers

1. A transformer encoder vs a mean-pooling baseline — they tie at ~85.5%. On a task whose signal is a handful of strong words, a small from-scratch transformer has no edge over a bag of embeddings, and it overfits more at ~14× the cost.
2. Subword tokenization (BPE) — slightly hurts (≈0.856 → 0.839). Sentiment lives in common whole words; BPE fragments them and lengthens sequences, so more of each review is lost to truncation.
3. Pretrained GloVe embeddings — no movement (+0.001 for the pooling model; the transformer overfits harder and slips). Pretrained word vectors help when labels are scarce; with 25k labeled reviews they're redundant.
4. MLM pretraining (controlled ablation, identical vocab and architecture) — a real head start (epoch-1 validation 0.810 vs 0.736, higher best validation) that converges to the same ~86% test ceiling, because labels aren't scarce and the ~11M-word in-domain pretraining corpus adds little the labels can't teach.

The lesson

All four levers land at ~85–86%. The ceiling belongs to the task and the data/compute regime, not to any single component — breaking it would take large-external-corpus pretraining beyond laptop scale. A benchmark that can explain its plateau is worth more than one that reports a cherry-picked win, and getting that explanation required controlling every layer of the stack — which is exactly why the stack is built from scratch.