Skip to content

2 min readZenith

RMSNorm: What LayerNorm's Mean Was (Not) Doing

Why modern LLMs dropped mean-centering from LayerNorm, and what benchmarking the two in Zenith showed.

transformersnormalization

Nearly every modern decoder-only model — LLaMA, Mistral, Gemma — uses RMSNorm instead of LayerNorm. The change looks cosmetic. It isn't, and implementing both in Zenith made the difference concrete.

The two normalizations

LayerNorm centers and scales:

y = (x - x.mean(-1, keepdim=True)) / torch.sqrt(x.var(-1, keepdim=True) + eps)
y = y * gamma + beta

RMSNorm only scales, by the root-mean-square, and drops the bias:

y = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * gamma

No mean subtraction, no beta. Fewer operations, fewer parameters, one fewer reduction over the hidden dimension.

Why dropping the mean is fine

The RMSNorm paper's observation is that LayerNorm's benefit comes almost entirely from re-scaling invariance, not re-centering. In a pre-norm transformer, the residual stream's mean carries little useful signal that the subsequent linear layer couldn't absorb anyway. Empirically, quality is on par; what changes is cost — one reduction instead of two, on the critical path of every block at every token.

Zenith supports both styles behind one config switch, which makes the comparison concrete. In its measured architecture ablation on tiny-shakespeare — same recipe, same ~10.7M parameters — the Llama-style stack (RMSNorm + RoPE + SwiGLU) reached 2.08 bits/char vs 2.11 for the GPT-2-style stack (LayerNorm + learned positions + GELU), and converged roughly twice as fast (best validation at epoch 10 vs 17). The honest caveat: that's the whole architecture bundle, not RMSNorm in isolation — and at this scale the quality floor is set by data, so the modern stack mostly buys convergence speed.

Implementation notes

  • Epsilon placement matters. rsqrt(mean(x²) + eps) is not the same as 1/(rms + eps) — match the reference or your checkpoints won't transfer.
  • Compute the reduction carefully in half precision. The squared-mean reduction is where fp16 goes wrong at the tails; the standard practice (as in LLaMA's reference) is to upcast, normalize, and downcast.
  • No beta means the next layer's bias does the work — or, in most modern configs, nothing does, because biases are dropped there too.

The takeaway

RMSNorm is a case study in subtractive design: identify which half of an operation carries the value, delete the other half, and bank the savings at every token. Most "architecture improvements" in modern LLMs look like this — small, multiplicative, and only visible if you implement the stack yourself.