Batch Size Beats Model Size: The Lesson from Training mentee-embed-v2 and v3

When MenteE AI published mentee-embed-v1 the most common question was: what happens if you make the model bigger? We answered it empirically with v2. The answer was: performance drops. This post explains why, and what v3 proves instead.

The v2 experiment: 125M, batch forced to 128

v2 increased the Transformer hidden dim to match a 125M parameter count — a 3× scale-up. On paper this should win: more capacity, more representation power. In practice, 125M parameters consume significantly more VRAM, which forced our training batch size down from 192 (v1) to 128 (v2). The results:

• Protocol A avg MRR@10: 0.585 (v1) → 0.429 ↓ (v2)
• Protocol C MRR@10: ~0.20 (v1) → 0.215 (v2) — barely moved

Bigger model, worse retrieval. That is not noise — it is a systematic signal.

Why batch size matters more than parameters in contrastive training

In InfoNCE-style contrastive learning, each training step asks the model to rank one positive against N−1 in-batch negatives. With batch size 128, N−1 = 127. With batch size 512, N−1 = 511. More negatives per step means:

• Harder gradient signal — easy negatives don't contribute meaningful loss
• Better calibration — the model must rank against a diverse, large pool every step
• More efficient training — each GPU forward pass does the work of ~4× smaller batches

Our relational distillation loss compounds this: we compute a full batch × batch cosine similarity matrix and MSE it against the teacher's. At batch 128 that is 16,384 similarity pairs. At batch 512 that is 262,144 — 16× more dense supervision per step. The 125M model simply couldn't access that regime because VRAM was exhausted by its parameters.

The v3 solution: smaller model, bigger batch

v3 returned to 41M parameters — freeing enough VRAM to run batch 512. Add 2.1M triplets (vs ~810K in v1), two distillation rounds, and 700K MS-MARCO passages, and the results speak clearly:

The 3× Protocol C jump happened while the model got smaller. That isolates batch size and MS-MARCO data as the two causal factors — not model capacity.

Practical rule for from-scratch embedding training

Under a fixed VRAM budget: pick the smallest model that fits your downstream latency requirement, then maximize batch size with the freed memory. Add in-domain retrieval data (MS-MARCO or domain-specific pairs) before scaling parameters. Our v1→v2→v3 ablation is a controlled experiment that demonstrates this on a real multilingual task.

What's next

The next question is whether Arabic and Urdu Protocol B (Wikipedia corpus retrieval) can be improved with domain-matched multilingual retrieval data. We have the pipeline — it is a data question now. Full numbers and reproduction steps at /research. Model at MenteEAI/mentee-embed-v3. Preprint: doi:10.5281/zenodo.22117673.