Why We Train From Scratch Instead of Fine-Tuning: The MenteE Embed Philosophy

Every major embedding model — MiniLM, E5, BGE, mpnet — fine-tunes a pretrained backbone. MenteE Embed trains from random initialization. This is harder, slower, and produces smaller numbers on English benchmarks. We chose it anyway. Here is why.

The fine-tuning trap

Fine-tuning gives you strong English performance for free — the backbone already knows English. But it also means your model inherits the backbone's biases: English-heavy vocabulary, limited Arabic/Urdu morphology, and a fixed architecture that may not fit your latency budget. When we tested MiniLM-L6-v2 on Arabic, it scored 0.084 on dialect — effectively random despite being "pretrained on 1B+ pairs."

What training from scratch gives us

• Custom tokenizer — 50K BPE trained on our trilingual corpus, giving Arabic and Urdu fair vocabulary share
• Controlled architecture — 41M params, 384-dim, 512-token context — exactly what we need, nothing more
• Language parity — no English head start means the model must learn all three languages equally
• Full reproducibility — random seed → trained model, no pretrained dependencies

The cost

Training from scratch requires more data, more steps, and careful bootstrap. Our two-stage recipe (MLM → relational distillation) solves the collapse problem that kills most from-scratch attempts. Stage A gives language awareness; Stage B gives retrieval geometry. The v1→v4 progression proves the method scales.

The result

v4 scores 0.874 MIRACL AR from random init — approaching pretrained mpnet (0.898) at 6.8× fewer parameters. That is not a coincidence — it is what happens when you design the model for the task instead of adapting someone else's model to the task.

Read more

Full training details at /research. Code: github.com/MenteE-s/mentee-embeddings. Model: MenteEAI/mentee-embed-v4.