MenteE AI · Technical Report
mentee-embed-v1
A 41M-parameter multilingual embedding model for Arabic, English, and Urdu — trained entirely from scratch and distilled from a state-of-the-art teacher.
Headline results
- ✓Beats all-MiniLM-L6-v2 on every in-batch retrieval metric (avg MRR@10 0.585 vs 0.396) — despite being trained from scratch.
- ✓val acc@1 = 0.820, surpassing paraphrase-MiniLM-L12-v2 (0.795) on the validation set.
- ✓Cross-lingual EN↔UR acc@1 = 0.757 — genuine English↔Urdu transfer with no shared script.
- ✓Distillation success: student cosine geometry matches the teacher's with rel_mse = 0.046.
1 · The Approach
Training a competitive embedding model from random initialization is hard: a small model given only “same / different” triplet labels cannot, on its own, discover the geometry of semantic similarity. We solved this with a two-stage recipe:
Stage A — Masked Language Modeling (from scratch)
1.1M trilingual sentences, 50K BPE vocabulary trained from scratch, 6,000 steps. This teaches the encoder the basic mechanics of three languages without any pretrained backbone.
Stage B — Relational Knowledge Distillation
The MLM encoder is too weak to learn retrieval from sparse labels, so we distill it using intfloat/multilingual-e5-base (768-dim) as teacher. Instead of one bit of signal per triplet, the student receives 960 dense numbers per batch — it learns to reproduce the teacher's full similarity structure.
loss = rel_weight · MSE(Cstudent, Cteacher) + ce_weight · InfoNCE(anchor, positive)
where C is the cosine-similarity matrix within each batch. This is the same relational-distillation idea used to build production embedders — applied here to a from-scratch 41M architecture.
MenteE AI · Trilingual embeddings trained from scratch · Arabic · English · Urdu · Method: MLM (from scratch) → relational knowledge distillation from multilingual-e5-base.