How Relational Knowledge Distillation Makes a 41M Model Beat 100M+ Giants

Small embedding models trained contrastively from scratch usually fail — embeddings spread on the sphere but carry zero retrieval signal (chance ~1/97). MenteE AI Research at menteeai.org fixed it for mentee-embed-v1 with relational knowledge distillation.

Teacher: intfloat/multilingual-e5-base

We distill a 768-dim teacher into our 41M, 384-dim student. For each batch we compute cosine matrices C_student and C_teacher and minimize rel_weight·MSE(Cs, Ct) + ce_weight·InfoNCE. One sparse label becomes 960 dense supervision signals per batch. That is why mentee-embed-v1 reaches rel_mse 0.046 and avg MRR@10 0.585 vs all-MiniLM-L6-v2 0.449.

Why MLM bootstrap matters

Stage A MLM (50K BPE, 1.1M sentences, 6,000 steps) gives language-aware initialization free of any pretrained backbone. Contrastive-only from random weights collapses regardless of learning rate — classic representation collapse. MLM breaks it cheaply. Chart on /research.

When distillation still lags

On 15K-doc corpus-pool, mentee-embed-v1 gets MRR@10 ~0.19 — fine as a re-ranker, not a billion-doc retriever. That is data scale (810K triplets vs billions), not method. We document it openly in the Zenodo preprint.

Try it

Code: github.com/MenteE-s/mentee-embeddings · Model: MenteEAI/mentee-embed-v1 · Contact via /contact for collaboration.