mentee-embed-v3 vs MiniLM vs mpnet: Arabic, English and Urdu Benchmark 2026

If you need an embedding model that works across Arabic, English and Urdu without paying for a closed API, your options in 2026 are limited. This post benchmarks mentee-embed-v3 from MenteE AI against the most common open alternatives on three protocols. All numbers are from our open technical report at /research and preprint doi:10.5281/zenodo.22117673.

The models compared

• mentee-embed-v3 — 41M, 384-dim, random init, Apache 2.0. MenteEAI/mentee-embed-v3
• all-MiniLM-L6-v2 — 23M, 384-dim, pretrained on 1B+ English pairs
• paraphrase-MiniLM-L12-v2 — 118M, 384-dim, pretrained
• paraphrase-mpnet-base-v2 — 278M, 768-dim, pretrained

Protocol A — In-batch retrieval (MRR@10, ~97 candidates)

Each query is ranked against ~97 in-batch candidates. This tests embedding quality in a realistic retrieval scenario.

Key takeaway: MiniLM-L6-v2 is effectively broken for Arabic and Urdu despite being pretrained on 1B+ English pairs. mentee-embed-v3 starts from random initialization and scores 0.475 AR / 0.443 UR — the only sub-50M model that works across all three languages. We trail pretrained models on English (0.766 vs 0.924–0.931), which is expected given the English-heavy baselines.

Protocol B — Wikipedia corpus (up to 15,201 passages, out-of-domain)

Honest gap: mentee-embed-v3 scores avg MRR@10 0.260 vs 0.633–0.670 for pretrained models. Zero Wikipedia passages were in our training data. This is a domain boundary, not a method failure — fine-tuning on Wikipedia-style passages would close this. If your use case is open-domain Wikipedia search, use mpnet or E5-base.

Protocol C — MS-MARCO (10,296 in-domain passages)

When the evaluation domain matches the training data, mentee-embed-v3 is strong: MRR@10 0.645, R@100 0.957. MiniLM-L6-v2 reaches 0.951 here because it was pretrained on massive English-only data including MSMARCO. We're 0.306 points behind with a 7× smaller model trained from scratch — not bad.

When to use mentee-embed-v3

• ✅ You need Arabic and/or Urdu retrieval — no other open sub-100M model is functional on both
• ✅ You need a tiny, fast, on-device model (41M, 384-dim)
• ✅ Your retrieval domain is conversational, NLI-style or MS-MARCO-style
• ✅ You want an open, reproducible model (Apache 2.0, full training code on GitHub)
• ⚠️ Your use case is open-domain Wikipedia search — use mpnet-base-v2 instead
• ⚠️ You need graded similarity (STS) as primary metric — retrieval-first trade-off

Get started

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",    trust_remote_code=True)

embeddings = model.encode(texts, tokenizer=tok)  # (N, 384)

Full model card: /embed-models · Technical report with all numbers: /research · Cite: doi:10.5281/zenodo.22117673