Embed Models
mentee-embed-v1
A compact, multilingual text embedding model trained from scratch for Arabic, English, and Urdu retrieval — distilled from a state-of-the-art teacher. This page is the model card: how it was built, how to use it, and an honest account of what it does well and where it falls short.
How we built it
Stage A — Masked Language Modeling
The encoder first learns all three languages by reconstructing masked tokens (15% masking, 80/10/10 scheme) over 6,000 steps — no pretrained backbone involved. Validation loss fell steadily across the run, confirming it absorbed the Arabic, English, and Urdu scripts and vocabularies.
Stage B — Relational Knowledge Distillation
The MLM encoder is too weak to learn retrieval from sparse labels alone, so we distill intfloat/multilingual-e5-base into it: the student learns to reproduce the teacher's full batch similarity structure (MSE on cosine matrices), combined with InfoNCE. The MLM bootstrap is what prevents representation collapse — contrastive training from random weights collapses to chance.
Fully reproducible
Every step — data fetching, cleaning, deduplication, tokenization, two-stage training, checkpointing, evaluation, packaging — is open and scripted end to end.
How to use it
The snippet below is the target interface. Today the weights load via src/model.py in the repo — a Sentence-Transformers-compatible export is planned and tracked on GitHub.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("MenteEAI/mentee-embed-v1")
sentences = [
"How do I file a tax return?",
"Steps to submit an annual tax filing",
]
emb = model.encode(sentences)
# emb shape: (2, 384) — cosine similarity measures meaning closenessStrengths
What this model does well.
In-batch retrieval
Beats all-MiniLM-L6-v2 on every in-batch metric (avg MRR@10 0.585 vs 0.396) — despite being trained from scratch on a single consumer GPU.
Cross-lingual transfer
English↔Urdu acc@1 = 0.757 with no shared script — genuine multilingual transfer, not keyword matching.
Validation performance
val acc@1 = 0.820, surpassing paraphrase-MiniLM-L12-v2 (0.795) on the validation set.
Tiny footprint
41M parameters — ~7× smaller than mpnet-base (278M), ~3× smaller than MiniLM-L12 (118M). Runs on a phone.
Fully reproducible
Every step — data to evaluation — is open and scripted. One command reproduces the full pipeline on a free Colab T4.
Limitations
What this model does not do well. We document these openly.
Corpus-pool retrieval lags
On open-domain ranking over 15K docs (Protocol B), MRR@10 ≈ 0.19 vs 0.94 for mpnet-base. Best used as a re-ranker, not a standalone billion-document search engine.
Training scope
Trained on NLI + parallel translation data only. Domain-specific retrieval (legal, medical) will need fine-tuning.
Sequence length
Capped at 128 tokens; longer documents should be chunked.
Format
Current model.pt loads via src/model.py; a Sentence-Transformers-compatible export is planned.
Data scale
~810K triplets (31.4M tokens) vs billions used by web-scale models. The gap to mpnet/MiniLM-L12 on MIRACL reflects ~1000× less training data.
Source & weights
All code, training configs, and evaluation scripts are public. Model weights and the full model card will be released under Apache 2.0.
Technical specifications
- Architecture
- Transformer encoder — 12 layers, hidden 384, 12 heads, FFN 1536
- Output
- 384-dimensional L2-normalized embeddings, mean pooling
- Context
- Up to 128 tokens — longer documents should be chunked
- Parameters
- ~41 million
- Tokenizer
- Custom ByteLevel BPE, 50K vocabulary, trained on our own corpus
- Training objective
- Stage A: masked LM → Stage B: relational knowledge distillation + InfoNCE
- Teacher
- intfloat/multilingual-e5-base (768-dim)
- Data
- ~810K triplets ≈ 31.4M tokens
- Hardware
- Single consumer GPU, bf16
Citations & prior work
mentee-embed builds directly on the following open datasets and published methods. We credit them explicitly.
all-NLI
Source of the 557K English NLI triplets used in Stage-B fine-tuning.
XNLI
Cross-lingual NLI corpus; premise / entailed / contradicted triplets derived for Arabic and Urdu.
MIRACL
Multilingual retrieval corpus used to build clean, held-out evaluation triplets across all three languages.
BERT (Devlin et al., 2019)
Masked language modeling objective used in Stage A.
SBERT (Reimers & Gurevych, 2019)
Siamese / dual-encoder contrastive fine-tuning recipe adapted for from-scratch training.
InfoNCE (Oord et al., 2018)
Contrastive loss with in-batch negatives and an explicit hard-negative column (temperature 0.07).