Think While You Speak · Overview

MenteE AI · Open Research

Published research

Three open research lines, all with full numbers and honest limitations. Think While You Speak: a NumPy-only architecture where a memory-aware monitor decides when a model should think. MenteE Code 3B: a coding model built from 53,567 execution-verified agent trajectories, regression published on purpose. mentee-embed: trilingual embedding models (Arabic, English, Urdu) trained entirely from scratch, v1 through v4.

Preprint · v1.0Zenodo · 18 Nov 2026 · DOI 10.5281/zenodo.24353178 · CC BY 4.0

Think While You Speak: Adaptive Computational Effort in Neural Dialogue Systems

Shah, Syed Syab Ahmad · Team MenteE AI — MenteE AI

A minimal two-pathway architecture where a memory-aware monitor decides per input whether to think. Routing gradient: 2.8% of easy inputs deliberated, 99.7% of medium, 99.2% of hard. 93.5% accuracy at 67.2% deliberation (10 seeds), and a 3-seed ablation showing that removing the episodic novelty feature drops accuracy from 94.3% to 73.7%. Pure NumPy, CPU only.

Show BibTeX
@misc{shah2026acp,
  title   = {Think While You Speak: Adaptive Computational Effort
             in Neural Dialogue Systems},
  author  = {Shah, Syed Syab Ahmad and Team MenteE AI},
  year    = {2026},
  publisher = {Zenodo},
  doi     = {10.5281/zenodo.24353178},
  url     = {https://doi.org/10.5281/zenodo.24353178},
  note    = {MenteE AI, https://menteeai.org}
}
New preprint

Think While You Speak — Adaptive Cognitive Processing

The Architecture

Two pathways and a monitor. The fast path answers by default; a small episodic memory decides when the answer deserves a second look. Implemented in pure NumPy, CPU only.

Preprint · v1.0Zenodo · 18 Nov 2026 · DOI 10.5281/zenodo.24353178 · CC BY 4.0

Think While You Speak: Adaptive Computational Effort in Neural Dialogue Systems

Shah, Syed Syab Ahmad · Team MenteE AI — MenteE AI

A minimal two-pathway architecture where a memory-aware monitor decides per input whether to think. Routing gradient: 2.8% of easy inputs deliberated, 99.7% of medium, 99.2% of hard. 93.5% accuracy at 67.2% deliberation (10 seeds), and a 3-seed ablation showing that removing the episodic novelty feature drops accuracy from 94.3% to 73.7%. Pure NumPy, CPU only.

Show BibTeX
@misc{shah2026acp,
  title   = {Think While You Speak: Adaptive Computational Effort
             in Neural Dialogue Systems},
  author  = {Shah, Syed Syab Ahmad and Team MenteE AI},
  year    = {2026},
  publisher = {Zenodo},
  doi     = {10.5281/zenodo.24353178},
  url     = {https://doi.org/10.5281/zenodo.24353178},
  note    = {MenteE AI, https://menteeai.org}
}

The four components

Fast pathway

A linear map, Wf·x + bf. Cheap by construction: a linear model cannot fit the quadratic or product-interaction targets, so the capability gap between the two paths is real rather than assumed.

Deliberate pathway

A 48-unit MLP over the concatenation [x, fast_out]. It can fit every target the fast path cannot, at roughly 54x the multiply-accumulates.

Monitor

p = σ(w·x + wn·novelty + b), a linear gate over the input plus one explicit episodic novelty scalar. The thresholded flag is the traceable routing decision.

Episodic memory

30 fixed keys (15 from the familiar cluster, 15 random). Novelty = 1 − max cosine similarity. Familiar inputs answer fast; novel inputs escalate.

Training objective

L = (y − y*)² + α · [−r log p − (1 − r) log(1 − p)] + λ · p · Cd

Task loss plus routed cross-entropy plus a compute penalty. The routing term teaches the monitor which inputs need the deliberate path; the penalty keeps always-deliberate from being free. Gradients are hand-derived and checked against finite differences.

Results

5,000 synthetic examples, three difficulty levels, 10 seeds. Task accuracy = |error| < 0.5. The hero summary below is generated straight from the evaluation JSON.

93.5%
Task accuracy · 10 seeds
98.7%
Routing accuracy
67.2%
Deliberation rate
1.7e-9
Max gradient error

Main comparison — 10 seeds, mean ± std

ModelAccuracyRouting Acc.Delib. Rate
ACP ★0.935 ± 0.0120.987 ± 0.0020.672 ± 0.000
Always Delib.0.937 ± 0.0060.667 ± 0.0001.000
Fixed MLP0.947 ± 0.0070.667 ± 0.0001.000
Random Router0.653 ± 0.0430.501 ± 0.0080.499 ± 0.005
Always Fast0.370 ± 0.0810.333 ± 0.0000.000

Deliberation rate is the fraction of inputs routed to the expensive path. The ground-truth base rate is 66.7% (Medium + Hard).

ACP summary: routing gradient, accuracy vs deliberation rate, memory ablation, and the capacity gap on hard inputs

Routing Gradient

Per-difficulty accuracy and deliberation. The point of the architecture is this column split: almost never deliberate on Easy, almost always on Medium and Hard.

Accuracy and deliberation by difficulty — mean across 10 seeds

ModelEasy AccMed AccHard AccEasy DelibMed DelibHard Delib
ACP ★0.9770.9650.8630.0280.9970.992
Always Fast0.2980.5490.2640.0000.0000.000
Always Delib.0.9870.9620.8631.0001.0001.000
Fixed MLP0.9860.9680.8861.0001.0001.000
Random Router0.6380.7590.5620.4960.5030.498

The ground-truth policy deliberates on exactly 66.7% of inputs (Medium + Hard). ACP's 67.2% means it recovers the optimal policy almost exactly.

Memory Ablation

Retrain the model with the novelty term removed, so the monitor sees only the raw input. This isolates what the episodic memory contributes.

Routing ablation — 3 seeds, mean ± std

MonitorAccuracyRouting Acc.Delib. Rate
x + novelty0.943 ± 0.0080.987 ± 0.0010.672 ± 0.000
x only0.737 ± 0.0880.921 ± 0.0030.662 ± 0.008

Without the novelty feature the monitor still deliberates on 66.2% of inputs overall, but sends 11.2% of Easy inputs and only 87.9% of Hard inputs to the deliberate path.

Ablation: accuracy with and without memory, and deliberation allocation by difficulty

The memory is not decorative

The x-only monitor routes at a similar overall rate but misallocates deliberation, and the misroutes land exactly where they cost the most: 12% of Hard inputs go to a linear path that cannot fit them. That is a 20.6-point accuracy collapse from one removed feature.

Strengths

The routing gradient is near-optimal

Deliberation rises from 2.8% on easy inputs to 99.7% / 99.2% on medium and hard, matching the 66.7% base rate almost exactly, at 98.7% routing accuracy.

The capacity gap is real

The linear fast path provably cannot fit the quadratic (Medium) or product-interaction (Hard) targets, so the routing measurement is meaningful rather than a memorization artifact.

Memory does measurable work

Removing the novelty feature drops accuracy from 94.3% to 73.7%: the x-only gate deliberates at the same overall rate but sends 12% of hard inputs to the path that cannot fit them.

Reproducible without a GPU

Pure NumPy on CPU, 10 seeds, and hand-derived gradients verified against finite differences (max relative error 1.7e-9).

Limitations

Stated in the preprint itself. These are the boundaries of the claim.

Mechanism study, not language

The task is synthetic 16-dimensional regression. Dialogue is the motivating application, not the experimental setting.

Routing is supervised

The monitor trains on ground-truth route labels. A deployed system would not have those; label-free routing is the obvious next step.

The monitor is not free

The 30-key memory scan costs ~496 MACs per input vs ~864 for the deliberate path, so at 67% deliberation there is no net FLOP saving yet. A few keys or a low-dimensional sketch would move the crossover.

Novelty detector, not episodic memory

Keys are fixed and hand-filled, and the readout is a single scalar. Learned writes and content-based retrieval are future work.

Roadmap

Ordered by how much each step strengthens the central claim.

Label-free routing

Train the monitor from task loss plus a compute penalty, or from the fast path's own residual, so the system decides when to think without an answer key.

Content-addressable memory

Store key-value episodes and feed a readout vector into the monitor, then test on tasks where the same input needs different compute depending on stored context.

A monitor worth paying for

Shrink the memory to a handful of keys or a low-dimensional sketch so the routing policy turns into a real FLOP saving.

Language

Move from synthetic vectors to a small language task: streaming dialogue where deliberation is visible between turns.

Reproduction

Pure NumPy on CPU. No GPU, no training script longer than a hundred lines.

Full pipeline

pip install -r requirements.txt
python test_gradients.py
python evaluate.py
python ablation_novelty.py
python charts/make_results_table.py
python charts/generate_charts.py

test_gradients.py verifies every hand-derived gradient against finite differences before anything else runs. Every table on this page is generated from evaluation_results.json and ablation_novelty.json.

MenteE AI · Open research · mentee-embed: trilingual embeddings trained from scratch (Arabic · English · Urdu) · Think While You Speak: adaptive computational effort, NumPy-only · MenteE Code 3B: a coding model built from execution-verified agent trajectories · Every table on this page is generated from the evaluation JSON in the public repositories.