Think While You Speak: A Preprint on Teaching Models When to Think, Not Just What to Answer
Our new preprint trains a model to decide per input whether to think. A fast linear path answers by default, an episodic memory gates a deliberate pathway, and the routing gradient is measured: 2.8% of easy inputs trigger deliberation, 99.7% of medium and 99.2% of hard. Pure NumPy, CPU only, DOI 10.5281/zenodo.24353178.
Today most AI systems operate with a binary switch. Reasoning is either on for every question, or off for every question. Ask a reasoning model what two plus two is and it will happily spend a minute of compute and a thousand tokens proving it. Ask a small fast model a multi-step problem and it will fail confidently in half a second. Both failure modes come from the same design decision: the amount of thinking is fixed, not learned.
Our new preprint, Think While You Speak: Adaptive Computational Effort in Neural Dialogue Systems, trains that decision instead. This post walks through the architecture, the measured results, and the honest limitations.
The architecture: two paths and a monitor
The system, which we call Adaptive Cognitive Processing (ACP), has four components:
- A fast pathway, deliberately linear. It answers by default and cannot fit nonlinear targets, which makes the capability gap between the two paths provable rather than assumed.
- A deliberate pathway, a 48-unit MLP that can fit everything the fast path cannot, at roughly 54 times the multiply-accumulates.
- A monitor, a linear gate over the input plus one explicit novelty scalar, producing a routing probability p. At inference, p above 0.5 means the deliberate answer ships.
- An episodic memory of 30 keys. Novelty is one minus the maximum cosine similarity, so familiar inputs answer instantly and novel inputs escalate.
Training combines task loss, routed cross-entropy against the ground-truth route, and a compute penalty. Every gradient is hand-derived in NumPy and verified against finite differences; the maximum relative error is 1.7e-9. The whole thing runs on a CPU.
The measured results
Five thousand examples, three difficulty levels, ten seeds. Task accuracy counts a prediction as correct when the absolute error is below 0.5.
The routing gradient is the headline. ACP deliberates on 2.8% of easy inputs, 99.7% of medium inputs, and 99.2% of hard inputs. The ground-truth base rate is 66.7%, and ACP lands at 67.2%: it recovers the optimal policy almost exactly. Accuracy is 93.5%, within 0.2 points of the always-deliberate model, at two-thirds of the deliberation.
The ablation: memory is doing real work
A fair question about any memory-augmented architecture is whether the memory actually matters, or whether a plain gate on the input would do the same job. So we retrained the model with the novelty feature removed, leaving the monitor to route from the raw input alone. Across three seeds:
- Task accuracy dropped from 94.3% to 73.7%.
- Routing accuracy dropped from 98.7% to 92.1%.
- The overall deliberation rate barely moved (66.2% vs 67.2%), but the allocation got worse: the x-only gate deliberated on 11.2% of easy inputs and only 87.9% of hard inputs.
That last line is the mechanism. The misroutes land exactly where they cost the most: hard inputs sent to a linear path that cannot fit them. The memory feature is not decorative; it stabilizes the routing decision at the margin where an error hurts.
What we are not claiming
The preprint states its own boundaries, and we repeat them here. This is a mechanism study on a synthetic task, not a language model evaluation; dialogue is the motivating application and the next step. The router trains on ground-truth route labels, which a deployed system would not have, so label-free routing is the obvious follow-up. And the monitor itself is not free: the 30-key memory scan costs about 496 multiply-accumulates per input against 864 for the deliberate path, so at this operating point there is no net FLOP saving yet. A smaller memory or a low-dimensional sketch would move that crossover.
Read it, run it, break it
The preprint is open access at doi:10.5281/zenodo.24353178. The full implementation, including the hand-derived gradients, the gradient checker, the evaluation harness, and every chart generator, is at github.com/MenteE-s/adaptive-cognitive-processing under MIT. The complete chapter with every table is on our research page.
If you build on it, disagree with it, or find a cheaper monitor that closes the FLOP gap, we want to hear about it. That is what publishing the method is for.
Author · MenteE AI — menteeai.org · syab.tech