ResearchNovember 19, 2026 · 8 min read·Syed Syab Ahmad

When Should an AI Model Think? Adaptive Computation vs Fixed Compute, Explained

Reasoning on every query burns compute. Reasoning off fails hard questions. A plain-language guide to adaptive computation: early exit, model cascades, mixture-of-experts routing, memory-based novelty, and what changes when the decision to think is learned instead of configured.

ResearchExplainerAdaptive ComputationAI

There is a question hiding inside every modern AI product roadmap, and almost nobody budgets for it: when should the model think? Reasoning mode on means every query pays the full price, including the ones that need none. Reasoning mode off means the hard queries fail quietly. Most teams pick one setting and live with both failure modes. This post explains the alternative, adaptive computation, in plain terms: what it is, the four main ways to do it, what each one costs, and what we measured when we built our own.

What adaptive computation means

Fixed compute means every input takes the same path through the same amount of machinery. Adaptive computation means the amount of processing depends on the input: easy inputs finish early, hard inputs get more work. Humans do this without noticing. You answer a familiar question reflexively and slow down only when something is unfamiliar. Cognitive scientists call the two modes System 1 and System 2; the analogy is useful as long as you remember that a two-path neural network is not a cognitive science proof, it is an engineering pattern.

Four ways to make compute adaptive

1. Early exit

Attach small classifiers to intermediate layers of a network and stop as soon as confidence is high enough. BranchyNet is the classic reference. Cost per input drops, but the exit decision rides on a confidence score, and confidence is not the same as difficulty: a model can be confidently wrong.

2. Model cascades

Run a cheap model first and escalate to an expensive one when the cheap one is uncertain. This is how face detection worked for two decades before it was trendy. The hard part is the handoff rule: too strict and you pay full price anyway, too loose and the hard cases get cheap wrong answers.

3. Conditional compute inside the network

Mixture-of-experts and related methods route tokens through different subnetworks. Capacity is huge, active compute is small. The routing is per token and learned, but nothing in the architecture separates a quick answer from a careful one.

4. Architectural separation with a monitor

Two genuinely different pathways: a cheap one that answers by default and an expensive one that corrects it. A monitor decides per input. This is what our new preprint builds, with one addition we have not seen combined with it elsewhere: the monitor reads an explicit novelty signal computed from a small episodic memory, so the escalation is anchored to remembered experience rather than only to a learned threshold.

The measured example

Our preprint trains exactly that architecture in pure NumPy on a CPU and measures the routing behavior across ten seeds. The result is a clean routing gradient: 2.8% of easy inputs trigger the expensive path, 99.7% of medium inputs, and 99.2% of hard inputs. Accuracy matches an always-deliberate model within 0.2 points at two-thirds of the deliberation. And when we ablate the memory feature, accuracy collapses from 94.3% to 73.7% because the router starts sending hard inputs to the fast path. The full numbers, tables and the honest limitations (supervised routing, monitor cost, synthetic task) are in the preprint and on our research page.

Practical takeaways for teams shipping AI

  • Measure your difficulty distribution before choosing a reasoning setting. If most of your traffic is easy, always-on reasoning is pure waste; if most of it is hard, always-off is silent failure. The distribution decides, not the demo.
  • Separate the two decisions: which pathway answers, and whether that pathway is trustworthy. Confidence-only routing conflates them.
  • Price the router too. A router that costs a large fraction of the expensive path cannot pay for itself. We publish our own crossover math in the paper rather than hiding it.
  • Log the routing decisions. A per-input flag of "did we think" is the cheapest observability you will ever add, and it is what makes the difficulty distribution measurable in production.

Where this is going

The open problems are clear: routing without supervision, routers that estimate their own uncertainty instead of memorizing a difficulty boundary, and memory that actually changes what the model knows rather than flagging what it has seen. Our preprint takes one measured step and publishes everything, including the parts that do not save compute yet. If the field is going to allocate computation the way people do, fluent by default and deliberate when necessary, it will be on top of measured building blocks exactly like this one.

SS
Syed Syab Ahmad

Author · MenteE AI — menteeai.org · syab.tech

Cite mentee-embed-v1: Shah et al. (2026). mentee-embed: Training Competitive Multilingual Text Embeddings from Scratch for Arabic, English, and Urdu. Zenodo. doi:10.5281/zenodo.22087139 · Technical Report · Model Card

Related articles