LLM Model Routing: When a Small Model Should Go First, and How to Prove It Pays

Three Designs and What Each One Costs You

The router design is the one studied in RouteLLM (Ong et al., 2024, ICLR 2025). Its routers estimate the probability that a strong model's answer would beat a weak model's answer, then compare that estimate with a cost threshold: below the threshold the query goes to the weak model, above it to the strong one. The authors deliberately call a single model per query and point out that cascades "rely on multiple LLM queries, which can increase latency".

The cascade design is the one in FrugalGPT (Chen, Zaharia and Zou, 2023). A query goes to a list of models in order, cheapest first. After each answer, a separate scoring model rates how reliable it is, and the cascade stops when the score clears that step's threshold. In the paper's case study on a financial news dataset, the learned cascade tried GPT-J, then J1-L, then GPT-4, accepting GPT-J's answer only above a score of 0.96.

Routing is a quality and cost decision for a healthy system. Switching models because a provider is failing is a different problem, with different triggers, covered in Run a Model Fallback Drill.

What the Papers Measured, and What They Did Not

Both papers report large savings, on their own benchmarks and model pairs:

• RouteLLM routed between gpt-4-1106-preview and Mixtral 8x7B. Measured against a random router at the point where the router recovers half of the quality gap between the two models, the best routers' cost-saving ratio over GPT-4 was 3.66x on MT Bench at 95% of GPT-4's score, 1.41x on MMLU at 92%, and 1.49x on GSM8K at 87%. The cost figures rest on the authors' own per-token estimates for the two models.
• FrugalGPT reports matching the best single model's accuracy with up to 98% lower cost on one of its three datasets, and 59% on the weakest of the three. The prices behind those figures were retrieved in March 2023.

Two findings matter more for your system than the headline numbers. First, RouteLLM's routers trained only on public chat preferences performed close to a random router on MMLU and GSM8K, because those questions looked nothing like the training data. Adding about 1,500 labeled in-domain questions, under 2% of the training set, moved all of them above random. Second, FrugalGPT's authors note that the cascade needs labeled examples from the same or a similar distribution as real queries. Both results point the same way. A router or cascade tuned on someone else's traffic tells you little about yours, and RouteLLM's own limitations section says performance "can vary widely" between routers on the same benchmark without a clear explanation.

The Number That Decides: What a Misroute Costs

A misroute is a request the cheap path answered wrongly that the large model would have answered correctly. Your price list tells you what routing saves. Only your evals tell you how many misroutes you buy with it.

A worked example, using Anthropic's list prices as checked on 2026-10-23: Claude Opus 5.5 at $4 input and $20 output per million tokens, and Claude Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens. A request with 1,000 input and 150 output tokens costs $0.0070 on Opus 5.5 and $0.000175 on Haiku 5.5, so routing it down saves about $0.0068. Now assume a wrong answer costs you $0.50 in support time, a refund or a lost user. Routing then pays only while fewer than about 1.4% of downgraded requests become misroutes ($0.0068 ÷ $0.50). A price ratio of 40 to 1 sounds decisive. The break-even miss rate is the number to watch.

The cascade has a second break-even. If C_small and C_large are fixed generation costs per call, C_check is the average cost of checking the small answer, and e is the share of requests that escalate, a two-model cascade costs C_small + C_check + e × C_large per request. It is cheaper than sending everything to the large model while e < 1 − (C_small + C_check) / C_large. With the prices and token counts above, ignoring check costs gives a limit of 97.5%; a paid scorer lowers it. The replay below counts generation tokens only and excludes router, check and infrastructure costs. Add those costs before using its net estimate as a budget decision. Latency on escalated requests and missed wrong answers also constrain the design. Swap in your own prices and token counts; the AI cost modeling guide shows how to estimate them per feature.

Choosing an Escalation Signal

A cascade is only as good as the check that decides whether the small model's answer stands. Rank candidate signals by how independent they are of the answer they judge:

1. Deterministic checks. The output fails schema validation, the generated code fails its tests, a tool call returns an error, or the response stopped on a refusal or a token limit. These signals are cheap, and a failure is concrete evidence rather than an opinion.
2. A separately trained scorer. FrugalGPT trains a small regression model, DistilBERT in its case study, on labeled examples to predict whether a given answer is correct. It is evaluated like any classifier, on held-out data from your traffic.
3. The model's self-reported confidence. Asking the small model "how sure are you?" uses the same model that may have just been wrong to grade itself. Neither paper uses it as the signal. Treat it as weak until your replay shows that it separates right answers from wrong ones on your gold set.

Retrieval scores belong in the same skeptical bucket: a high similarity between the question and the retrieved passage says nothing about whether the answer is right, as Cosine Similarity Is Not an Answer Confidence Score explains.

Latency and the Prompt Cache

An escalated cascade request waits for the small model's full answer, then the check, then the large model's full answer. Measure the 95th percentile on escalated requests separately from the median; the average hides the users who wait twice.

Routing also splits your prompt cache. Cached prefixes are stored as the model's internal key-value states, and OpenAI's prompt caching guide lists model among the settings that affect the cached prefix, because a different model can use different weights and caching behavior. A long system prompt cached on the small model does nothing for the large model's first call. Anthropic's invalidation table does not list the model as a row, so on Claude, treat each model as its own cache and verify the hit rate with the method in Prompt Cache Hit Rate: Why Your Cached Prompt Isn't Caching. With a router, each model's cache sees only part of the traffic. With a cascade, the large model's cache sees only the escalated requests.

Replay a Gold Set Before You Ship

You can test a routing policy offline. Take a few hundred real, anonymized requests with known good answers. Run every request through both models once, grade each answer against the gold answer, record token counts, and record your deterministic check's verdict on the small answer. Then replay the policies over those records. No user ever sees a misroute.

// replay.mjs: run a graded gold set through three routing policies, offline.
// Every record was answered by BOTH models once and graded against the gold answer.
// The six records below only show the format; they are not measurements.
const PRICE = { // USD per million tokens, Anthropic pricing page, checked 2026-10-23
  small: { in: 0.10, out: 0.50 },  // Claude Haiku 5.5, prompts up to 100,000 tokens
  large: { in: 4.00, out: 20.00 }, // Claude Opus 5.5
};
const HARM_PER_MISS = 0.50; // your assumption: USD lost per wrong answer
const THRESHOLD = 0.5;      // router: send to large when score >= THRESHOLD

const gold = [
  // score: router's estimate that the large model is needed (0..1)
  // checkOk: your deterministic check on the small answer (schema, tests, tool errors)
  { id: "q1", score: 0.1, small: { in: 900, out: 120, pass: true, checkOk: true },   large: { in: 900, out: 140, pass: true } },
  { id: "q2", score: 0.2, small: { in: 1100, out: 90, pass: false, checkOk: true },  large: { in: 1100, out: 95, pass: true } },
  { id: "q3", score: 0.7, small: { in: 2400, out: 300, pass: false, checkOk: false }, large: { in: 2400, out: 350, pass: true } },
  { id: "q4", score: 0.4, small: { in: 800, out: 60, pass: true, checkOk: true },    large: { in: 800, out: 70, pass: true } },
  { id: "q5", score: 0.9, small: { in: 3000, out: 400, pass: true, checkOk: true },  large: { in: 3000, out: 420, pass: true } },
  { id: "q6", score: 0.6, small: { in: 1500, out: 200, pass: false, checkOk: false }, large: { in: 1500, out: 210, pass: false } },
];

const cost = (tier, t) => (t.in * PRICE[tier].in + t.out * PRICE[tier].out) / 1e6;

function replay(name, decide) {
  let usd = 0, passed = 0, misses = 0, largeCalls = 0;
  for (const r of gold) {
    const path = decide(r); // ["small"], ["large"] or ["small", "large"]
    for (const tier of path) usd += cost(tier, r[tier]);
    if (path.includes("large")) largeCalls++;
    const answeredBy = path[path.length - 1];
    if (r[answeredBy].pass) passed++;
    else if (answeredBy === "small" && r.large.pass) misses++; // a misroute
  }
  return { name, passRate: passed / gold.length, usdPer1k: (usd / gold.length) * 1000,
           largeShare: largeCalls / gold.length, misroutes: misses };
}

const results = [
  replay("single large", () => ["large"]),
  replay("router", (r) => (r.score >= THRESHOLD ? ["large"] : ["small"])),
  replay("cascade", (r) => (r.small.checkOk ? ["small"] : ["small", "large"])),
];
const base = results[0];
for (const p of results) {
  const savedPer1k = base.usdPer1k - p.usdPer1k;
  const harmPer1k = (p.misroutes / gold.length) * 1000 * HARM_PER_MISS;
  console.log(p.name.padEnd(13), `pass ${(p.passRate * 100).toFixed(0)}%`,
    `| $${p.usdPer1k.toFixed(2)}/1k req | large ${(p.largeShare * 100).toFixed(0)}%`,
    `| misroutes ${p.misroutes} | net vs large $${(savedPer1k - harmPer1k).toFixed(2)}/1k`);
}

The script runs with node replay.mjs on Node 18 or later. Replace the six format records with your graded set and the prices with the ones you pay. Read the last column first: it subtracts the assumed cost of misroutes from the token savings. Note record q2, where the small model was wrong but the deterministic check passed it. That is the kind of miss a cascade cannot catch, and only a graded gold set reveals it.

Before you ship, sweep the router threshold across the range and pick the cheapest setting whose pass rate stays within the margin you already accept from the large model alone. Confirm the chosen threshold on a separate held-out set before deployment. Re-run the replay whenever either model, the prompt or the traffic mix changes. If you test a self-reported confidence score as a cascade signal, add it as a field and compare its values on passing and failing records. Overlap means no threshold perfectly separates those records; the score may still be useful. Measure the wrong answers accepted and correct answers escalated at each threshold on held-out data.

Related Guides