Methodology

1. Observe. Run the workload on the current configuration and record every call: tokens, cost, latency, validator verdicts.
2. Reason. Nemotron (the analyst model) reads the measured profile and returns structured proposals in JSON.
3. Constrain. Each proposal is checked against an allowlist of three knobs: route a category to a tier, cap output tokens, enable escalation. Anything else, including commands or code, is refused before execution.
4. Approve. A person approves testing. Nothing touches the live router.
5. Execute and verify. The candidate runs on the same tasks. An independent gate accepts it only if no task that passed before now fails, the pass count holds, there are no provider errors, and cost falls. The LLM cannot override a failing validator.
6. Refine. Up to three rounds, with the measured regressions fed back to the analyst.
7. Decide. A person accepts, rejects, or exports the verified recommendation.

The optimizer verifies on the same recurring workload it tunes. That is the intended use; a result does not transfer automatically to other workloads.

For each tier i the router holds p_i, the probability that the tier's output passes the quality gate for the task's category, and c_i, its expected cost per call. Starting at tier s and escalating on failure:

E[cost | s] = Σ_{k=s..K} c_k · Π_{j=s..k-1} (1 − p_j)
P[pass | s] = 1 − Π_{k=s..K} (1 − p_k)

It picks the cheapest start tier whose chain meets the quality target, whose own first-attempt pass probability is at least the floor (default 50%, since a cheap call that usually fails still wastes compute and doubles latency), and that fits the cost and latency limits. If none qualifies it starts at the most reliable tier.

p_i is a Beta posterior mean with pseudo-count k = 4: a documented heuristic prior per (category, tier), reduced for complex tasks, updated with the observed pass/fail counts. The prior is an assumption, not a trained model, and it fades as evidence accumulates. Expected tokens use observed averages once a (category, tier) has three or more graded attempts.

Every benchmark task and every preset is graded deterministically. Validators include JSON Schema, exact field equality on dotted paths, exact / label / numeric (with tolerance) / regex answers, required terms, word and bullet limits, and Python unit tests executed in a sandboxed subprocess with a timeout.

When a task has no deterministic check, the output is marked unverified. An optional model judge can score it, but the score is labelled an estimate and its cost is counted as overhead. Judge scores are never used in benchmark results.

On failure the router may make one bounded repair attempt on the same tier (structural failures only, with the validator's message as feedback), then escalate to the next tier with a fresh prompt.

• EcoBench v1: 49 original tasks in 7 categories (extraction, summarization, classification, structured JSON, reasoning, coding, data interpretation). 14 calibration and 35 held-out test cases.
• Conditions: fixed:<tier> always uses one model with no escalation; adaptive routes from the prior only (cold); adaptive_warm routes with evidence from the calibration split only, so no test case informs its own routing.
• All conditions run the same cases at temperature 0, so comparisons are paired.
• Pass rates carry Wilson 95% intervals. Differences versus the baseline carry paired bootstrap 95% intervals over tasks.
• Cost is provider-reported tokens × configured price. Hidden reasoning tokens are included because they are billed as completion tokens.
• Results come only from stored records of real calls. Runs on the offline mock provider are labelled sample data everywhere and never reported.

Loading parameters from the backend…

• 35 held-out tasks give wide intervals; small differences are not claimed as significant.
• Category heuristics in the analyzer are keyword-based and can misclassify; callers can set the category explicitly.
• Prices and model IDs are configuration, verified against the authenticated Nebius catalog at startup and in Settings.
• Environmental figures are scenario estimates, not measurements, and are shown with their bounds.