AT&T Cut AI Costs 56%: The Model Routing Playbook

This week, AT&T disclosed a number that should get the attention of every finance leader signing off on an AI budget: it cut its internal AI spend by 56% by routing 40% of employee AI usage away from Claude and GPT-class models toward open-weight alternatives — Nvidia Nemotron, Meta Llama, Google Gemma — for the tasks that didn't need a premium model in the first place. The quality hit, measured across its internal benchmarks, was 2%. AT&T's "Ask AT&T" platform reportedly processes around 45 billion tokens a day, and the company says it's pushing the routed share toward 60-70%. At that scale, a 56% cost reduction isn't a rounding error — it's millions of dollars a year, redirected from API bills to anything else the business wants to fund.

The mechanism behind it has a name: model routing. Instead of sending every request — a one-line Slack summary and a complex multi-step reasoning task alike — to the same expensive, top-tier model, a routing layer classifies each request by what it actually requires and sends it to the cheapest model capable of doing that job well. Simple classification, extraction, and formatting tasks go to smaller, often open-weight models that cost a fraction as much per token. Genuinely hard reasoning, high-stakes writing, or ambiguous requests still go to the frontier model. Nobody using the system has to think about which model to pick — the routing layer decides, invisibly, on every single call.

Why This Matters Beyond One Telecom

AT&T is a large, well-resourced company that can afford to build this kind of infrastructure in-house, and it still took deliberate engineering investment to get there — this isn't something that happens by default when a company adopts AI. What's notable is the timing: this story broke the same week OpenAI cut GPT prices by as much as 80% on some tiers, and it lands in a broader pattern that's been building all year. Model prices are falling, usage-based billing is spreading, and the businesses getting hurt aren't the ones adopting AI — they're the ones adopting it without ever revisiting which model handles which task. Most companies running AI in production today send every request to whichever model they set up first, months or years ago, and never touch that decision again. That's the equivalent of hiring a senior partner at a law firm to photocopy documents, every single day, indefinitely, because nobody assigned the photocopying to someone cheaper.

What Model Routing Actually Requires

The idea is simple. The execution is where most internal attempts stall out, because routing done carelessly just moves the failure mode from "too expensive" to "unreliable in ways nobody notices until a customer complains."

• Task classification that's actually accurate — misrouting a complex request to a cheap model doesn't just produce a worse answer, it can produce a confidently wrong one that looks fine until someone downstream relies on it
• A fallback path — when a cheap model's output fails a confidence or quality check, the request needs to escalate to a stronger model automatically, not silently ship a bad result
• Per-task-type cost and quality tracking — routing decisions should be revisited as models get cheaper or better, not set once and forgotten, which is exactly the mistake routing is meant to fix in the first place
• A router that's invisible to the end user — employees or customers interacting with the system shouldn't need to know or care which model answered; the routing has to be infrastructure, not a decision they're asked to make

AT&T built this with LiteLLM, an open-source routing layer, which is a reasonable starting point — but the routing logic itself, the classification rules, the fallback thresholds, and the ongoing tuning as new models ship, is the actual engineering work. The tool is free. Getting the routing right for your specific mix of tasks is not.

A Realistic Scenario

A Wizeb client, a mid-size e-commerce brand running an AI-powered customer support and product-tagging pipeline, was sending every request — from "what's your return policy" to genuinely ambiguous complaint triage — through the same frontier model, because that's what the original build used and nobody had revisited it since. Their monthly model spend had crept up steadily as support volume grew, and the instinct was to negotiate a better enterprise rate with their existing provider. We audited the actual task mix instead: roughly 70% of requests were high-volume, low-complexity — FAQ-style answers, product tag extraction, order-status lookups — that a much cheaper model handled just as well in blind testing. We built a routing layer that classified incoming requests and sent that 70% to a lower-cost model, reserved the frontier model for complaint triage and anything the classifier flagged as ambiguous, and added an automatic escalation path for any low-confidence response. Their monthly AI spend dropped 51% with no measurable drop in customer satisfaction scores, and the frontier model's share of total requests fell from 100% to under a third — used only where it was actually earning its cost.

How to Check Whether This Is Your Problem

1. 1Pull your last month of AI API usage and sort requests by task type — if you can't do this because everything runs through one undifferentiated pipeline, that's the finding itself
2. 2Estimate what share of those requests are simple, high-volume, and low-stakes — formatting, extraction, classification, FAQ-style answers — versus genuinely complex reasoning or high-stakes output
3. 3Ask whether your current setup sends 100% of requests to the same model tier regardless of that split — if yes, you're very likely paying frontier-model prices for commodity work
4. 4Before negotiating a bigger discount with your current model provider, calculate what routing even a third of your volume to a cheaper model would save — for most companies, it's a larger number than any discount a vendor will offer
5. 5If you're building or buying an AI system today, ask whether model choice is a one-time decision baked into the architecture, or a routing layer that can be tuned and re-tuned as prices and capabilities shift

How Wizeb Approaches This

Cost optimization is one of Wizeb's core service lines because it's one of the highest-leverage things we can do for a client's AI spend without touching quality. We audit the actual task mix running through a client's AI systems, build routing logic that sends each request to the cheapest model that can handle it reliably, and put fallback and monitoring in place so a misrouted request gets caught and escalated automatically instead of shipping quietly wrong. AT&T just proved this works at a scale most businesses will never operate at. The same logic scales down cleanly — the savings percentage tends to be similar whether you're processing 45 billion tokens a day or a few million. If your AI spend has been climbing and nobody has revisited which model handles which task since the system was first built, that's usually the fastest place to find real savings. Start at wizeb.com/services/token-optimization.

Three Questions Before Your Next Budget Review

1. 1What percentage of your AI requests are going to your most expensive model by default, not by necessity?
2. 2If AT&T can cut its bill 56% with a 2% quality hit by routing task-appropriately, what's the equivalent number sitting unclaimed in your own AI spend?
3. 3Is your model choice an architecture decision made once, or infrastructure that gets tuned as prices and models keep changing under you?