What people budget for: the compute
When teams estimate the cost of an AI assistant, they almost always start — and stop — at the model bill. It is the easy number to calculate. You take the average tokens a conversation consumes, apply the per-token price of your chosen model, and multiply by expected traffic. A typical support conversation might consume 2,000 tokens of context and instructions plus 300 tokens of generated answer. At the price of a mainstream mid-tier model, that lands somewhere in the region of a fraction of a cent to a few cents per answered query. Multiply by fifty thousand queries a month and you get a compute bill that usually looks surprisingly manageable.
That calculation is not wrong. It is just incomplete. Compute is the visible tip of a cost structure that runs much deeper, and the deeper layers grow faster than the traffic does.
The line items nobody budgets
The real cost of running an AI assistant at scale is dominated by work that keeps the answers good. None of it shows up on an inference invoice, and all of it takes human time:
- Evaluation. Every prompt change, model upgrade, or knowledge-base update can silently change answers. Someone has to run regression checks against real questions before each change ships.
- Prompt and pipeline maintenance. Instructions that worked last quarter drift out of date as your product, pricing, and documentation change. This is recurring engineering time, not a one-off setup task.
- Failure handling. Retries, timeouts, fallback models, and the queries the assistant gets wrong and hands to a human all carry cost. A five percent failure rate on fifty thousand queries is 2,500 conversations a month someone still has to handle.
- Observability. Logging, tracing, and reviewing conversation samples are what tell you the assistant is drifting before your users do.
- Knowledge upkeep. The documents, policies, and product data the assistant reads need owners and refresh cycles. A stale knowledge base quietly converts a working assistant into a confidently wrong one.
In practice, for a production assistant with meaningful traffic, this human layer routinely costs more than the compute. A rough planning rule many teams converge on: expect the ongoing engineering and review effort to equal at least half of a full-time engineer, and more if the assistant touches regulated or revenue-critical workflows.
How the maths changes with scale
Unit costs generally improve as you grow, but not automatically. Three effects work in your favour. First, higher volume often qualifies you for better pricing tiers or lets you commit to reserved capacity. Second, caching repeated questions — and real traffic is far more repetitive than people expect — can serve a meaningful share of queries at a fraction of the cost. Third, routing lets you stop paying premium prices for easy questions: a small, cheap model handles greetings, lookups, and simple rewrites, while the expensive model is reserved for genuinely hard queries.
Self-hosting enters the conversation here. Serving open-weight models on your own hardware turns variable per-token spend into a fixed monthly cost, which looks attractive at scale. The honest break-even question is not the hardware price — it is whether you have the operational capacity to run, monitor, and upgrade that fleet. Hardware you already know how to operate is cheap; hardware that needs a specialist you have not hired is not.
A simple model for your own numbers
You do not need a finance department to forecast this. Work through four steps:
- Measure real conversations, not assumptions: average tokens in, tokens out, and the share that end in human handoff.
- Multiply by your per-token price, then add roughly ten percent for retries and malformed responses that get regenerated.
- Layer on the human costs: evaluation, maintenance, and review time, converted to a monthly figure at real salary levels.
- Add headroom of thirty to forty percent for traffic spikes, new use cases, and the experiments you will inevitably run.
The result is almost always higher than the pure inference estimate — and far more useful, because it is a number you can actually hold a budget to.
Where the savings really are
Teams that run assistants efficiently share the same habits. They cache aggressively and measure hit rates. They route queries by difficulty instead of sending everything to the biggest model. They cap answer length and trim system prompts, because output tokens cost more than input tokens. They batch offline work like summarisation and classification instead of running it interactively. And they review their model mix every quarter, because pricing and capability shift fast enough that last year's optimal setup quietly becomes this year's overspend.
The real cost of an AI assistant at scale is therefore not a line item — it is a system: compute, people, and process feeding each other. Budget for all three and the numbers stay boring, which is exactly what you want from production infrastructure.
