GuidesLLM cost monitoring

LLM cost monitoring: how to track OpenAI API costs

Updated 17 October 2026 · 8 minute read

The short answer: record the model and the input, output and cached token counts of every LLM call, tag each call with the feature, team and customer it served, and multiply by your prices per million tokens. The provider's invoice tells you the total. Only your own per-request data tells you which feature, customer or bug caused it.

How LLM pricing works

Providers such as OpenAI and Anthropic charge per token, a piece of text of roughly three to four characters of English. Prices are quoted per million tokens and differ in four ways:

• Input and output are priced separately. Input is everything you send: system prompt, conversation history, retrieved documents and tool definitions. Output is what the model generates. Output tokens usually cost several times more than input tokens.
• Models differ widely. A large model can cost many times more per token than a small one from the same provider.
• Cached input is cheaper. When the start of a prompt repeats, providers can bill those tokens at a reduced cached rate. Some also charge for writing to the cache.
• Hidden tokens count. Reasoning models generate internal reasoning tokens that are billed as output, even though you do not see them in the reply.

Prices change often, so read them from your provider's pricing page or your contract, not from a blog post.

An illustrative calculation

Example only, with made-up round prices: say a model costs $2 per million input tokens, $0.20 per million cached input tokens and $8 per million output tokens. A support assistant sends 3,000 input tokens and gets 500 output tokens back.

One cent looks harmless. At 100,000 requests a day it is $1,000 a day, or about $30,000 a month. If 2,000 of the 3,000 input tokens are a fixed system prompt served from the cache, input falls to 1,000 × $2 + 2,000 × $0.20 per million, or $0.0024, and the request costs $0.0064. That is 36% less for the same answer.

Why LLM bills surprise teams

• Prompts grow quietly. Every message in a chat resends the whole history. More retrieved documents, longer system prompts and extra tool definitions all add input tokens to every call.
• Retries multiply cost. A timeout or a malformed JSON reply triggers a retry, and you pay for both attempts. A retry loop during a provider incident can double a day's spend.
• Agents loop. An agent that calls tools and the model repeatedly can make dozens of calls for one user request, each carrying a longer context. Without a step limit, one stuck task can cost more than a thousand normal ones.
• Prototype models reach production. A large model chosen while prototyping ships unchanged.
• Nobody owns the bill. Several teams share one API key, so the invoice is one number and no one can say which feature caused the increase.

What to measure

Look at distributions, not only averages. A p99 of 40,000 input tokens on a feature that averages 3,000 points at a bug.

Attribution: tag every request

Attribution means each call carries enough context to answer "who spent this?". Add these tags where the call is made, because that is the only place that knows them:

• the feature or endpoint, for example support-assistant or summarise-ticket;
• the team that owns it;
• the customer or tenant, and the user, as an internal ID rather than an email address;
• the environment, so test traffic does not inflate production figures.

Separate API keys per team help, but only request-level tags go finer than the team.

Use the OpenTelemetry GenAI conventions

OpenTelemetry defines semantic conventions for generative AI, so different libraries describe LLM calls the same way. Each call becomes a span with attributes such as:

Add your own attributes for feature, team and customer on the same span. Because the span sits inside the request's trace, you can see the cost of an LLM call next to the database queries and HTTP calls of the same request. The GenAI conventions are still marked as in development, so check which version your instrumentation library emits.

Budgets and guardrails

• Budgets per team or feature, by day or month, with warnings at thresholds such as 50%, 80% and 100%.
• Hard limits that block calls when a budget is exhausted, for features where an overrun is worse than an outage.
• Caps in code: a maximum output token setting on every call, a step limit for agents, and retries with exponential back-off and a small maximum.
• Model allow lists, so expensive models are used only where someone approved them.
• Alerts on rate, not only totals: a spend spike in the last hour matters more than the month-to-date figure.

The gateway pattern

A gateway is a proxy between your applications and the providers. Applications call the gateway with their own token, and the gateway holds the real provider keys. Because every call passes one point, the gateway can meter tokens, apply budgets and model rules, and record a trace, without changes in each service. The trade-off is one more hop on the request path, so it must be reliable and add little latency.

Quality versus cost

The cheapest model is not a saving if answers get worse and users retry or leave. Score answers, with automated evaluations, user feedback or both, and put the scores next to cost per feature. Common savings that rarely hurt quality: put fixed content at the start of the prompt so it can be cached, trim conversation history, retrieve fewer but better documents, and route simple tasks to a smaller model.

Track LLM costs with Televane

Televane's AI monitoring calculates LLM cost from OpenTelemetry GenAI spans, from the provider usage API, or from calls through its AI gateway, using your contracted prices per million tokens. Price changes need approval from a second person.

• Cost breakdowns by provider, model, application, feature, team, project, customer and user, with trends, a month-end forecast, CSV export and chargeback with your own markup.
• Findings for spend spikes, retry storms, low cache reuse and unapproved models.
• An AI gateway for OpenAI and Anthropic, streamed or not. Provider keys are stored encrypted, applications use scoped Televane tokens, and every call is metered and traced. Prompts and completions are never stored.
• Budgets by day or month that warn, or block calls through the gateway when they are used up, and model governance to approve, block or deprecate models.
• Quality next to cost: post evaluation scores and compare them with spend.

Basic AI cost monitoring is on the Free plan; the AI gateway is on Pro and above (see pricing). To send GenAI spans, follow Send OpenTelemetry data. There is more for product teams on the AI teams page, and you can create a free account to try it.

Frequently asked questions