What Is LLM Inference? What Happens Between Your Prompt and the AI's Answer

Imagine spending hundreds of millions of dollars building a brain, and then it just sits there. It cannot answer a single question on its own. Strange, right? That is exactly what a freshly trained AI model is: a giant file of numbers, doing nothing.

What brings it to life is called inference. It happens every time you send a message to ChatGPT, Claude or any AI tool, and it is where almost all of your AI bill comes from. Understand it once and a lot of confusing things suddenly make sense: why long chats get expensive, why the first word sometimes takes a second, and why some jobs cost half as much if you are willing to wait.

In the next few minutes we will follow one message from your keyboard to the answer, step by step, and after each step we will tell you what it means for your business. The honest caveat: the exact machinery differs between providers, so we describe the common pattern, and every price we quote is from Anthropic's own pricing page as of October 2026. The step order follows a good explainer video, which we link at the end. The business angle is ours.

What is inference in AI in simple terms?

Inference is running a trained AI model on new input to produce an output. You ask, it answers. That is one inference.

Inference vs training: what is the difference?

Training is how the model is built. It happens once, takes months and is paid for by the AI company. Inference is using the model. It happens every single time anyone sends a message, which is why, over the life of a popular model, the running cost of answering questions keeps growing long after training is done.

A lot of people think the expensive part of AI is the model itself. Not for you. You never pay for training. You pay for inference, one message at a time, measured in tokens. So everything below is really about your bill.

Step 1: Your words become tokens

The model never sees your words. Before anything else, your text is cut into tokens, which can be a whole word, part of a word or a punctuation mark, and each one is turned into a number. The model only ever works with those numbers.

Here is why that matters to you: AI APIs bill per token, not per word or per message. Anthropic estimates its current tokenizer fits about 555,000 English words into a million tokens, which is roughly 1.8 tokens per word. So a 1,000-word document you paste in costs about 1,800 tokens of input, every time you send it.

Steps 2 to 4: Check, route, and wait your turn

Your message then reaches the provider's data center. It is checked against your API key, so they know who is paying. A router sends it to hardware that suits it, and then something clever happens: batching.

Running one request at a time through a huge model would be like a bakery firing up an industrial oven for a single croissant. So providers group many requests together and run them as one batch. Modern inference engines use continuous batching: as soon as one request finishes, a new one slots straight into its place, so the expensive chips are never sitting idle.

What this means for you: when a provider can fit your job in whenever there is spare room, it is cheaper for them, and some pass that on. Anthropic's Batch API is 50 percent off for requests that can wait for results. If a job is not urgent, such as tagging last month's support tickets or writing product descriptions overnight, you are paying double by running it live.

Step 5: Prefill, where the model reads your whole prompt

Now the real work starts. Inference has two phases, and they behave very differently.

In prefill, the model reads your entire prompt in one go: the instructions, the documents, the chat history, all of it, in parallel. As it reads, it builds a running memory of everything so far.

KV cache explained

That running memory is called the KV cache. Without it, the model would have to reread your whole conversation from scratch for every single word it writes. With it, it reads once and keeps the notes. It is the reason a long chat does not get slower and slower with every reply.

Providers can also keep that work between requests. That is what prompt caching is: if you send the same long instructions again and again, the provider can reuse the reading it already did. On Anthropic's pricing, a cache read on Claude Opus 5.5 or Sonnet 5.5 costs 5 percent of the normal input price. The catch is that it only works when the start of your prompt is exactly the same each time.

Step 6: Decode, where the answer is written one token at a time

Then decode begins. The model writes the answer one token at a time. For each step it looks at everything so far, ranks every possible next token by how likely it is, picks one, adds it to the answer, and goes again. One token. Then the next. Then the next.

Prefill is limited by raw computing power, because it is a burst of parallel maths. Decode is limited by how fast data can move out of memory, because each new token has to look back at the whole KV cache. Two different bottlenecks, which is why the two phases feel different when you use an AI tool.

What is time to first token?

Time to first token, or TTFT, is how long you wait before the first word appears. It mostly measures prefill. The gap between each following word is called inter-token latency, and it measures decode. Together, they are what people mean when they say a model feels fast or slow.

Picture a customer on your website asking your AI assistant a question. If your assistant sends a 30-page manual with every message, the customer stares at a blank box while prefill reads all 30 pages. Cut the context to what matters and the first word shows up faster. That is a customer experience problem as much as a technical one.

What does LLM inference cost, and why is output more expensive?

Remember decode writes one token at a time, each one needing its own pass through the model. Reading your prompt happens in one parallel burst. So writing is the slower, costlier part, and most APIs price it that way. Claude Opus 5.5, for example, is $4 per million input tokens and $20 per million output tokens on Anthropic's pricing page, five times more for every token it writes.

Put the whole journey together and you get five levers you control:

1. 1.Cap the output. Ask for 100 words when 100 words will do. Output is the expensive part.
2. 2.Trim the input. Send the relevant pages, not the whole manual. Cheaper, and a faster first word.
3. 3.Keep the start of the prompt identical so caching can work.
4. 4.Batch anything that can wait. Half price for patience.
5. 5.Use the smallest model that passes your quality check. We compare real job costs across models in our Claude models guide.

In our experience, these five levers matter more for a small business bill than which AI company you pick.

What is an LLM inference engine?

An inference engine is the software that takes the model's numbers and turns them into a service that can answer many requests quickly: scheduling, batching, managing the chips. You may have heard the names vLLM, Ollama or llama.cpp. Those are inference engines, and you can even run some of them on your own computer.

Most of them offer an OpenAI-compatible endpoint, which means the same request format works no matter which model is running underneath. Why should you care? Because it means you are not locked in. An app built against one model can often switch to another by changing one setting. And for sensitive data, a small model running on your own machine through something like Ollama never sends your files anywhere.

You do not pay for the brain. You pay every time it thinks. Make each thought shorter.

Do you need to understand all this?

Not the maths. You will never tune a GPU. But the shape of it explains your AI bill better than any pricing page: tokens in, tokens out, reading is cheap and parallel, writing is slow and costly, repetition can be cached and patience can be discounted.

That is also why the way you build your prompt and context matters so much, which we cover in prompt vs context vs harness engineering. Which of the five levers is your business not using yet? If you want us to look at your AI setup and find where it is overpaying, tell us what you are running.

Source for the step order: the YouTube explainer that inspired this post, at youtube.com/watch?v=615-yWyS1n0.