Why AI Tokens are so Expensive - Computerphile

Summary of key technological concepts and token-cost analysis

What a token is (core concept)

• A token is either a whole word or a piece of a word.
• Punctuation and symbols are also tokens (e.g., full stops, curly brackets).
• For languages like Chinese, a character may map to 1–2 tokens.
• Exact token boundaries depend on the tokenizer used by the model provider.
• Tokenizers are based on frequency:
  • Common strings (like “the” or “a”) are likely single tokens.
  • Rarer text is split into smaller token pieces.
• The tokenizer acts like a string parser in front of the LLM:
  • Token IDs are then converted into embeddings (high-dimensional numeric vectors learned during training).

Why tokenization isn’t “just for code”

• Even if a model is used for JavaScript, providers typically keep a large shared vocabulary of tokens.
• Models are generally:
  • trained on general data, then
  • refined (fine-tuned / instruction-tuned) for domains like coding.
• This is described as more cost-effective than training only on code.

Autoregressive generation drives cost

• LLMs are autoregressive, predicting one next token at a time.
• Each new token requires re-processing the entire context so far:
  • system prompt
  • user text
  • prior generated “thought”/outputs
• This repeated re-processing makes inference inefficient, especially for agentic coding where the model may:
  • reason for a long time, and
  • repeatedly call tools.

Concrete cost structure (input vs output tokens)

• Cost is tied to how many tokens are processed and generated.
• The video distinguishes:
  • Input tokens: everything sent into the model (e.g., system prompt + user query + prior intermediate content)
  • Output tokens: tokens the model emits (including hidden reasoning and tool-call structures)
• A simplified loop is described:
  1. The model takes system + user prompt
  2. Produces some “thought” tokens
  3. Feeds those generated tokens back in
  4. Repeats for each step → the “context length” grows, increasing compute.

KV caching and why it only helps sometimes

• The speaker explains KV caching (key/value caching) to avoid recomputing transformer context every step.
• KV caching helps most when context is reused efficiently.
• When context is huge (massive documents, long histories, many tool-read contents), caching can be less effective because:
  • GPU memory limits
  • caches may expire, forcing prefilling again when context “drops”
• Result: cost can spike with large context windows.

Why agentic coding explodes token usage

• Chatbots: typically answer using the current user prompt with relatively small context.
• Coding agents may:
  • read multiple files,
  • repeatedly run tool calls (e.g., “read this file”, “apply patch”),
  • incorporate file contents back into the model’s next reasoning step.
• This repeatedly re-includes large file contents across many iterations, and each tool step increases the re-processed context, multiplying cost.

Example: tool-using coding agent token blow-up

The video describes a scenario where:

• A system prompt exists (thousands of tokens)
• The user asks for a bug fix (hundreds of tokens)
• The agent “thinks” (thousands of output tokens)
• It then makes tool calls to read files:
  • tool call output tokens + thousands of file-input tokens
• It repeats “think → tool call → read file → think” multiple times before producing a patch.

Even a “simple” bug fix involving a couple of files can reach tens of thousands of tokens processed due to multiple steps.

Real-world observed “per token” incentives

• Some companies use internal metrics like token usage to gauge “moving to AI.”
• The speaker argues this creates bad incentives, such as:
  • asking longer questions,
  • looping reasoning,
  • selecting costlier behaviors/models.
• They note agentic usage was often subsidized under flat monthly pricing.
• Once pricing shifts to pay-per-token/credits, costs become painful.

Product/pricing changes mentioned

• Example: GitHub Copilot changed from a request-based monthly cost to an AI token credit model, reducing how much can be done per month.
• They also mention providers like Anthropic, which uses caps for premium tiers—requiring waiting when limits are hit.

Where savings / “reasonable use” is suggested

• The speaker suggests cheaper, practical patterns:
  • small, succinct questions
  • quick fixes
  • code completion
• They are skeptical about broad adoption of heavy agentic workflows due to sustainability concerns for typical companies.

---

Main speakers / sources

• Computerphile / primary narrator: Lewis (mentioned as “Lewis”; the host appears to be Mike/Computerphile presenting alongside him)
• Lewis: co-speaker mentioned during a prior tokenization example
• Video title/source: “Why AI Tokens are so Expensive - Computerphile”