LLMs Explained: Tokens, Embeddings, Transformers and More

Summary

CJ of Syntax explains how a large language model (LLM) turns a prompt into a response, tracing the process from early language and neural-network research through tokenization, embeddings, transformers, and model tuning. The central idea is that an LLM generates text by repeatedly predicting a likely next token—but those predictions depend on a large system of learned numerical representations and operations.

The evolution from chatbots to LLMs

• Early language prediction: CJ begins with Claude and Betty Shannon’s text-guessing exercise, which illustrated how context makes written language more predictable.
• Rule-based chatbots: Early programs such as ELIZA, PARRY, ALICE, and SmarterChild relied on hand-written rules, patterns, scripts, or templates. They could seem conversational without understanding what users said.
• A different mechanism: Modern LLMs are not simply more elaborate rule-based chatbots. They learn statistical patterns from large collections of text and use those patterns to generate responses.

How the process works

1. A prompt becomes model input

A neural network cannot directly process a block of text, so the prompt must first be converted into numerical inputs.

A neural network consists of connected units arranged in layers. The connections have learned numerical values called weights. During training, the model makes predictions, measures how wrong they are, and adjusts its weights through a process such as backpropagation.

2. Text is divided into tokens

A token is a unit of text used by a model. It may be a whole word, part of a word, punctuation, whitespace, or multiple pieces for an emoji.

The video demonstrates byte pair encoding (BPE), which repeatedly merges frequently occurring adjacent pieces to build a vocabulary. Common text may be represented by a single token, while rare words can be split into smaller pieces.

Token counts matter because they affect API costs and how much text fits into a model’s context window.

3. Tokens are converted into embeddings

Each token has an ID, but the ID alone does not describe its meaning. An embedding represents a token as a vector—a list of numbers that places it in a high-dimensional space.

The video connects embeddings to the idea that a word’s meaning depends on the words around it. Words used in similar contexts tend to have nearby vectors. Word2vec illustrates this idea, including the well-known analogy king − man + woman ≈ queen. These relationships emerge from patterns in training text rather than being explicitly programmed.

The demonstration trains vectors from nearby-word pairs and uses vector similarity to find related words.

4. A transformer uses context

Modern LLMs described in the video use transformer architectures. A transformer block has two main components:

• Self-attention: Each token produces query, key, and value vectors. Comparing queries and keys helps determine which other tokens are relevant; their values contribute information to the result.
• Feed-forward network: Each token’s context-enriched representation is further processed.

Multi-head attention repeats attention calculations in parallel. Different heads may learn different patterns during training. Transformers stack many blocks so later layers can build on earlier processing. The video presents syntax, meaning, and reasoning as tendencies associated with different depths, rather than as explicitly assigned functions.

5. The model predicts and selects a next token

The transformer produces scores, called logits, for possible next tokens. Softmax converts those scores into a probability distribution.

The model then selects a token from that distribution:

• Temperature affects how predictable or varied the sampling is. Lower values make results more predictable; higher values make them more variable.
• Top-p restricts sampling to a subset of likely tokens according to a probability cutoff.

The selected token is appended to the input, and the process runs again. This is autoregressive generation: a response is built one token at a time, rather than planned as a complete paragraph in advance.

6. Conversation history and context windows

A deployed model typically receives more than the latest user message: it may also receive system instructions and conversation history. The video describes conversation history as the model’s practical memory between turns. The model itself does not retain conversational memory between requests unless that history is sent again.

The context window is the amount of tokenized material the model can consider at once. If the available context is exceeded, some material may not be included. The presenter also notes that long contexts can make it harder for a model to use earlier information reliably.

Training, alignment, and tools

• Pre-training: A model learns patterns from large text collections, such as web pages, books, code, and other material. The presenter characterizes the resulting base model as a sophisticated form of autocomplete.
• Fine-tuning: The model is trained further on curated examples, such as question-and-answer pairs, to encourage more useful chatbot-style responses.
• RLHF: In reinforcement learning from human feedback, people rate or rank responses. Those judgments help shape the model’s behavior, though what counts as a good response depends on the training process and its evaluators.
• Tool calling: A model can be given descriptions of available tools and trained to produce structured requests, such as a JSON object asking for a web search. A separate application, or harness, executes the request and returns the result to the conversation. The model generates the call; the surrounding software performs the action.

Demonstrations and closing perspective

CJ builds small examples of a rule-based chatbot, an XOR neural network, a BPE tokenizer, a word2vec-style embedding model, and a simple transformer trained on short children’s stories. The demonstrations show that even small models can learn patterns, but they are tiny compared with production systems.

CJ’s concluding view is that LLMs are powerful statistical pattern matchers, not necessarily replicas of human thought. Because they learn from text, he argues that their abilities should not automatically be equated with human intelligence. He describes them as a potentially different, “alien” form of intelligence and stresses that transformers may not be the final AI architecture.

Speakers and sources featured

Speaker

• CJ — Presenter and host of Syntax; explains the concepts and walks through the code demonstrations.

Historical figures and researchers mentioned

• Claude Shannon and Betty Shannon
• Alan Turing
• Joseph Weizenbaum
• Kenneth Colby
• Warren McCulloch and Walter Pitts
• Frank Rosenblatt
• Marvin Minsky and Seymour Papert
• David Rumelhart, Geoffrey Hinton, and Ronald Williams
• Philip Gage
• Rico Sennrich, Barry Haddow, and Alexander Birch
• Gottlob Frege and J. R. Firth
• Tomas Mikolov
• Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio
• Jakub Uszkoreit
• John McCarthy and Nathaniel Rochester

Papers, projects, and examples discussed

• Alan Turing’s paper Computing Machinery and Intelligence and the Turing Test
• ELIZA, PARRY, ALICE, and SmarterChild
• Perceptrons by Marvin Minsky and Seymour Papert
• Research on backpropagation by David Rumelhart, Geoffrey Hinton, and Ronald Williams
• Byte pair encoding and its use in neural machine translation
• Word2vec and its skip-gram and CBOW approaches
• Attention Is All You Need
• The Dartmouth Summer Research Project on Artificial Intelligence
• Common Crawl
• ChatGPT, Claude, Gemini, and GPT-OSS 120B as model examples