The model doesn’t read letters

First, the text is cut into tokens: pieces from the model’s vocabulary. They can be whole words, parts of words, spaces or digits. After that, the model sees only the IDs of these pieces.

“Tally” has a tiny vocabulary: twelve tokens, the digits 0 to 9, + and =. For it, 17+25= is six tokens. Modern models have vocabularies of 100 to 260 thousand tokens.

A word becomes a point in space

A token’s ID is a row number in a big table. The model takes that row: a list of 32 numbers. Such a list is called an embedding vector.

During training, these numbers shift so that words with similar roles end up close together. Directions in this space take on meaning: “gender”, “plural”, “capital city”.

Word order matters

Attention, the model’s main node, can’t see order by itself. To it, both sentences below are the same three tokens.

So a second vector, one that encodes position, is added to each token’s vector. “Tally” and GPT-2 simply learn a separate vector for each position: on the scene, that is the table wpe.

Modern models (Llama, Qwen, DeepSeek, Mistral) use RoPE. The dot product of the rotated query and key depends on their content and the distance between the tokens, but not on their absolute positions. Scaling the angles stretched the context window from 2 thousand to 128 thousand tokens and beyond. Qwen3.8-27B (2026) has a base context window of 262 thousand tokens, and YaRN angle scaling stretches it to about 1 million.

First, even out the scale

Before each node, the vector is normalized: subtract the mean, divide by the spread, then multiply by a learned γ and add β. Without this, the input scale of every node would depend on depth, and a deep network would be hard to train.

Each token looks for the neighbors it needs

Self-attention decides which earlier tokens matter for the current one. Each token’s vector is multiplied by three matrices, giving three shorter vectors of 8 numbers each:

One token’s query is multiplied by the keys of all earlier tokens and by its own key. The product is divided by the square root of the key length (by √8 in “Tally”) so the scores don’t get too large. A high score means a match. Then softmax turns the scores into shares that add up to 100%, and the values are mixed in those shares.

Who looks at whom

Do the same for every token, and you get a square attention matrix. Rows show who is looking; columns, at whom. The arcs above the tokens show the same row more clearly.

The upper-right triangle is empty. This is the causal mask: a token can’t see future tokens. Otherwise, during training the model would simply peek at the answer.

Sixteen ways of looking

One block of “Tally” has four attention heads, each with its own Q, K and V. They work in parallel, and in a trained model each one learns to look at something different. “Tally” was never trained, so for three heads in the first block the query and key weights were set by hand, the way they might turn out after training: the first looks at the previous token, the second at the same digit place in the previous number, the third at the “=” sign. The weights of the other thirteen heads are synthetic, so their patterns are random. Here are all sixteen heads on your sum:

The outputs of the four heads (8 numbers each) are joined into a vector of 32 numbers and multiplied by the matrix Wo. The result doesn’t replace the input vector; it is added to it.

Think over each token separately

Attention mixes information between tokens. The MLP processes each token separately: it widens the vector fourfold, passes it through GELU, a nonlinear function, and compresses it back. GELU barely changes large positive numbers and brings negative ones almost to zero, so each neuron fires only on some of the inputs.

Most of the model’s parameters are here. Interpretability research shows that individual neurons and directions in the MLP correspond to specific concepts, which is why the MLP is often called the model’s memory.

Blocks add rather than overwrite

Normalization, attention, addition, normalization, MLP, addition — that is one transformer block. Each token’s vector flows through the blocks as if along a highway, and each node only reads from it and adds its own correction.

This is what makes it possible to stack dozens and hundreds of blocks: the signal and the gradient pass straight through without fading. “Tally” has four blocks; GPT-3 has 96, Kimi K3 has 93 and Mistral Medium 3.5 has 88.

A probability for every word in the vocabulary

The last vector is multiplied by a “vector × vocabulary” matrix. This gives a logit, a score, for every token in the vocabulary, and softmax turns the scores into probabilities. The model doesn’t pick a word; it gives a distribution.

A low temperature sharpens the distribution: answers are predictable. A high one flattens it: more variety, more risk. Top-p cuts off the tail: it keeps the most likely options that together make up p.

The answer comes one token at a time

The chosen token is appended to the end, and the whole model runs again, now on a longer sequence. Two answer digits mean two full passes through the network; a hundred tokens, a hundred passes. Every chatbot writes its answers the same way.

A whole LLM in one diagram

The largest language model differs from “Tally” in size, data and training, but not in principle. If you understand how a few tens of thousands of numbers add 17 and 25, you understand what happens inside any chatbot.

The error as a single number

Training has a very simple goal: predict the next token in real text. The model sees “The capital of Ukraine is”, outputs probabilities, and the correct answer is known: “Kyiv”.

The error is computed as −log p, where p is the probability the model gave the correct token. Confidently right: the loss is near zero. Confidently wrong: the loss is large.

This loss is averaged over trillions of tokens of training text. Training as a whole is a search for the numbers in the tables that minimize the average loss.

Step by step, downhill

How do you find the best numbers? For each parameter, you compute the gradient: which way and how much the loss changes if the parameter shifts slightly. Then all parameters are shifted at once toward lower loss. The gradients for all parameters are computed in one pass by an algorithm called backpropagation.

From internet text to an assistant

The chatbot you talk to goes through three stages. The architecture is the same in all of them; the data and the goal change. After them, the model can be fine-tuned for your task: on the diagram, this step is dashed.

Fine-tuning for your task is the same second stage, only on your own examples. Full fine-tuning changes all of the model’s parameters. For GPT-3 175B, this takes 1.2 TB of GPU memory, and the result is one more copy of the weights, 350 GB in size.

LoRA (low-rank adaptation, 2021) keeps the d × d weight matrix W unchanged and trains only an update to it. The update is the product of two narrow matrices: B of size d × r and A of size r × d. The product B · A has the same size as W and is added to W. The number r is the rank of the update; in the LoRA experiments on GPT-3, it ranges from 1 to 64. Instead of d × d numbers, 2 × d × r are trained. At the start, B is filled with zeros, so the model answers exactly as before fine-tuning. After training, B · A is added to W once, and answers are no slower.

The authors of LoRA added updates with r = 4 only to the Wq and Wv matrices of GPT-3 175B. Trainable parameters fell to 18.9 million from 175,255.8 million, 9,285 times fewer. The paper rounds this to 10,000 times. Training needed 350 GB of GPU memory instead of 1.2 TB. The file with the updates takes 35 MB instead of 350 GB. A hundred fine-tuned versions of the model take 354 GB: one model and a hundred update files. A hundred full copies would take 35 TB.

QLoRA (2023) stores the unchanged model in a 4-bit format and trains the LoRA updates in 16-bit. This way, a model with 65 billion parameters was fine-tuned on a single GPU with 48 GB of memory. Regular 16-bit fine-tuning of such a model needs more than 780 GB.

Changing the weights isn’t always needed. Often a prompt is enough: an instruction and a few examples in the request itself. If the model lacks fresh or private knowledge, RAG helps: the system finds the relevant document chunks and adds them to the request’s context. How this works is shown in the section “Memory that grows with every word”.

From tens of thousands to trillions

On the scene, each large model is a tower of cubes: one cube is a billion parameters, and the height is proportional to size. Orange marks the parameters that work on every token: in a dense model that’s the whole tower; in a mixture of experts, the lower part (gathered at the bottom for clarity; in reality these weights are spread across all layers). “Tally” and GPT-2 small are smaller than one cube, so they are shown in the magnifier. Press “Zoom out”, and the camera will travel from “Tally” to Kimi K3, with each step showing how many times larger a model is than the previous one.

Size isn’t all that matters; data does too. GPT-3 read 300 billion tokens, Llama 3 over 15 trillion and DeepSeek-V4-Pro (2026) over 32 trillion. DeepSeek-V4.1-Flash was trained on 45 trillion tokens of text and images. The Chinchilla study (2022) showed that, for a fixed compute budget, the optimum is ≈20 tokens per parameter. Modern models are trained much longer, because a smaller model is cheaper to run.

Memory that grows with every word

During generation, the keys and values of earlier tokens don’t change. Instead of being recomputed, they are stored: this is the KV cache. On the scene, everything in it is highlighted. The cache grows with every token. When a model serves many long conversations at once, the cache takes up more GPU memory than the weights themselves.

The context grows not only from the conversation but also from search. This is how RAG (retrieval-augmented generation) works. Documents are cut into chunks in advance, and an encoder — a separate small model — turns each chunk into a vector. The question is turned into a vector too, and the search takes a few of the nearest chunks. They are added to the context before the question. The model answers based on these chunks and can cite them. To the model, the retrieved text is ordinary tokens: each one costs computation, and their keys and values are stored in the KV cache.

The model’s weights don’t change. For the model to answer from fresh or private documents, it is enough to update the index. In the paper that named RAG (Lewis et al., 2020), the index was Wikipedia: 21 million chunks of 100 words. The authors took 82 world-leader posts whose holder changed between 2016 and 2018. With the 2016 index, the model correctly named 70% of the 2016 leaders; with the 2018 index, 68% of the 2018 leaders. With the other year’s index, only 4–12%. The knowledge was updated by swapping the index, without fine-tuning.

A trillion parameters, a fraction at work

On the scene, “Tally” has been given a mixture of experts: the MLP of each block is cut into 8 experts. The router is a small matrix that works out, for each token, which two experts are most useful to it; only they light up. On the scene, the tokens take turns, and for each one the router picks its own pair of experts. GLM-5.3 (August 2026) does the same, only with 256 experts plus one shared, and 8 switch on per token. Kimi K3 has 896 experts: 16 switch on per token, plus two shared ones.

The knowledge of a giant at the per-token cost of a mid-sized model. Almost all large open-weights models of 2026 are built this way: DeepSeek-V4, Kimi K3, Qwen3.8, GLM-5.3, MiniMax-M3. In DeepSeek-V4-Pro, 49 billion of its 1.6 trillion parameters work on each token, about 3%. But all the experts have to sit in memory, so such a model saves compute, not memory.

Think before you answer

The architecture is the same; what has changed is how the model uses generation. Before answering, it writes out a long chain of thought: it breaks the problem down, checks itself, backtracks. Every thinking token is one more full pass through the network.

Such models (OpenAI o1 in 2024, DeepSeek-R1 in 2025 and their successors) go through an extra stage of reinforcement learning. The model is rewarded for a correct final result on problems that can be checked automatically: math, code, logic. The model learns to build chains of reasoning on its own. In 2026, reasoning became standard in flagship models. In Claude Opus 5.5, Kimi K3 and GLM-5.3, it can’t be turned off; you can only choose its effort level, from low to max.

Before → after

Every notable change in architecture and training: what was replaced, by what and where it is used. Entries are ordered by year. New model releases, records and Ukrainian projects are in the “News” feed.