How LLM Pre-Training Actually Works: Next-Token Prediction Explained

Every large language model begins with a task that sounds too small for the result: given the tokens so far, predict the next token. No labels are written by hand, and no one tells the model what a fact or a verb is. Yet a model trained well on that objective can complete code, summarise documents, translate and answer questions it was never explicitly taught. This article walks through the pre-training loop — how text becomes training examples, how loss is computed, and how gradient updates turn next-token prediction into general capability.

Labels that create themselves

Most supervised learning needs annotated data, which is slow and expensive. Language modelling avoids that bottleneck because text is its own supervision. Hide the token that actually came next in any sentence and ask the model to predict it. The hidden token is the label, already there. This is self-supervised learning: the data supplies its own target.

That trick scales. A lab can assemble trillions of tokens from books, articles, documentation and code, clean and deduplicate them, and turn every position into a training signal. Radford and colleagues demonstrated the modern form in 2018 with the first GPT model: a 12-layer decoder-only Transformer with about 117 million parameters, pre-trained on BookCorpus — roughly 7,000 books — then fine-tuned for specific tasks. Its objective is still used today: causal language modelling, where each position may only look at what came before it.

One sequence, many predictions at once

Formally, a language model factorises a sequence's probability into conditional probabilities: first token, times second given the first, times third given the first two, and so on. Training maximises that product — equivalently, minimises its negative logarithm.

In practice, a training example is a window of tokens, often a few thousand long in modern models and 512 tokens in GPT-1. Suppose the window is the token sequence for a short phrase. The model is not asked one question about that window; it is asked a question at almost every position. At position one it predicts token two, at position two it predicts token three, and so on. The inputs are the sequence shifted one way and the targets are the same sequence shifted the other way.

Two mechanisms make that efficient. Teacher forcing means the model always conditions on true previous tokens, not its own guesses. The causal mask in self-attention prevents a position from attending to future positions. Together they let one forward pass produce a prediction at every position in parallel, without leaking answers. Generation is different — produce one token, append it, repeat — but training never waits for that loop.

From logits to loss: what the number actually means

At each position the model's final layer emits one raw score, called a logit, for every token in its vocabulary. A softmax turns those scores into probabilities that sum to one. The training loss for that position is the cross-entropy, which for a single correct token reduces to a simple expression: loss = −log(p_correct), where p_correct is the probability the model assigned to the token that actually came next.

Three values show how the function behaves:

If the model assigned p = 0.90 to the correct token, the loss is −log(0.90) ≈ 0.11 — right and confident, barely penalised.
If it assigned p = 0.45, the loss is −log(0.45) ≈ 0.80 — right as its top guess, but uncertain.
If it assigned p = 0.01, the loss is −log(0.01) = 4.61 — confidently wrong, and punished far more steeply.

The loss for a batch is the average of that quantity over all predicted positions. Averaging matters: a single number summarises millions of individual predictions, and backpropagation distributes its gradient back through every parameter that contributed to them.

Two reference points help you read a loss curve. First, a randomly initialised model assigns roughly equal probability to every token, so its expected loss is about log(V) for a vocabulary of size V. For a small 65-token character vocabulary that is about 4.17; for a vocabulary of 50,257 tokens, as in GPT-2, it is about 10.8. If a fresh model does not start near that value, something is wrong before training begins. Second, perplexity is simply exp(loss). It can be read loosely as the effective number of equally likely choices the model is hesitating between at each step. Lower is better, and because perplexity exponentiates the loss, small loss improvements near the end of training still correspond to visibly better predictions.

The training loop, step by step

Tokenise and pack. Text is split into subword tokens by a tokenizer such as byte-pair encoding, converted to integer IDs, and packed into fixed-length sequences. Special care goes into deduplication and quality filtering, because repeated or low-quality text teaches repeated or low-quality behaviour.
Embed. Each token ID is looked up as a learned vector, and information about its position in the sequence is added, so the model can distinguish the same token used in different places.
Run the Transformer blocks. The sequence passes through a stack of decoder blocks — masked self-attention followed by a feed-forward network, with residual connections and normalisation. Attention lets each position gather information from earlier positions that help predict what follows.
Predict. A final linear layer, often called the unembedding or language-model head, maps each position's representation to vocabulary-sized logits.
Score and average. Softmax and cross-entropy compare each position's distribution with the true next token, and the losses are averaged. In implementations this is done as one numerically stable fused operation rather than a separate softmax followed by a logarithm.
Backpropagate and update. Gradients flow backward through the whole network and an optimiser updates the weights. Large runs typically use AdamW, a learning rate that warms up from zero and then decays — commonly on a cosine schedule — and mixed-precision arithmetic, with activations in a lower-precision format such as bfloat16 while optimiser state is kept more precisely.
Repeat at scale. Steps are grouped into very large batches and distributed across many accelerators using combinations of data parallelism, tensor parallelism and pipeline parallelism. Gradient checkpointing — recomputing some activations on the backward pass instead of storing them — is often used to trade extra compute for memory.

Nothing in that loop changes as models grow. GPT-1 in 2018 and the much larger models that followed use the same objective and essentially the same loop. What changed was scale: GPT-3, for example, used 175 billion parameters and was trained on roughly 300 billion tokens. A useful rule of thumb for the compute involved is C ≈ 6 × N × D, where N is the parameter count and D is the number of training tokens — the factor reflects the cost of the forward and backward passes for each parameter on each token.

Why predicting the next token teaches more than the next token

The objective looks narrow, but the task is not. To assign high probability to correct continuations across a corpus, a model must internalise whatever makes text predictable: agreement and tense for grammar, who did what for facts, the next step for an argument, syntax and types for code. None of those abilities is labelled separately; each is a means to lowering next-token loss.

Scale makes that pressure systematic. Kaplan and colleagues showed in 2020 that test loss falls as a smooth power law in model size, dataset size and compute, when none is the bottleneck. Their exponents — roughly 0.076 for parameters, 0.095 for data and 0.050 for compute — mean improvements are predictable but diminishing. Later compute-optimal work revised how a budget should be split between parameters and tokens, but the lesson survived: pre-training loss is highly predictable, and capabilities tend to improve as it falls.

What pre-training does not give you

A freshly pre-trained model — a base model — is a powerful text continuer, not yet an assistant. It has not been taught to follow instructions, to refuse harmful requests, or to prefer helpful answers over merely plausible ones. Those behaviours come from post-training stages such as supervised fine-tuning on instruction examples and preference-based alignment, which reuse the same next-token machinery on much smaller, curated datasets.

Pre-training also carries limits. Teacher forcing creates exposure bias: in training the model only continues correct prefixes, while in generation it must continue its own output, so errors can compound. If benchmark questions leak into the corpus — data contamination — scores overstate real ability, which is why labs deduplicate against test sets. And because the objective rewards plausible continuations, not verified truth, a base model can produce fluent false statements. The breadth comes from predicting everything in the corpus; the failures come from prediction being the target.

Pre-training is not mysterious. Shift a corpus by one token, ask for a distribution at every position, score it with −log(p_correct), average, backpropagate, and repeat trillions of times. A single self-labelling objective, at sufficient scale, builds the foundation every modern language model stands on.