Tokens, transformers, context windows, why LLMs hallucinate, and how to choose between Claude, GPT, and open-source models — explained in plain English.
Large language models power the AI tools transforming software, yet how they actually work is often shrouded in either intimidating mathematics or hand-waving mystique. The truth sits in between and is genuinely understandable without equations. Grasping the core ideas — what these models really do, how they were built, and why they behave as they do — makes you far more effective at using them, because you understand their nature rather than treating them as inscrutable oracles. This is how large language models work, explained plainly.
At their core, large language models do something surprisingly simple to state: they predict the next piece of text given what came before. Trained on enormous amounts of text, a model learns the patterns of language so well that, given some input, it can predict what word is likely to come next, then the next, and so on, generating coherent text one piece at a time. Everything these models do — answering, writing, summarizing — emerges from this next-word prediction performed extremely well. Understanding that the fundamental operation is prediction, not comprehension in a human sense, is the single most clarifying insight about how they work.
It helps to picture a single answer being built step by step. Suppose you ask a model to finish a simple sentence. Internally, it does not plan the full sentence in advance and then type it out. It produces one piece, appends it, and repeats:
Input so far: "The capital of France is"
Step 1 adds: " Paris"
Input so far: "The capital of France is Paris"
Step 2 adds: "."
Input so far: "The capital of France is Paris."
Step 3 adds: (a signal meaning "I am finished")Each step looks at the entire text so far, not just the last word. That is why a model can keep a long answer consistent: when it writes the tenth paragraph, the first nine are part of what it is looking at. It is also why a small early mistake can snowball. If the model writes a wrong detail in the first sentence, every later step treats that detail as part of the established context and builds on it, the way a storyteller who misnames a character early on keeps using the wrong name for the rest of the tale.
Models do not actually work with words exactly, but with tokens — chunks of text that might be whole words, parts of words, or punctuation. Text is broken into these tokens, and the model predicts tokens rather than words. This detail matters in practice because it affects how models process text and how usage is measured and billed, typically per token. Understanding that the model's unit is the token, a piece of text not always identical to a word, explains some of the model's behavior and is important for practical concerns like cost, since you pay by the token. It is a small technical reality with real practical implications worth knowing.
A human reader sees letters and words; the model sees a fixed vocabulary of text chunks that were chosen before training, based on which chunks show up often. Common words tend to be a single token. Rarer or longer words get split into several familiar pieces. An illustrative split might look like this (the exact boundaries differ from one model to another, so treat this as a picture of the idea rather than real output):
"The cat sat." -> [The] [ cat] [ sat] [.]
"unbelievably" -> [un] [believ] [ably]
"myproject_config.yaml" -> [my] [project] [_config] [.yaml]Once you know the model sees chunks rather than letters, several puzzling behaviors make sense:
These models are built by training on vast quantities of text — much of the written internet and more — during which the model adjusts itself to get better and better at predicting the next token across all that text. Through this process over an enormous amount of data, it learns the patterns, structures, facts, and styles present in the text. The result is a model that has absorbed a great deal of what was in its training data. Understanding that the model's capabilities come from learning to predict text across a massive corpus — that its knowledge and abilities are patterns extracted from training data — explains both where its impressive abilities come from and where its limitations originate.
Training is easiest to picture as a guessing game. Take a sentence from the training text, hide the next piece, and ask the model to guess it. At first the model's guesses are essentially random. After each guess, it is shown the real answer and nudged slightly so that next time, in a similar situation, the right answer becomes a bit more likely. Repeat that across an enormous amount of text and the tiny adjustments add up to something remarkable.
An important nuance is that the model learns patterns in language, not a database of facts. It does not store and look up facts the way a database does; rather, it has learned statistical patterns so rich that it often produces correct information, because correct information was common in its training. But this also means it can produce plausible-sounding but incorrect information, because it is generating likely text, not retrieving verified facts. Understanding this distinction — patterns versus a fact store — is crucial, because it explains why models are often right but sometimes confidently wrong, which directly shapes how you should and should not rely on them.
Imagine a friend who has read an astonishing number of books but never took notes and cannot look anything up. Ask them about something famous and discussed everywhere, and they will almost certainly be right, because they have encountered it again and again. Ask them about something obscure — like the exact release date of a minor software version — and they will still answer fluently, but now they are reconstructing from vague impressions. Sometimes the reconstruction is right. Sometimes it merely sounds right.
The tendency of models to sometimes generate false information, often called hallucination, follows directly from how they work. Because they generate plausible text based on patterns rather than retrieving verified facts, they will sometimes produce something that sounds right and fits the pattern but is simply not true. The model is not lying or malfunctioning; it is doing exactly what it does — producing likely text — in a case where the likely text happens to be wrong. Understanding that hallucination is an inherent consequence of next-token prediction, not a bug to be fully eliminated, is essential to using these models responsibly, with verification where correctness matters.
A model considers a limited amount of text at once when generating — this is its context window. Everything you provide and everything generated so far, up to that limit, is what the model uses to predict what comes next. It does not remember beyond a conversation's context or learn from your interactions; it works from what is in the current context. Understanding the context window explains both how the model uses what you give it and its boundaries — why it cannot recall things outside the current context and why providing relevant information in the context is how you make it work with specific data. It is a practical concept that shapes how you interact with the model.
If the model does not remember between requests, how does a chat app follow a long conversation? The answer is simple and a little surprising: the application sends the whole conversation again every time. Each new message you type is added to the transcript, and the full transcript goes to the model, which then predicts the next reply. Conceptually, each request looks something like this:
[Instructions] You are a helpful assistant for myproject.
[User] What does the deploy script do?
[Assistant] It copies the build to /srv/app and restarts the service.
[User] And where does it log errors?
[Assistant] ...the model predicts from here...The "memory" lives in the application, not in the model. This has practical consequences. Longer conversations cost more, because the whole history is processed each time. Very long conversations eventually hit the context limit, at which point the application must trim or summarize older turns — which is when a chat tool can seem to "forget" something you said an hour ago.
A natural question is whether these models truly understand, and the honest answer is nuanced. They do not understand in the human sense — they have no consciousness, beliefs, or genuine comprehension. Yet they exhibit behaviors that look remarkably like understanding, because they have learned the patterns of language and reasoning so thoroughly. The practical stance is to treat them as extraordinarily capable pattern-based text generators that often behave as if they understand, while remembering they do not actually know things. Holding this view keeps you from both underestimating their usefulness and overestimating their reliability, which is exactly the balanced perspective that makes you effective with them.
Because a model's knowledge comes from its training data, it knows nothing about events or information after its training was completed — its knowledge has a cutoff. Ask about something recent, and the model either does not know or may fabricate. This is why providing current information in the context is how you work with up-to-date data, and why models are paired with retrieval systems for current knowledge. Understanding the knowledge cutoff explains a key limitation and its solution: the model's built-in knowledge is frozen at training time, so for anything current or specific, you supply it in the context rather than expecting the model to already know it.
The phrase "paired with retrieval systems" sounds technical, but the idea is ordinary. Before the model answers, a separate piece of software searches a collection of up-to-date or private documents — your help center, your product docs, today's news — for passages related to the question. Those passages are placed into the context, and the model is asked to answer using them. It is the difference between asking the well-read friend to recall something from memory and handing them the relevant page first.
Knowing that the model predicts text based on what came before explains why prompts matter so much. The text you provide sets the context that shapes what the model predicts next, so a clear, well-structured prompt that establishes the right context leads to better output, while a vague one leaves the model to fill in ambiguity. You are, in effect, steering the prediction by what you put before it. Understanding that prompting works by shaping the context the model predicts from — not by issuing commands to a thinking entity — demystifies why prompt quality matters and makes you better at writing prompts that reliably get the results you want.
Here is a vague request and a clearer version of the same request. Read them as the model would: as the beginning of a document it must continue.
Vague:
Write something about our outage.
Clearer:
You are writing a short status update for customers of app.example.com.
Facts (use only these):
- Between 09:10 and 09:45 UTC, logins failed for some users.
- Cause: an expired certificate on an internal service.
- It is fixed; no data was lost.
Tone: calm, plain English, no jargon. Length: under 120 words.
End with a one-sentence apology.The first prompt leaves nearly everything open: audience, length, tone, and the facts themselves. Faced with that ambiguity, the model fills the gaps with the most generic plausible outage announcement — and may invent details such as a cause or a duration. The second prompt narrows the range of likely continuations to almost exactly what you want.
This understanding translates into practical wisdom. Because models predict plausible text from patterns, verify their output where correctness matters and do not trust them for guaranteed facts. Because they have a knowledge cutoff and context window, provide current and specific information in the context. Because prompts shape prediction, write clear prompts. Because they generate rather than retrieve, expect variability. Each practical guideline for working with models follows from how they actually work. Understanding the mechanism is not academic — it directly informs how you build reliable AI features, which is precisely why grasping how these models work, even without the math, makes you genuinely more effective with them.
When a model predicts the next token, it does not pick a single certain answer but works from probabilities across possible next tokens, and there is usually some randomness in which is chosen. This is why a model can give different responses to the same prompt — the generation is not perfectly deterministic. Settings can make it more or less random, more focused or more varied. Understanding that generation involves probabilistic choice among likely continuations explains the variability you observe, and why you can tune toward consistency or creativity. It also reinforces that the model is producing likely text rather than computing one fixed correct output, which is central to its nature.
Most model APIs expose a setting usually called temperature that adjusts this choice. Think of it as a dial. Turned low, the model sticks closely to the most likely candidates and its answers become more consistent from run to run. Turned higher, less likely candidates get more of a chance, and answers become more varied and sometimes more surprising — and also more likely to wander off course. The exact range and default differ between providers, so check your provider's documentation rather than assuming a number.
One caution for beginners: turning randomness all the way down makes output more repeatable, but it does not make it more correct. A model can give the same wrong answer every time. Low randomness buys consistency, not truth, and even at the lowest setting you should not assume perfectly identical output on every run.
Beyond the initial training on vast text, models are often further shaped to be more helpful, follow instructions, and behave safely — a refinement on top of the base capability. This is why modern models are good at following your instructions rather than just continuing text, and why they tend to respond helpfully. You do not need to do this yourself; the models you use through an API have already been refined this way. Understanding that the models you interact with have been specialized to be helpful and instruction-following — not just raw text predictors — explains their cooperative, useful behavior and why prompting them with clear instructions works as well as it does.
To see why this refinement matters, imagine giving a raw model — one that has only done the initial training — the text "What is the best way to learn a new programming language?" A raw predictor asks itself what text usually follows a line like that. On the internet, a question like this might appear in a list of frequently asked questions, so a perfectly reasonable continuation is... more questions: "What is the best way to learn a second spoken language? How long does it take to become fluent?" It is continuing the document, not answering you.
The refinement stage teaches the model a different habit. It is shown many examples of a request followed by a helpful, well-behaved response, and it is further guided by feedback about which responses people judged better. Afterward, the most likely continuation of a question is an answer, the most likely continuation of an instruction is carrying it out, and requests for clearly harmful help tend to be declined. An analogy: the initial training is like a broad education, and the refinement is like on-the-job training for a specific role — the knowledge comes mostly from the first, the manners and working style mostly from the second.
One striking aspect of these models is that capabilities like translation, summarization, and answering questions were not explicitly programmed — they emerged from learning to predict text across an enormous, varied corpus that included all of these things. Because the training text contained examples of every kind of language task, the model learned to perform them. This emergence from general training is why a single model can do so many different things. Understanding that the model's broad abilities arise from comprehensive training rather than from being built task by task explains both its remarkable generality and why its behavior can sometimes surprise even those who built it.
Pulling it together, a practical mental model is this: a large language model is an extraordinarily capable text predictor that has absorbed the patterns of language and knowledge from massive training, generates plausible continuations probabilistically, works only from its training plus the context you give it, and has been refined to be helpful. It is not a thinking being and not a fact database, but a powerful pattern-based generator. Holding this mental model lets you anticipate the model's behavior — its strengths, its variability, its tendency to fabricate, its dependence on context — which is exactly what makes you effective at building with it, turning an opaque tool into one whose nature you genuinely understand.
When a model does something strange, the mental model from this tutorial almost always points to the cause.
| Symptom | Likely cause | What to try |
|---|---|---|
| Invents a function, setting or reference that does not exist | Plausible pattern, missing information | Put the real documentation or code in the context; allow "I don't know" |
| Gets something recent wrong or claims it has not happened | Knowledge cutoff | Supply current information, or use a tool with search |
| Forgets something said early in a long chat | Older turns trimmed to fit the context window | Start a fresh conversation with a short summary of what matters |
| Different answers to the same question | Probabilistic generation | Lower the randomness setting; make the prompt more specific |
| Miscounts letters or fumbles long arithmetic | Works in tokens, not letters or digits | Use ordinary code or a calculator tool for exact counting and arithmetic |
Large language models, demystified, do one thing at their core: predict the next piece of text — the next token — given what came before, having learned the patterns of language from training on enormous amounts of text. Everything they do emerges from this prediction done extremely well. They work with tokens rather than exact words, which matters for cost; they learn statistical patterns rather than storing facts, which is why they are often right but sometimes confidently wrong; and that pattern-based generation is exactly why they hallucinate plausible falsehoods. They consider only a limited context window and have a knowledge cutoff at training time, so you supply current and specific information in the context. They do not understand in the human sense but behave remarkably as if they do. Crucially, prompts work by shaping the context the model predicts from, which is why prompt quality matters so much. Grasping these ideas — without any mathematics — turns the model from an inscrutable oracle into a tool whose behavior you can anticipate, which is what makes you genuinely effective at building reliable features with it.
Found a mistake in this tutorial? Let us know and we will fix it.