Every model has a hard limit on how much text it can consider at once. That limit — the context window — governs how long a document you can paste, how much of a codebase an agent can hold, and how long a conversation stays coherent before the model starts forgetting the beginning. Understanding what a token is, and what happens as you approach the limit, dissolves most of the confusion about why a model lost the plot.

What a token actually is

Models do not read characters or words. They read tokens: chunks of text produced by a tokenizer that splits language into common pieces. A token is often a whole short word, sometimes a word fragment, sometimes a space or a punctuation mark. A rough English rule of thumb is about four characters per token, or roughly three-quarters of a word — so 1,000 tokens is on the order of 750 words. The mapping is not exact and differs by tokenizer: numbers, code, emoji, and non-English text often use more tokens per character than plain English prose.

Two consequences follow. First, when a provider prices by the token or advertises a window in tokens, you can estimate but not perfectly predict how much of your text fits. Second, everything counts against the window — your instructions, the pasted document, the model's own reply, and, in a chat, the entire prior conversation. The window is a shared budget, not just a cap on your input.

A concrete sense of scale helps. The sentence you are reading is a few dozen tokens. A typical email runs a few hundred. A 20-page report is on the order of ten to fifteen thousand tokens, and a large codebase can reach hundreds of thousands. Because the model's reply draws from the same budget, a tool has to reserve room for the output: ask for a long answer on top of a nearly full window and either the input gets truncated or the response gets cut off. Leaving headroom for the reply is part of staying inside the limit.

Why the limit exists

The context window is a design and cost constraint, not an arbitrary cap. The attention mechanism at the heart of a transformer compares tokens against one another, and in the naive form that comparison grows with the square of the sequence length. Doubling the context does not double the work; it can quadruple it. Longer windows also need more memory to hold the intermediate state for every token in play. Labs spend real engineering effort — and you spend real money — to push windows out, which is why longer-context tiers usually cost more per call.

As of 2026, several frontier hosted models offer windows around one million tokens, and some open-weight models advertise far larger figures still. A million tokens is enough to hold a large codebase or a stack of long documents at once. But the advertised number is a ceiling on what the model will accept, not a promise of quality across the whole span.

What happens when you hit the limit

When your conversation or input exceeds the window, something has to give, and the behavior depends on the tool. Some interfaces refuse the request outright. Many chat apps silently drop or summarize the oldest messages to make room — which is why a long chat can suddenly forget a constraint you set an hour ago. Some agent frameworks compact the history, replacing older turns with a summary. In every case the model is not choosing to ignore you; the earlier text is simply no longer in front of it.

The practical tell: if a model that was following your rules starts violating one you set much earlier in a long session, you have probably pushed the relevant instruction out of the window. The fix is to restate the important constraint, or start a fresh session with the essentials up top.

The long-context tradeoffs

A bigger window is not free, and not only in dollars.

  • Cost and latency scale with what you send. You pay for input tokens, and a prompt stuffed with a million tokens of context costs and takes far more than a focused one. Sending an entire repository when the answer lives in two files is a common and expensive habit.
  • Attention degrades across long inputs. Independent testing has repeatedly found that models recall information best at the very start and the very end of a long context, and worst in the middle — the widely cited lost-in-the-middle effect. A fact buried at the 60% mark of a huge document may be missed even though it technically fit.
  • Usable context is shorter than advertised. Effective recall often falls off well before the stated limit. Many evaluations show quality dropping long before the window is full, so the number on the spec sheet is an upper bound, not a working target.

Practical habits

  • Put the most important instructions first or last, not buried in the middle of a long paste.
  • Send what is relevant, not everything you have. Curating the three pages the model needs beats dumping three hundred it does not — cheaper, faster, and usually more accurate.
  • Start fresh when a chat gets long and muddled. Carrying a bloated history forward degrades quality; a clean session with a short summary of what matters often works better.
  • Watch for the forgetting tell and restate constraints that have drifted out of range.
  • When the material genuinely will not fit or keeps changing, reach for retrieval rather than a bigger paste. Pulling in only the relevant passages at query time — the subject of the RAG guide — is often both cheaper and more accurate than filling a giant window.

The mental model to keep: a token is the unit the model reads and you pay for; the window is a fixed budget that every part of the exchange draws from; and the honest capacity of that budget is smaller than its advertised size. Work with those three facts and most why did it forget mysteries disappear.