From Thousands to Millions of Tokens: Do Context Windows Still Need Optimization?
In 2020, the GPT-3 research paper described a 2,048-token context window. The original paper documents that limit. By October 2026, OpenAI's GPT-6 Astra API model page lists 1,050,000 tokens. That is roughly a 500-fold increase in advertised capacity across two different models and eras—not a 500-fold increase in understanding, and not a promise that every AI app exposes the same limit.
So should we stop worrying about context? No—but the goal has changed. In a small window, optimization often meant fitting the task at all. In a large one, it means giving the model the *right* evidence, preserving room for its answer, and paying for and evaluating the result deliberately.
This is a source-backed explainer, not our own model benchmark. Model specifications below are snapshots of particular releases and surfaces. Historical announcements describe what was offered then; they are not current purchasing or availability advice.
What the window actually measures
A context window is the maximum amount of tokenized information a model can use in one request. It can include instructions, conversation history, retrieved documents, tool results and, depending on the model, image or other multimodal input. The budget also has to accommodate generated output and, for applicable models, reasoning tokens. OpenAI's API documentation explicitly distinguishes those components. A one-million-token window does not mean you can send one million input tokens *and* receive an unlimited answer.
Do not confuse this with long-term memory. An app may store a conversation, search old files or compact prior turns, but the model still receives a bounded working set for a particular request. The model's API limit, the application's upload or conversation limit, and what a specific subscription exposes can differ. Even the word *token* is not the same as a word, character or line of code; token counts vary with content and tokenizer.
How we got from 2K to a million-plus
| Milestone | Window and qualification | What changed |
|---|---|---|
| 2020: GPT-3 paper | 2,048 tokens for the models described in the paper | A few examples or short documents could fit; full books or repositories could not. |
| March 2023: GPT-4 launch | 8,192 tokens for GPT-4; 32,768-token GPT-4-32k had limited API access | Longer documents became practical, but access and cost differed by version. |
| May 2023: Claude expansion | Claude expanded from 9K to 100K tokens | Hundreds of pages could be submitted in one request in the announced offering. |
| November 2023: GPT-4 Turbo preview | 128K tokens, initially a preview for paying API developers | A much larger prompt became available without implying unlimited conversation state. |
| February–May 2024: Gemini 1.5 Pro | 1M tokens began as a limited developer/enterprise preview; Gemini Advanced gained a 1M window in May | Million-token context moved from an experimental offering toward broader product use. |
| October 2026 snapshot: GPT-6 Astra API model page | 1,050,000-token context window; separate 128K maximum output | Million-scale capacity is present in a current API model, with model-specific pricing and limits. |
This is a selected timeline, not a ranking or a claim that each vendor invented the next step. The entries mix research, previews, API releases and a consumer product; their access rules are deliberately labeled. The practical shift is real: many tasks that once required chopping source material into pieces can now fit in one request. But “fits” answers only the capacity question.
Capacity is not the same as effective use
Imagine placing a critical policy exception in the middle of a long collection of contracts. The model accepts every page. Will it find the exception and apply it correctly? That is an empirical question, not something a token-limit number can answer.
The 2023 *Lost in the Middle* study tested multi-document question answering and key-value retrieval. In the models and tasks studied, performance could be worse when relevant evidence sat in the middle rather than near the beginning or end. That result is not a universal rule for every model released in 2026, but it showed why accepted input length and usable information should be measured separately.
Later evaluations asked harder questions. RULER added multi-hop tracing and aggregation beyond simple “find this exact string” tests. NoLiMa reduced literal overlap between a question and its supporting passage, so a model had to infer the connection. Both papers report weaknesses in the particular models and tasks they evaluated. They do not establish a single effective context length for all modern models.
That distinction matters when reading a “needle in a haystack” result. Finding a unique sentence is useful; synthesizing conflicting clauses across 40 documents, spotting a dependency across a codebase, or maintaining a decision across a long agent session is different. Test the behavior that matches your job.
Why context optimization still pays
Relevance and interference. More material can supply missing evidence, but irrelevant material can also introduce competing facts, stale instructions and distracting examples. A curated packet with the task, the authoritative sources and their dates is often easier to verify than a giant undifferentiated dump. Do not remove a source merely to make the prompt shorter; remove it when it does not help answer the question.
Cost and cache behavior. API input is metered. As one concrete October 2026 example, the GPT-6 Astra model page says prompts above 272K input tokens use higher input, cache and output rates for the full request. That is a model-specific threshold, not an industry-wide rule. OpenAI's prompt-caching documentation explains how reusable prefixes can lower the cost of repeated context; caching does not make irrelevant material useful or automatically eliminate the first-request cost.
Latency—without the simplistic claim. Smaller input can help, especially at truly massive sizes, but it does not always dominate response time. OpenAI's latency guide says cutting a normal prompt in half may yield only a small latency improvement, whereas generating fewer output tokens is often more consequential. Measure end-to-end time for your own model and workload instead of assuming every token saved is a major speedup.
Output headroom and controllability. If history, tools and documents consume most of a shared context budget, there is less room for the answer or reasoning. Long raw logs also make it harder to identify which evidence supports a conclusion. Source labels, bounded excerpts and a clear question improve reviewability. When a document is sensitive, sending only necessary passages can also reduce needless exposure; follow your provider and organization's data policies regardless of window size.
A practical decision: send, retrieve, compress or split?
| Situation | Good starting approach | Check before trusting it |
|---|---|---|
| One coherent source, and relationships across the whole source matter | Send the full source if it fits, with the task and output format clearly stated | Can the answer cite the right passage, including one placed in the middle? |
| A large collection with a narrow question | Retrieve relevant passages first; include identifiers, dates and neighboring context | Did retrieval omit a conflicting or decisive passage? |
| A long, ongoing conversation or agent run | Keep a compact state of decisions, constraints, open questions and source pointers | Did the summary silently lose an exception or reverse a decision? |
| A task that depends on several independent analyses | Split into bounded analyses, then reconcile their evidence | Does the final synthesis preserve disagreements and provenance? |
Three examples make the trade-off concrete:
- Research: To compare two reports, start with both full reports if they fit and cross-document relationships matter. For a single statistic across hundreds of reports, search first and include the source pages and dates.
- Customer support: Retrieve the relevant, current policy and the customer's case facts; do not send the entire policy archive when one jurisdiction and product determine the answer. Check superseded policies and exceptions.
- Coding: Give the agent the task, repository instructions, affected interfaces, tests and relevant files—not a blind dump of every file and build log. Expand when dependencies or failing tests show a need. For a full agent workflow, see our context-engineering guide for Claude Code and Codex.
There is no fixed “best” percentage of the window to fill. A short context can omit critical evidence; a long one can be justified. The right approach depends on task success at the length you use, not on an arbitrary token threshold.
How to test whether your context strategy works
Choose a small set of representative tasks and compare at least two strategies: full context and a targeted/retrieved context. Keep the model version and evaluation criteria fixed. Include a fact near the beginning, middle and end; at least one question requiring synthesis rather than an exact-string lookup; and a case where two sources disagree. Record answer correctness, evidence citations, missing exceptions, input/output tokens, cache use, cost and wall-clock time. Review failures before deciding that the longer or shorter strategy wins.
Bottom line: We have moved from thousands to million-scale token windows, and that removes many old capacity constraints. It does not remove the need to decide what evidence matters, how to arrange it, what to leave out, and how to verify that the model used it. Context optimization in 2026 is less about squeezing text into a tiny box and more about making a very large box dependable.
