Context Engineering: Your Prompt Isn't the Problem
Introduction
Every developer building with LLMs hits the same wall. The demo works. You add three more tools, a chat history, and a few documents — and suddenly the model starts ignoring instructions, calling the wrong tool, or confidently making things up.
The instinct is to blame the prompt. So you rewrite it. You add IMPORTANT in caps. You add "do not hallucinate." It works for one query and breaks on the next.
The prompt was never the problem. The problem is everything around the prompt — the tool results, the history, the retrieved chunks, the system instructions — all of it competing for the same finite window. That's context engineering: deciding what the model sees, in what order, and what gets thrown away.
What actually reaches the model
When you call an LLM in an agent loop, your carefully written prompt is a small fraction of the payload. A realistic request looks more like this:
// what you think you're sending
const request = "Summarize this user's spending"
// what you're actually sending
const request = [
systemPrompt, // ~400 tokens
toolDefinitions, // ~1,200 tokens (12 tools, full JSON schemas)
conversationHistory, // ~6,000 tokens and growing every turn
retrievedDocs, // ~4,000 tokens (top-10 chunks, mostly irrelevant)
rawToolResults, // ~9,000 tokens (an unfiltered API response)
userQuery, // ~15 tokens
].join("\n")The user's actual question is 0.07% of that payload. Then we act surprised when the model doesn't prioritize it.
A large context window doesn't fix this. Window size is a limit, not a target. Filling 200k tokens because you can is like allocating a 2GB array because you have the RAM.
Rule 1: Treat the window as a budget
Before optimizing anything, measure it. You can't fix what you don't count.
// context-budget.ts
type Segment = { name: string; content: string; priority: number };
const BUDGET = 32_000; // tokens we're willing to spend
const estimate = (s: string) => Math.ceil(s.length / 4); // rough, good enough
export function assemble(segments: Segment[]): string {
// highest priority survives when we run out of room
const sorted = [...segments].sort((a, b) => b.priority - a.priority);
let spent = 0;
const kept: Segment[] = [];
for (const seg of sorted) {
const cost = estimate(seg.content);
if (spent + cost > BUDGET) {
console.warn(`dropped "${seg.name}" (${cost} tokens, over budget)`);
continue;
}
kept.push(seg);
spent += cost;
}
console.log(`context: ${spent}/${BUDGET} tokens`);
return kept.map((s) => s.content).join("\n\n");
}The console.warn is the important line. The first time you run this on a real agent loop, you find out something you assumed was essential has been silently dropped for weeks — or that one tool result is eating half your budget.
Rule 2: Retrieve, don't stuff
The naive RAG implementation grabs the top-K chunks by similarity and pastes them in. The problem: similarity scores are relative. If nothing in your knowledge base is actually relevant, you still get ten chunks back — they're just ten bad chunks that now look authoritative to the model.
// bad: always returns 10, relevant or not
const chunks = await vectorStore.search(query, { topK: 10 });
// better: threshold first, then cap
const results = await vectorStore.search(query, { topK: 20 });
const chunks = results
.filter((r) => r.score > 0.75) // irrelevant is worse than absent
.slice(0, 5); // hard ceiling regardless of score
if (chunks.length === 0) {
// tell the model the truth instead of feeding it noise
return "No relevant documents found in the knowledge base.";
}An empty result is a feature. A model told "nothing was found" will say it doesn't know. A model handed five irrelevant chunks will synthesize an answer out of them, and that answer will sound great.
Rule 3: Compact the history
Conversation history grows linearly and never stops. By turn 30 you're paying for turn 2, which nobody will ever reference again.
The fix is a rolling summary — keep recent turns verbatim, compress everything older into a few lines.
// compact-history.ts
import { GoogleGenAI } from "@google/genai";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
type Turn = { role: "user" | "assistant"; content: string };
const KEEP_VERBATIM = 6; // last 3 exchanges stay untouched
export async function compact(turns: Turn[]): Promise<Turn[]> {
if (turns.length <= KEEP_VERBATIM) return turns;
const older = turns.slice(0, -KEEP_VERBATIM);
const recent = turns.slice(-KEEP_VERBATIM);
const response = await ai.models.generateContent({
model: "gemini-3-flash-preview",
contents: `Compress this conversation into at most 8 bullet points.
Preserve: decisions made, values the user gave (names, IDs, numbers),
and any constraint the user stated. Drop pleasantries and reasoning.
${older.map((t) => `${t.role}: ${t.content}`).join("\n")}`,
});
return [
{ role: "user", content: `[Earlier conversation]\n${response.text}` },
...recent,
];
}Note what the summarization prompt protects: identifiers and constraints. Those are what break when a summary is too aggressive — the model forgets the user said "in EUR" fifteen turns ago and starts quoting dollars.
Rule 4: Tool results are the worst offender
This is the one that surprised me most when I built out tool orchestration. A single unfiltered API response can be larger than your entire prompt.
In my weather example, the model needed the temperature and conditions. The API returned dozens of fields — air quality indices, moon phase, UV forecasts, a full hourly breakdown.
// weather-tool.ts
export default async function get_current_weather(city: string) {
const res = await fetch(`https://api.weatherapi.com/v1/current.json?q=${city}&key=${KEY}`);
const data = await res.json();
// don't return data. return what the model needs.
return {
city: data.location.name,
temp_c: data.current.temp_c,
condition: data.current.condition.text,
humidity: data.current.humidity,
wind_kph: data.current.wind_kph,
};
}Five fields instead of forty. Same answer quality, a fraction of the tokens — and every irrelevant field you remove is one fewer thing the model can get distracted by.
The same applies to your tool definitions. Twelve tools with verbose descriptions and deeply nested JSON schemas is a meaningful chunk of budget spent before the conversation even starts. If a tool hasn't been called in your last hundred traces, it isn't earning its place in the window.
Rule 5: Position is not neutral
Models don't attend uniformly across the window. Content at the beginning and the end tends to get weighted more heavily than content buried in the middle — the "lost in the middle" effect that shows up across model families.
So the practical ordering:
- Top: system instructions, role, hard constraints.
- Middle: retrieved documents, compacted history, tool results — the bulk material.
- Bottom: the user's actual query, plus a restatement of the one or two rules that matter most.
That last part feels redundant and isn't. If the output format matters, say it in the system prompt and immediately before the query. The token cost is trivial next to the cost of parsing a malformed response.
The checklist
Before blaming the model, check:
- Do you know your token breakdown per segment, or are you guessing?
- Do retrieved chunks have a score threshold, or just a
topK? - Does history get compacted, or does it grow forever?
- Do tools return filtered results, or raw API responses?
- Are the critical instructions at the top and the bottom?
- Is every tool definition earning its tokens?
Conclusion
Prompt engineering was about writing better instructions. Context engineering is about everything else in the payload — and at this point, everything else is most of the payload.
The mental shift that made it click for me: context is state, and state needs management. You wouldn't let a variable grow unbounded through a loop, or fetch forty columns when you need five. The context window deserves the same discipline you'd give any other resource.
Most "the model isn't smart enough" bugs turn out to be "the model couldn't find the answer in the noise I sent it."