Context Compaction: Keeping the Hot Path Fast

⚙️ Most long-running agents do not fail because the model got dumber

Most long-running agent failures are not mysterious reasoning collapses. They are usually the result of a very ordinary systems problem: the model is forced to carry too much irrelevant context through too many turns. Every extra log chunk, tool transcript, pasted document, and stale planning note becomes part of the cost of making the next decision. The model has to read that material again, weigh it against the current objective, and decide what matters now. That sounds manageable until a session gets large enough that the truly important state is buried under historical clutter.

This is why context compaction matters. It is not summarization for aesthetics, and it is not a prompt trick for making demos look clean. It is a control mechanism for keeping the hot path small enough that the model can still act quickly and correctly. In practical agent systems, the hot path is the active working set used to decide the next action in the agent loop. If that working set becomes bloated, latency rises, token spend climbs, and tool choice often degrades because attention is being spent on old noise instead of the present decision.

That distinction matters a lot for AI agents for developers. Developers are building systems that call tools, inspect outputs, update state, and continue across many turns. In that environment, memory is not one feature. It is a layered systems problem. As LangChain’s memory guidance separates short-term state from persistent memory, compaction should target the thread-scoped working context first, not the durable source of truth stored elsewhere. This is the practical side of AI agent memory and the broader problem of short-term vs long-term memory in agents. For further background, LangChain’s memory docs are a useful anchor: Deep Agents memory.

Why this matters operationally is simple. When teams blame the model for getting worse over time, they often miss that the model is being asked to solve a retrieval problem before it can solve the task itself. A long-running agent may still have the right information somewhere in context, but if that information is mixed with outdated plans, repeated logs, and irrelevant transcript history, the next action becomes slower and less reliable. What looks like weak reasoning is often weak state management.

🧠 Hot path versus cold path is the design boundary that makes compaction work

The cleanest way to think about compaction is to separate hot-path state from cold storage. Hot-path state is whatever directly affects the next step. That usually means the current goal, active constraints, the latest observations, the current plan, unresolved errors, and stable pointers to artifacts that may need to be reopened. Cold storage is everything that might be useful later but does not need to be reread every turn: full logs, large tool outputs, retrieved source documents, transcripts from completed subtasks, and bulky artifacts such as generated files.

This boundary matters because the model pays for every token that remains in context. If you keep raw artifacts in the live prompt, the system often gets slower before it gets smarter. A coding agent is the obvious example. A test run may produce thousands of lines of output, but the next action usually needs only a few facts: the command failed, the failing file is auth.spec.ts , there were 12 failing tests, and the full log is stored at a known path. Everything else is recovery material, not immediate decision material.

There is also an architectural lesson here. State, memory, and storage should not be collapsed into one bucket. Thread state is for the active loop. Long-term memory is for reusable facts, preferences, or skills across sessions. Cold storage is for artifacts and audit records. LangChain’s broader memory concepts make this distinction explicit, and that clarity helps avoid one of the most common mistakes in LLM agent architecture: trying to solve token pressure by stuffing everything into “memory” without defining retrieval rules or scope boundaries. This also affects agent state management, agent orchestration, and even how durable execution for agents is designed. See LangChain memory concepts for the design rationale.

In practice, this boundary changes how you model state. Instead of asking, “Should we keep this?” ask, “Will the next decision be worse if this disappears from the prompt?” If the answer is no, it probably belongs in cold storage with a stable reference. That framing is useful because it forces the team to think in terms of action quality rather than attachment to raw history. Good compaction is not about deleting information. It is about putting each kind of information in the right layer.

🔧 Compaction is a policy engine, not an ad hoc summary step

Good compaction is procedural. It should be triggered by thresholds, shaped by content type, and backed by storage guarantees. LangChain’s context-engineering docs describe this clearly. When the session crosses roughly 85 percent of the model’s input window, Deep Agents first tries to offload oversized tool inputs and outputs. If that is not enough, it falls back to summarizing older context while preserving a recent slice and a canonical record of original messages in storage. If the context still overflows, it retries with a more aggressively compacted state. That is not a vibe-based prompt edit. It is an operational policy with deterministic fallback behavior.

A useful working policy is to start earlier than the hard limit. Around 70 to 80 percent of budget, suppress verbose traces and stop adding large artifacts into the live prompt. At 85 percent, summarize older turns and retain a recent working set. On overflow, retry using a stricter compacted state. The reason to do this before failure is simple: last-minute compaction is usually worse compaction. When you wait for overflow, the system is forced to compress aggressively under pressure, which increases the risk of losing constraints or lineage.

This policy framing matters because long context windows create false confidence. Teams assume the model can “handle it” because the published limit is large. In production, what matters is not just whether the request fits. It is whether the active window stays focused enough to support reliable tool use. That has direct consequences for agent reliability in long-running agents, especially when they rely on iterative tool use or nested workflows. The practical mechanics are documented in LangChain Deep Agents context engineering, and they are worth studying because they treat compaction as infrastructure, not decoration.

A helpful way to implement this is to think in stages. Stage one is prevention: stop obvious prompt bloat before it accumulates. Stage two is reduction: compact older or lower-value material while preserving recent state. Stage three is recovery: if the request still risks overflow, retry with a stricter version of context and keep the original artifacts available for targeted reopening. This staged design matters because it converts a chaotic prompt-management problem into a predictable control loop that can be monitored and improved.

📦 Reference substitution is usually better than lossy summarization

One of the sharpest mistakes teams make is summarizing raw artifacts that are already persisted somewhere safe. If the underlying object exists in a file system, object store, or memory backend, the better move is often reference substitution. Replace the bulky content with a compact record containing a preview and a stable pointer. That preserves recoverability. The agent can reopen the full artifact only when needed instead of relying on an imperfect summary that may omit exactly the detail required for debugging.

The pattern is simple. Instead of keeping a full test log in prompt context, keep something like this:

{
"test_run": "failed",
"log_preview": "12 tests failed in auth.spec.ts",
"full_log_path": "s3://runs/847/log.txt"
}

This works because it keeps the result semantically useful while avoiding prompt bloat. The preview tells the model what happened. The path preserves lineage. If the agent decides that deeper inspection is required, it can call a tool like read_file or retrieve the object explicitly. LangChain’s compaction guidance uses this exact philosophy for large tool outputs: offload content to storage and keep a file path plus a short preview in context. That is a much healthier default than flattening everything into prose.

Why this matters in the real world is straightforward. Summaries are compressions. Compressions can erase edge details, ordering, stack traces, IDs, and exact error strings. Those are often the details that matter most when an agent is recovering from failure. Reference substitution keeps the live context small without destroying the ability to inspect the original artifact later. It is one of the most practical patterns available to teams building tool calling agents.

There is also a subtle reliability benefit here. A reference can be validated, versioned, and reopened consistently. A summary cannot. If an artifact changes over time, a pointer with a version or checksum gives the agent a stable way to reason about which object it saw earlier. That becomes important in multi-step workflows where the same file may be regenerated or overwritten. In those cases, a clean reference is not just a space-saving trick. It is part of maintaining continuity across turns.

🗂️ Different content types need different compaction strategies

Compaction fails when it treats all content as if it were the same. Conversation history, shell logs, retrieved documents, and generated artifacts have different shapes and different failure modes. A structured summary works reasonably well for older conversation turns because the point is to preserve intent, decisions, unresolved questions, artifact locations, and next steps. That same approach is usually a poor fit for terminal output, where the beginning and end often matter more than the middle. OpenAI’s coding-agent guidance explicitly calls out shell output bounding that preserves the head and tail while omitting the noisy center. For terminal-heavy loops, that is often exactly right. See OpenAI’s coding-agent environment write-up.

Retrieved documents need a different treatment again. Once the agent has extracted the claims, citations, and unresolved questions it needs, the full text rarely belongs in the hot path. Keep the extracted facts and a retrieval pointer. If the model needs to verify a quote or inspect adjacent paragraphs, it can fetch the document again. This is especially important in research and support workflows, where auditability matters as much as answer quality.

Artifacts such as generated code, notebooks, reports, or spreadsheets should almost never be copied into prompt context in full after they are saved. The prompt should carry identifiers, paths, versions, and a short status note. This is a broader lesson in agent state management: preserve handles, not blobs. When you preserve handles, the working set stays compact and the system remains recoverable. When you preserve blobs, every turn becomes slower and less discriminating.

The reason content-type rules matter is that each artifact answers a different question. Older conversation answers, “Why are we doing this?” Shell output answers, “What just failed?” Retrieved material answers, “What evidence supports this?” Generated artifacts answer, “What did we produce?” If you compact them all into generic prose, you lose the structure that made them useful in the first place. Better policies respect the job each content type performs inside the workflow.

🚨 The real tradeoff is continuity, not just token count

It is tempting to frame compaction as a pure efficiency optimization, but that misses the harder part. Compaction changes failure modes. Under-compaction is obvious: token burn rises, latency worsens, overflow becomes more likely, and tool selection can degrade because the model is navigating a noisy prompt. Over-compaction is subtler and often more damaging. The agent forgets why a file mattered, drops a non-negotiable constraint, loses the chain of prior tool observations, or no longer remembers which subtask already failed.

That is why some information should be treated as non-removable from the active state. System and developer instructions, compliance rules, task acceptance criteria, unresolved errors, active assumptions, current artifact locations, and identifiers needed for tool continuity are not optional details. If those disappear during summarization, the model may continue coherently at the sentence level while becoming operationally incoherent at the workflow level. This is one reason agent systems can look articulate but act unreliable.

LangChain’s overflow fallback and OpenAI’s native compaction support both reflect this operational reality. Long sessions fail without explicit context management, but naive compaction can fail too. The discipline is to decide in advance what must survive every compression step. In practice, this means using structured summaries rather than generic prose. Capture intent, completed work, current blockers, artifact pointers, and next actions as fields. If a summary cannot preserve those fields, it is not a safe compaction artifact. The quality question is not “Did tokens go down?” but “Can the agent still continue correctly?”

This is why compaction should be evaluated as a continuity layer. The agent’s job is not to produce a plausible paragraph about past work. Its job is to preserve the minimum state needed to keep acting correctly. A shorter prompt that loses a blocking constraint is worse than a longer prompt that stays operationally faithful. That tradeoff is easy to miss because compacted output can look neat and coherent while silently dropping the details that drive action.

💻 A coding agent makes the compaction problem painfully obvious

Consider a long-running coding agent working inside a shell-enabled environment. It searches the repository, reads files, edits code, runs tests, watches logs, and iterates. In a few turns, test output and terminal chatter begin to dominate context. Most of that output is low-value on the next step. The agent usually needs a compact state like this: which command ran, whether it succeeded, which file path is implicated, what the top error category is, and where the full logs are stored.

OpenAI’s recent work on computer-equipped Responses API agents points in this direction. The platform supports compaction for long-running loops and also recommends bounding shell output so the result includes the most informative leading and trailing sections without flooding the prompt. That is a useful default because shell output is often bursty. The top includes command context and early errors. The bottom includes final failures and exit status. The middle is frequently repetitive.

Here is a practical state shape for the hot path after a failed test run:

{
"goal": "Fix auth regression in login flow",
"active_constraints": ["Do not change public API", "Keep OAuth tests passing"],
"latest_observation": {
"command": "npm test -- auth.spec.ts",
"status": "failed",
"error_summary": "12 failures in auth.spec.ts after token refresh",
"full_log_path": "s3://runs/847/log.txt"
},
"next_step": "Inspect token refresh handler and reopen failing log if stack traces are needed"
}

This keeps the working set focused. It also respects a basic truth about AI coding agents: they do not need to reread every log line on every turn. They need enough context to decide the next action and enough references to recover detail on demand. That is the center of compaction done well.

Step by step, the logic is straightforward. The goal field keeps the task anchored so the agent does not drift into unrelated refactors. The active_constraints field preserves rules that shape acceptable solutions. The latest_observation object turns a noisy test run into actionable facts: the command, the result, a compact error description, and a recovery path to the full log. The next_step field matters because it converts observation into intent, reducing the chance that the agent rehashes old state instead of moving forward. A small structure like this often outperforms hundreds of lines of raw transcript because it concentrates exactly what the next turn needs.

🧩 Implementation patterns that hold up in production

If you are implementing compaction in a real agent loop, it helps to think in schemas and flows rather than natural-language summaries alone. A compacted state should have explicit slots for goal, active constraints, current plan, latest observations, unresolved errors, artifact pointers, and next-step candidates. Older conversational content can be collapsed into a structured session summary with fields like intent , decisions , artifacts_created , pending_questions , and handoff_notes . This is more robust than free-form prose because it preserves machine-useful fields that generic summaries often drop.

Artifact pointers also deserve a stable format. Include a storage URI, content type, a short preview, and ideally a version or checksum if the artifact may change. That reduces ambiguity when the agent needs to reopen something later. A pointer like {"path":"s3://runs/847/log.txt","type":"text/plain","preview":"12 auth failures","version":"v3"} carries enough metadata to support recovery without bloating the prompt.

Failure recovery should be explicit too. If a compacted session produces a bad next step, the agent should be able to request rehydration of a specific artifact or older slice of context rather than blindly broadening the entire prompt. This matters because bad recovery behavior often turns one compaction mistake into runaway latency. The better pattern is narrow reopening: fetch the exact log, file, or subtask transcript needed, extract what matters, then return to a compact state.

OpenAI now supports native compaction for long-running agent loops and even exposes standalone compaction capabilities, which means you do not always need to hand-roll everything yourself. But server-side support does not remove the need for state design. You still decide what belongs in instructions, what belongs in the active turn, and what should remain in external storage. Native features help. They do not replace architecture.

From a developer’s perspective, the key is to make the policy observable. Every compaction event should emit traceable metadata: why it triggered, what was removed or summarized, what references were created, and whether the next step later required rehydration. Once that information is visible, teams can tune policy with evidence instead of guesswork. This is where agent observability and agent evaluation and tracing start to matter. Without observability, compaction tends to become one of those features that “usually works” until a subtle continuity bug appears in production.

🧪 Test compaction like a continuity feature, not a token feature

A surprising number of teams evaluate compaction only by checking whether token counts fell. That is necessary, but it is not enough. Compaction should be tested as a continuity feature. After summarization or offloading, can the agent still select the right tool, preserve the right constraint, and continue from the correct artifact? If not, the system got cheaper at the cost of becoming less correct, which is usually a bad trade.

A practical way to test this is with replay-based evaluations. Take real traces from long sessions, compact them at the points your policy would trigger, and then ask the agent to continue. Compare behavior before and after compaction. Did it remember the active acceptance criteria? Did it reopen the correct artifact when needed? Did tool choice degrade? OpenAI’s current emphasis on tracing and evals is relevant here because compaction is exactly the kind of system behavior that should be measured through workflow traces, not judged by intuition alone. See trace grading guidance.

Continuity regression tests are especially useful. Create cases where the only way to succeed is to preserve a specific prior fact through compaction: a non-negotiable constraint, a file path, a failed attempt that should not be repeated, or a user preference that affects tool choice. Then verify that your policy keeps those fields alive. This turns a vague concern about “losing context” into concrete tests that can fail in CI.

Instrumentation should also track whether the agent had to reopen offloaded artifacts after compaction and whether those reopen events improved or hurt task quality. If reopen rates are extremely high, your compacted state may be too thin. If reopen rates are near zero but task quality drops, your summaries may be misleading. Those signals tell you where the policy is failing.

🔐 Compaction can create governance problems if storage design is sloppy

Moving data out of the prompt and into stores is good for performance, but it changes the security model. Shared writable memory or artifact stores can become prompt-injection surfaces if the agent later reads from them as if they were trusted instructions. LangChain’s production guidance warns about this directly. If compacted summaries or offloaded artifacts are reused across users, assistants, or sessions, treat them as untrusted unless application logic validates them or the storage is read-only where appropriate. See LangChain production guidance.

This matters because compaction often creates a false sense of cleanliness. The prompt looks smaller, so the system feels safer. But the real control plane may now live in the retrieval path. If an attacker can influence a shared summary store or inject malicious content into a reusable artifact, the agent may later hydrate that content into the hot path. At that point, the storage layer is no longer a passive archive. It is part of the execution loop.

Scope boundaries help. Separate stores by tenant, task, and trust level. Keep developer instructions and policy constraints in protected channels, not in writable summaries. Preserve audit records of what was compacted, where it was stored, and when it was later reopened. This is another reason compaction should be treated as a systems discipline. Once the agent depends on cold storage for recovery, storage governance becomes part of agent reliability.

The long-term point is easy to underestimate. As soon as an agent relies on offloaded artifacts to continue work, storage quality becomes task quality. Access control, immutability, provenance, and auditability are no longer back-office concerns. They directly affect whether the system can recover safely and whether a human can later understand what happened. Fast prompts are useful, but reliable recovery is what makes the design sustainable.

📈 A practical deployment checklist for keeping the hot path fast

If you want a concrete starting point, begin with four decisions. First, define the hot-path schema. Decide exactly which fields remain in live context every turn: goal, active constraints, current plan, recent observations, unresolved blockers, and artifact pointers. Second, define content-type rules. Conversation turns get structured summaries. Shell output gets head-plus-tail truncation. Large tool outputs get externalized with previews and stable references. Full artifacts stay out of the prompt after persistence.

Third, define thresholds. A good initial policy is to start suppressing verbose traces around 70 to 80 percent of context budget, trigger stronger compaction around 85 percent, and on overflow retry with a more aggressively compacted state. Fourth, define tracing events and metrics. Record pre- and post-compaction token counts, latency, overflow rate, retry rate, reopen events for offloaded artifacts, and downstream task quality. Without those metrics, you are tuning blind.

There is also a useful implementation attitude here: be opinionated about what does not belong in context. Teams often try to preserve optional detail “just in case.” That instinct is exactly how hot paths become junk drawers. If an artifact is persisted and recoverable, it usually should not live in the prompt. If a prior turn no longer affects the next action and has been captured in a structured summary, it usually should not remain verbatim. Compaction is not about remembering less. It is about remembering the right things in the right layer.

The broader point is slightly sobering and helpful at the same time. A good agent is not the one that remembers everything. It is the one that keeps the working set small, preserves continuity, and knows how to recover the rest on demand. That is what makes long-running agents feel reliable instead of merely verbose.

🔄 Why this matters for modern agent frameworks

Context compaction is not tied to a single framework. The same principles apply whether you are building with the OpenAI Agents SDK, implementing memory flows with LangGraph memory, or coordinating state across multi-agent systems. The reason is simple: every serious agent runtime eventually has to decide what remains in the active loop, what gets persisted, and what gets reopened only when needed.

This becomes even more important with supervisor worker agents and explicit agent handoffs. In those designs, compact state is not just a performance optimization. It is the contract between one agent and the next. If the handoff contains raw clutter instead of structured state, orchestration becomes brittle. If it contains a compact goal, constraints, artifact pointers, and clear handoff notes, the receiving agent can continue with much higher reliability.

The same logic applies to reusable skills and procedural memory for agents. A long-running system should not keep every historical execution detail in the prompt. It should promote only the parts that deserve reuse, such as durable task strategies, stable preferences, or known recovery procedures. This is where memory design stops being a storage problem and becomes part of the behavior design of the system.