✨Article Summary (AI Generated)

DeepSeek open-sourced its agent runtime framework deepseek-harness (dsh) on 2026-08-13, and it crossed 90,000+ stars within two days. This article draws on a full read of the source, a side-by-side comparison with Pi, Codex CLI, Claude Code, and OpenCode, and cross-checks against dozens of real reviews from X, Hacker News, and Chinese-language communities to unpack its "everything is a plugin" Cordis core, its event-driven Turn/Step main loop, its append-only session log, and its self-modifying toolset — verifying, claim by claim, what the praise and the criticism actually get right. It closes with a clear recommendation on who should (and shouldn't) adopt it.

Prologue: 28 Hours, 90,000 Stars ​

On August 13, 2026, DeepSeek open-sourced a project called deepseek-harness (CLI name dsh). It crossed 50,000 stars in about 12 hours and roughly 92,000 by hour 28 — for reference, the previous velocity record holder, OpenClaw, took 84 days to reach 200,000.

The buzz is real, and so is the controversy. My timeline showed two opposing takes at once:

• One side: this is "the seed of an Agent OS," "the low-level skeleton no other framework has."
• The other: "needlessly heavy," "wrapping an entire plate of dumplings' worth of vinegar just to chase self-evolution."

Oddly, both camps included people who had genuinely read the source. So for this piece I did three things: read the whole repo cover to cover, compared it point by point against Pi, Codex CLI, Claude Code, and OpenCode, and pulled together the substantive commentary from X (Twitter), Hacker News, and Chinese-language communities, checking each claim against the actual code — to see how much of what people are saying actually holds up.

The one-line verdict up front: dsh isn't really "DeepSeek's version of Claude Code" — it's closer to a radical experiment in what an agent runtime should look like. It's not the right tool today for most people who just want to write code, but the bet it's making is worth a serious look from anyone building agent infrastructure.

Part One: What It Actually Is ​

1.1 The Basics ​

• Positioning: an open-source agent runtime framework (harness/runtime) — the CLI, the web UI, and the automation server are all just "assembled shapes" sitting on top of it.
• Release: 2026-08-13, currently at version 0.1.0-rc.5, explicitly labeled a developer preview. The README states plainly: "THERE WILL BE COMPATIBILITY-BREAKING CHANGES."
• License: MIT, fully open source.
• Stack: a TypeScript monorepo (57 package groups, roughly 500K lines of TypeScript), with about 300 lines of C11 underneath the Linux sandbox.
• Model support: not locked to DeepSeek's own models — a plugin adapter layer supports roughly 40 providers, including OpenAI, Anthropic, Google, Kimi, and any OpenAI-compatible endpoint; wiring one up takes a few lines of YAML.
• Scale (two days after launch, per the GitHub API): 92,700+ stars / 8,400+ forks.

An easy-to-miss piece of backstory: when DeepSeek previously published model benchmark numbers (for DeepSWE, for instance), it drew criticism for "vendor-reported, unreproducible" scores, and the team promised at the time to "open-source the harness used for evaluation." Multiple signals now point to dsh being exactly that promise kept — it's the internal framework DeepSeek uses to run agentic benchmarks against its own models. That explains why the repo shipped so mature on day one: 22 contributors, with the top contributors sitting at thousands of commits each — clearly this had been iterating internally for a long time.

1.2 Core Idea: Everything Is a Plugin ​

Most coding agents work like this: write a core agent loop (take input → call the model → run tools → repeat), then leave a few extension points around the edges (MCP, skills, and so on). The core is privileged code; extensions are second-class citizens.

dsh flips that structure. Its foundation is a general-purpose plugin framework called Cordis (its design traces back to the paper A Programming Paradigm for Spatiotemporal Composability, and it has already spent four years in production inside the Koishi chatbot framework). From there —

the model adapter is a plugin, the tool registry is a plugin, session storage is a plugin, the sandbox is a plugin, the UI is a plugin, and even the agent's main loop itself is a plugin.

This isn't marketing copy. In the source, the agent loop is literally just an ordinary package (packages/core/agent-loop) that can be swapped out via config like any other plugin. The minimal shape of a plugin is a file that exports an apply(ctx) function, mounted with a single line of YAML:

import type { Context } from '@deepseek-ai/cordis'
export const name = 'hello'
export function apply(ctx: Context) {
  console.log('hello from my first plugin')
}

The official release ships four preset run modes: standard mode (a full-featured coding agent), code mode (TypeScript SDK orchestration), minimal mode (just a shell and a file editor, for running benchmarks), and creative mode (runtime inspection and plugin experimentation). But these four are just four of the official "build patterns" — as beta tester Jiayuan Zhang put it in an apt comparison: dsh is like a Lego car kit, and the official presets are just one recommended way to assemble it, printed on the box.

Part Two: Core Design, Unpacked ​

2.1 An Event-Driven Main Loop: Turn and Step ​

dsh's main loop is a standard ReAct pattern (the model thinks → calls a tool → reads the result → thinks again), but instead of a hard-coded while loop, it's decomposed into a set of events:

turn/start → agent/pre-step → step/start
  → system-prompt/assemble (assembles the prompt and tool schema)
  → agent/request → llm/stream → assistant/message
  → tools/pre-execute → tools/execute → tools/post-execute
  → step/end → agent/turn-stopping → turn/end

A step is "one model request plus whatever tool calls it triggers"; a turn is zero or more steps, from receiving input through to everything being done. The key point: almost every stage is an interceptable event — a plugin can rewrite messages or refuse execution at agent/pre-step, or swap out a tool's result at tools/post-execute. The loop isn't the framework's private property; it's a public protocol every plugin gets to participate in.

The direct consequence: turning a single agent into a multi-agent collaboration architecture just means swapping out the loop plugin — no need to fork the whole project. Compare that to Codex CLI, where changing the main loop means editing its Rust core.

2.2 Append-Only Session Logs: Everything the Model Sees Is on the Record ​

dsh enforces a runtime invariant: "Model-visible means logged" — anything that goes into a model request must be reconstructible from the session log. The session log is an append-only event stream, and the conversation history the model sees is projected from that log, which supports resume, fork, search, and replay.

Some on Hacker News have called this a killer feature: at a moment when U.S. vendors are increasingly encrypting reasoning traces and making them hard to audit, dsh made "fully traceable" an architectural guarantee rather than a nice-to-have. GeekPark's hands-on testing confirmed the experience in practice: the system prompt, the chain of thought, and every tool call and result are preserved in full.

Context compaction isn't a black box baked into the loop either — it's an independent plugin: over-budget tool results get trimmed first, and if that's not enough, a summary node is generated to replace a chunk of history. The whole process is locked with three log events, so even a crash mid-compaction is reconstructible from the log.

2.3 Three Things "Nobody Else Has" ​

① Code Mode (run_code): the model writes a chunk of TypeScript that batch-calls tools via await tools.name(args), and only whatever gets printed or returned goes back to the model. This is a solution to the "a dozen scattered tool calls burn through the context window" problem — a dozen round-trips collapse into a single execution of one code block.

② A subagent can be delegated to a competitor: dsh's subagent backend (ctx.subagents) supports multiple providers, and startlingly, that list includes claude-code and codex — meaning you can dispatch a subtask from inside dsh whose actual executor is Claude Code or Codex CLI. This "harness-agnostic" stance has no precedent among competing products.

③ A self-modifying toolset (cordis_*): the agent can inspect its own plugin tree at runtime, write a new plugin on the fly, and mount it for use. This is the seed of a "self-evolving agent" — though two caveats are worth stating plainly: it's not enabled in any official preset by default and has to be turned on explicitly, and a plugin written on the fly lives only in memory — it's gone on restart, with no way to persist it yet.

2.4 Sandboxing and Permissions: Strict Where It Should Be ​

The security design here isn't sloppy: Linux uses bwrap + Landlock (paired with a custom-built C launcher that's fail-closed), macOS uses Seatbelt, and Windows uses restricted ACL tokens. The approval model is a closed enumeration, and anything anomalous is rejected as "unavailable" by default rather than silently allowed. It's even honest about distinguishing "full" versus "partial" sandbox enforcement — Landlock on older kernels only qualifies as partial, and it doesn't misreport itself as full.

But there's a layering gap worth flagging: the sandbox governs tool execution, while plugins themselves run inside the harness process. 36Kr's hands-on testing pointed out explicitly that any plugin can reach the shell and the filesystem — right now, the trust model for installing third-party plugins basically comes down to good faith.

2.5 An Easter Egg: Building a Harness With a Harness ​

What struck me most in this repo wasn't the code — it was the 1,386 Agent Notes under .agents/: architectural decision records classified as "implemented / rejected / archived / proposed," plus published post-mortems. The documentation runs to roughly 170,000 lines, nearly on par with the main codebase, and type snippets embedded in the docs are automatically diffed against the source in CI to prevent drift; 100% test coverage per file is a hard gate.

These traces strongly suggest that the repo itself was built with heavy AI-agent involvement — designing, reviewing, and writing post-mortems. dsh is the first user of its own philosophy.

Part Three: Side by Side — Four Different Roads ​

The coding-agent harness landscape right now roughly splits into four camps (star counts per the GitHub API as of 2026-08-15):

(For completeness: the fully open-source OpenCode sits at 197k stars, Gemini CLI at 107k, and Aider at 48k — their approaches roughly track Claude Code, Codex, and pair-programming tools respectively, and I won't dig into them further here.)

3.1 Versus Codex: Declarative vs. Imperative ​

This is the highest-quality technical debate I found in the community, from developer grapeot, who read both codebases line by line:

• Codex is declarative: a plugin is just a folder on disk (a Markdown skill, an MCP config, a shell script) that never enters the harness process. Reloading a config change takes 2-3 seconds, and the barrier to entry is close to zero.
• dsh is imperative: plugins carry state and run directly inside the harness process, registering and calling into each other. Hot-swapping a plugin at runtime means handling dangling references, terminating background tasks, coordinating dependency chains, and rolling back cleanly from a crash — which is why it needed a heavyweight runtime like Cordis; the core module that just manages plugin lifecycles alone runs to 750 lines.

An apt renovation analogy: Codex hands you a finished apartment plus a pegboard wall you can hang anything on — hang the wrong thing, take it down, hang something else. dsh hands you a house where you can rework a load-bearing wall without cutting the water or power. The question is: how often do you actually need to rework a load-bearing wall?

3.2 Versus Pi: Two Extremes ​

Pi sits at the opposite end of the spectrum: it's the minimalist agent libGDX author Mario Zechner built because he couldn't stand Claude Code's complexity creep — four default tools, and a philosophy that "what you leave out matters more than what you add." Interestingly, the two aren't opposites in every sense — dsh's multi-provider model adapter layer actually uses Pi's own @earendil-works/pi-ai library, so in a sense dsh is "a plugin skyscraper built on top of Pi's model layer."

And the token-efficiency comparison is brutal: one developer's preliminary test found that, on the same model, Pi's uncached input runs around 4.5K tokens versus roughly 47.6K for dsh — a difference of an order of magnitude (the tester flagged confounding factors of their own, and noted dsh is still a preview). The cost gap between minimalism and full pluggability shows up plainly on the bill.

3.3 Versus Claude Code: Not a Fight Over Features, But Over Business Model ​

On product maturity today, dsh and Claude Code aren't in the same league — even a developer who had a month of early access before launch said flatly that "as a coding agent to actually use, the experience genuinely isn't as polished as Claude Code or Codex." But 36Kr's analysis pointed to something more fundamental: by opening the entire capability set under MIT for free, dsh is effectively declaring that the harness layer shouldn't be a paid product at all, trying to pull the competition back to model capability and pricing themselves. That's not a shot at Claude Code's feature list — it's a shot at "the harness as a paywall" as a business model.

Part Four: What the Community Is Saying, and Whether It's Right ​

I filtered the commentary I collected down to claims backed by code or hands-on testing, then checked each one against the actual repo.

4.1 The Praise: Mostly Holds Up ​

• "Clever engineering in the plugin architecture" / "the seed of an Agent OS" — holds up. Loop-as-plugin, the three-role capability-surface design, transaction rollback — all verifiable in the code.
• "Full traceability is the killer feature" — holds up. "Model-visible means logged" is a runtime-enforced invariant, not documentation flourish.
• "The seed of self-evolution" — holds up, but is overstated. The cordis_* tools genuinely exist, but they're off by default and their output can't persist — still a real distance from "a self-evolving agent."
• "Generation quality is solid" — independent hands-on tests from ifanr and GeekPark both came out well (a Three.js mini-game, a site rebuild, at roughly $3 in cost). Credible, though the sample size is still small.

4.2 The Criticism: Also Mostly Holds Up ​

• "Over-engineered for everyday development" — I think this one holds up. The Chinese-language piece from yage.ai, titled roughly "wrapping an entire plate of dumplings' worth of vinegar just to chase self-evolution," makes a solid point-by-point rebuttal: search services only need brief interactions, restarting an MCP server takes 2-3 seconds anyway, and a skill is plain text that doesn't need framework-level hot reload. The "swap components without stopping the runtime" capability dsh solves for is real, but it's a rare requirement for the vast majority of use cases. One beta tester even observed that DeepSeek's own models frequently can't figure out how to use a plugin correctly and just edit their own code instead — "faster, and about as effective anyway."
• "Token usage runs high" — preliminary testing backs this up (roughly 10x versus Pi, roughly 3x versus other frameworks), and there's one confirmed, specific bug behind part of it: dsh reads both CLAUDE.md and AGENTS.md from a project, and if the two files have identical content (common, since many projects keep them in sync for cross-tool compatibility), the instruction set gets injected twice, doubling the system prompt outright. No official fix as of this writing.
• "Poor compatibility, early-stage ecosystem" — holds up. The official compatibility list shows 41 compatible integrations against 219 flagged as needing attention or further investigation; 36Kr's hands-on test found all 5 third-party tools it tried failed outright. The plugin repository crossed 2,000+ submissions in two days, but quantity isn't quality.
• "Documentation reads like word salad" — partially holds up. User-facing onboarding docs are genuinely thin, but the internal architecture documentation runs to roughly 170,000 lines and is checked against the code for drift — the problem isn't a lack of documentation, it's that "documentation written for agents to read" and "documentation written for newcomers to read" are two entirely different things, and the latter is currently missing.

4.3 The Benchmark Controversy: A Transparency Problem, Not a Proven Fabrication ​

This part needs the most careful phrasing:

1. Inconsistent scoring: V4-Pro's SWE-bench Verified score has two versions in circulation — 80.6% self-reported by the vendor versus 96.4% from third-party evaluator Vals — a 16-point gap most likely explained by different variants or methodologies, with no authoritative explanation available yet.
2. Scores from minimal mode: the official agent benchmark numbers were run in dsh's minimal mode, so the question of "does this score reflect the model's capability or the framework's boost" is a fair one — though, seen from another angle, an open-source harness is precisely what makes "reproduce it yourself" possible for the first time.
3. Non-public internal leaderboards: the official results table mixes in two non-public internal benchmarks (DSBench-FullStack and DSBench-Hard), and the two leaderboards rank things in contradictory order. Commentator MaxForAI's framing is worth keeping in mind: "what benchmarks a lab builds for itself tends to reveal exactly what it's optimizing for."
4. No one has publicly reproduced and disproven any official number yet. The skepticism is concentrated on transparency, not fabrication.

Worth noting honestly as well: the release coincided with a V4-Pro API price increase (corroborated by multiple independent sources), plus some users grumbling that "Pro writes worse code than Flash." That's a batch of negative sentiment unrelated to the harness itself but still coloring the discourse — and this kind of "triple disappointment" commentary generally comes without specific usage details, so it's of limited evidentiary value.

Part Five: Conclusion ​

What It Gets Right ​

1. Genuinely new architecture: loop-as-plugin, a fully swappable runtime, subagents delegable to competing tools — none of these exist in Claude Code, Codex, or Pi.
2. Traceability made into an architectural guarantee: for teams that need to audit, replay, or study agent behavior, this is currently unmatched.
3. A thoroughly open posture: MIT license, no model lock-in, compatibility with a rival's MCP naming conventions, and a direct challenge to the "pay for the harness" business model.
4. A rare display of engineering culture: 1,386 decision notes, docs gated against code drift, and the "built with its own harness" proof-of-concept — on its own, a valuable sample of AI-native engineering practice.

Where It Falls Short ​

1. It's a preview build right now: the team itself has promised breaking changes; interfaces will shift dramatically, and production use isn't recommended.
2. Niche needs, borne by everyone: the complexity and token overhead that come with the heavyweight runtime are paid by every user, while the benefit of "hot-swap the loop at runtime" only accrues to a small slice of explorers.
3. Rough edges in efficiency and quality: a context-duplication bug, a roughly order-of-magnitude token gap, and near-zero third-party compatibility.
4. Trust-building isn't finished: the transparency questions around benchmark methodology and internal leaderboards will need time and third-party reproduction to settle.

Should You Use It? ​

• You just want a good coding agent to write code with: not recommended, at least not yet. Claude Code (for the ecosystem), Codex CLI (for safety and stability), and Pi (for minimalism and cost) are all more mature choices today.
• You work on agent infrastructure, multi-agent systems, or self-evolving agent research: it's worth a weekend to read the source and the .agents/ directory — the capability-surface design and event-driven loop currently have no other working reference implementation.
• Your organization needs auditable agent execution records: worth watching closely — the append-only log is its least controversial strength.
• You're on the fence: give it three to six months. The plugin ecosystem will shake out, the interfaces will stabilize, and someone will reproduce the benchmarks. By then we'll know whether "everything is a plugin" grew into an Agent OS, or fell into the "plugin fatigue" that one HN commenter worried about.

One closing thought. Reading through this repo, what came to mind wasn't a competing product — it was the history of operating systems: dsh is betting that "agents will grow into systems complex enough to need an OS," while Pi is betting that "an agent should stay a sharp, small knife." Both bets could very well pay off — they're just aimed at different users, on different timelines. And a model company choosing to lay its own evaluation harness fully open under MIT, decision records included, has already shifted the industry's baseline for what "transparent" means — regardless of how this particular bet turns out.

This article is based on a snapshot of the repository (v0.1.0-rc.5) as of 2026-08-15, along with public community discussion. dsh is iterating quickly, and specific details here may age fast; all benchmark figures are labeled by source, and where vendor-reported and third-party numbers diverge, no conclusion has been reached — judge accordingly. Primary sources: deepseek-ai/deepseek-harness, the DeepSeek Harness website, Hacker News discussion, long-form source-code analyses from several developers on X, and independent hands-on testing from ifanr, GeekPark, and 36Kr.

✨Article Summary (AI Generated)

Following the evolution from Prompt → Context → Harness → Loop, this article unpacks the origins of Loop Engineering, its six-stage control flow, verifiers, stopping conditions, and three essential guardrails. Through Ralph Loop, Claude Code's /goal and /loop, Codex Automations, and Karpathy's autoresearch, it shows that the new idea is not the while statement itself, but an engineering system that combines verification, budgets, external state, and feedback. It concludes with a task-selection matrix and adoption path, arguing that a loop's value is capped by a verifier the agent cannot game.

Introduction: You Are the Slowest Part of the Chain ​

Consider a scene that plays out every day in 2026.

An engineer opens a terminal and tells a coding agent, “Fix the failing CI test.” The agent works for three minutes and reports back. The engineer reads the result: “No, you changed the wrong file; look at the auth module.” Three more minutes pass. “The test passes, but you broke another case.” After five rounds, the issue is fixed in forty minutes. The agent worked for fifteen of them. During the other twenty-five, it was waiting for a human to read output, make a decision, and type the next prompt.

Models reason faster and tools execute faster. The person at the keyboard has not. The bottleneck has moved from the model to the human.

At an Acquired Unplugged event hosted by WorkOS on June 2, 2026, Claude Code creator Boris Cherny described his response:

“I don't prompt Claude anymore. I have loops that are running. They're the ones that are prompting Claude and figuring out what to do. My job is to write loops.”

Five days later, OpenClaw creator Peter Steinberger, now at OpenAI, posted the viral line: “You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents.” The following day, Google engineering leader Addy Osmani published Loop Engineering and gave the practice a name.

Rather than follow a research-report sequence of definition, timeline, and critique, this article traces the evolution of the developer-agent collaboration chain: how each layer emerged, what actually lives inside a loop, which parts are old ideas in new packaging, what is genuinely new, and whether your own work should use one.

1. Four Delegations: Prompt → Context → Harness → Loop ​

Loop engineering did not appear from nowhere. It is the fourth delegation of work along the same collaboration chain. At every layer, a human hands one more piece of manual work to the system and moves up a level.

Each layer contains the previous one rather than replacing it. Loop authors still write prompts; those prompts have simply moved from words typed on the spot to templates assembled automatically during every iteration.

A popular Chinese-language analogy puts the four layers plainly:

Prompt is how you ask it. Context is what you let it see. Harness is the environment you place it in. Loop is how you make that system turn by itself.

The timing closely matches Anthropic's engineering publications, although “loop engineering” itself is still not an official Anthropic term. Osmani's definition is the anchor for everything that follows:

“A loop here can be thought of a recursive goal where you define a purpose and the AI iterates until complete.”

2. Anatomy of a Loop: Six Stages and One Exit ​

Anthropic's Building agents with the Claude Agent SDK (Thariq Shihipar, September 29, 2025) gives the minimal version: “gather context → take action → verify work → repeat.” The complete Loop Engineering control flow has six stages:

The human's position has changed. Instead of standing inside ASSEMBLE and typing every turn, the engineer stands outside the graph, designs how the six stages connect and what qualifies as STOP, and then reviews the delivered result. Geoffrey Huntley puts it this way:

“Your job is to sit on the loop, not in it.”

Routine, Workflow, and Loop ​

Not everything that runs automatically is a loop. There is one dividing line: does it inspect its own work and use that result to decide whether to continue?

This echoes Anthropic's classic distinction in Building Effective AI Agents (Erik Schluntz and Barry Zhang, December 19, 2024): workflows orchestrate LLMs and tools through predefined code paths, whereas agents dynamically direct their own process based on environmental feedback—in other words, they use tools inside a loop.

3. Prehistory: Loops Were Not Invented in 2026 ​

The control-flow diagram should look familiar. It is a cybernetic feedback loop, a thermostat, or a Kubernetes reconciliation loop. Academia and the community spent four years building it before “loop engineering” received a name.

ReAct, submitted on October 6, 2022 and later published at ICLR 2023, established the inner Thought → Action → Observation loop. Reflexion (Shinn et al., 2023) added self-critique and memory. AutoGPT showed the public both the appeal of “give it a goal and let it run” and the failure mode of disappearing down a rabbit hole.

Ralph Loop: Proving the Pattern Before It Had a Name ​

Geoffrey Huntley's Ralph loop, or Ralph Wiggum technique, first demonstrated publicly in June 2025 and published in July in Ralph Wiggum as a “software engineer”, proved the pattern before it had a label. Its entire implementation is one line of Bash:

while :; do cat PROMPT.md | claude-code ; done

That is all: an infinite loop feeding the same prompt file to a coding agent. Huntley's own definition was, “Ralph is a technique. In its purest form, Ralph is a Bash loop.”

The apparently foolish design contains one very smart decision: every iteration starts with a fresh context.

Progress is not something the model remembers; it is something the repository contains. Each agent wakes up, reads the spec, inspects the repository, analyzes the gap, completes one item, commits, and dies. Its successor sees everything through Git. Huntley's self-deprecating summary is perfect:

“The technique is deterministically bad in an undeterministic world.”

Ralph has impressive but qualified claims. A YC hackathon team reportedly ported six repositories overnight for about $600 in API usage and more than 1,000 commits. Huntley says three months of continuous operation produced CURSED, a programming language with a self-hosting compiler. Both reports come from advocates and lack independent reproduction. Huntley also drew a hard boundary: “There's no way in heck would I use Ralph in an existing code base.” It was for greenfield work.

Anthropic released an official Ralph Wiggum plugin in December 2025. The productization of a playful community hack marked the loop's transition into a platform primitive.

4. The Naming Event: Ten Days in June 2026 ​

Why did a four-year-old pattern suddenly acquire a name and dominate discussion in June 2026? A short chain of public statements created the immediate spark.

Three deeper conditions were all necessary:

1. Models became strong enough. Opus 4.5+ and GPT-5.x-Codex-class models can operate autonomously for long periods and verify themselves through tests and compilers.
2. Primitives became products. Claude Code's /goal and /loop, and Codex Automations, replaced much of the custom Bash required a year earlier.
3. Leading practitioners spoke at once. Cherny, Steinberger, Karpathy, and Ng described loop-centered workflows during the same two weeks, giving a diffuse practice a name and credible faces.

One widely repeated quote needs a warning label: Jensen Huang's alleged “Nobody writes prompts anymore. The new job is to write and handle loops.” No clean NVIDIA transcript verifies it, and reposts disagree even about whether the source video was 23 or 53 minutes long. Treat it as a paraphrase of a real direction, not an authenticated quotation.

5. Method: Five Building Blocks, One Memory Layer, Four Nested Loops ​

Osmani's Building Blocks ​

Osmani describes five components plus an external-state layer. The structure maps almost directly onto both Claude Code and OpenAI Codex, evidence that loop shape is becoming tool-independent.

His explanation for using sub-agents is memorable: “The model that wrote the code is too nice grading its own homework.”

LangChain's Four-Layer Loop Stack ​

Sydney Runkle at LangChain described four nested loops on June 16, 2026, each operating at a different order of time scale.

Loops 1 and 2 complete the immediate work. Compounding happens in Loops 3 and 4: systems learn from production traces and improve their own configuration. Andrew Ng's three loops express the same idea at different granularity: coding in minutes, developer feedback in hours, and external feedback in days or weeks.

6. The Load-Bearing Wall: The Verifier ​

If you remember one sentence from Loop Engineering, make it this: a loop's value is capped by its verifier, not its model.

Without a reliable test for completion, a loop either never stops or stops in the wrong place and confidently reports success. Serious 2026 writing converges completely on this point.

Separate the Maker From the Checker ​

The community's rule is uncompromising: “The checker is never the same agent as the maker.” It follows the same separation of powers as the initializer and coding agents in Anthropic's harness work.

Good and Bad Stopping Conditions ​

Four Failure Modes ​

How /goal Builds the Verifier Into STOP ​

Claude Code's /goal, released in v2.1.139 on May 11, 2026 and checked against official documentation, is a useful specimen: maker and checker are built directly into the stopping condition.

The worker and evaluator are different models with different perspectives—the product form of maker/checker separation.

7. Three Guardrails to Install on Day One ​

The second consensus among serious writers is that guardrails are mandatory. A loop without guardrails is a liability, not an asset.

The budget guardrail has a repeatedly cited real-world footnote. According to secondary reporting, Uber exhausted its annual AI budget in four months and then capped agent tooling at $1,500 per engineer per month. A budget ceiling is not paranoia; someone has already paid the tuition.

8. The Primitives Have Been Productized ​

The table explains another part of “why now.” What required hand-maintained Bash in 2025 became a product feature in 2026.

Boris Cherny's example of how a typical day begins is Loop Engineering's hello world:

/loop babysit all my PRs. Auto-fix build issues, and when comments
come in, use a worktree agent to fix them.

His own record is the strongest first-hand evidence in the field. In the 30 days ending in late December 2025—not June 2026, as later retellings often claim—he landed 259 PRs, 497 commits, +40K/−38K lines, all written by Claude Code with Opus 4.5. He had uninstalled his IDE in November 2025.

The same shape appears outside coding. Andrej Karpathy's autoresearch, released March 7, 2026, gained roughly 25,000 GitHub stars in five days and more than 66,000 by early April. It loops through propose change → train → evaluate, retaining only changes that lower validation loss and using Git revert for failed experiments. The first demonstration reportedly ran about 700 experiments in two days. Fortune called it “The Karpathy Loop.” Loop, mechanical verifier, and externalized Git state are all present.

9. A Cooler Look: Three Critiques and What Each Gets Right ​

Popular concepts attract a backlash. All three major critiques deserve attention because each contains something true.

Osmani himself warns about comprehension debt: the faster a loop produces code, the wider the gap between the code that exists and the code you understand. Two people can build the same loop and obtain opposite results. The Register argued on June 24, 2026 that his conclusion undercut the concept because loops change the work without removing the engineer.

Osmani's closing line sets the right tone:

“Build the loop. But build it like someone who intends to stay the engineer, not just the person who presses go.”

10. Decision: Which Tasks Belong in a Loop? ​

The decision matrix compresses the argument into two axes: can “done” be mechanically verified, and what is the cost of failure?

The five task types in the lower-right share three traits: they are repeatable, machine-verifiable, and inexpensive to get wrong. Start there. A concrete graduation criterion from the research is: run one loop unattended for a full week, exceed the budget zero times, and have more than 90% of its PRs mergeable after your review. Only then expand its authority.

The adoption order matters: build the verifier before the loop.

Epilogue: The Scarce Skill Has Moved ​

Return to the engineer in the introduction. Of forty minutes spent fixing tests, the human occupied twenty-five. Loop Engineering does not answer with “type faster.” It removes the person from the chain and replaces each intervention with a verifier plus three guardrails.

The scarce skill is no longer phrasing—the prompt layer addressed that. It is no longer supplying material—the context layer addressed that. It is not even building the environment, which harness products are increasingly solving. It is writing a stopping condition the loop cannot fool.

Someone who can write that condition can let 259 PRs grow in 30 days. Someone who cannot gets a loop that deletes tests to turn CI green and a four-figure bill.

They may have built exactly the same loop.

✨Article Summary (AI Generated)

Starting from the fundamental constraint that large language models are stateless, this article breaks agent memory into a four-stage lifecycle: extraction, storage, retrieval, and update/forgetting. It compares vector stores, temporal knowledge graphs, and file-based memory through systems such as Mem0, Letta, Graphiti, Codex, and Claude Code. The central conclusion is that long-running memory depends less on a fashionable retrieval backend than on auditable maintenance, merge-before-add discipline, and forgetting driven by real usage.

TL;DR ​

• Memory is an engineering workaround for a finite context window. An LLM is stateless; an external system must put the right part of the past back into the current context. Every memory system can be understood through four stages: extraction → storage → retrieval → update/forgetting.
• Three trade-offs appear repeatedly: passive extraction versus active self-editing, vectors versus graphs versus files, and database-backed memory middleware versus human-readable memory for personal and coding agents.
• Mainstream products are converging on the same shape: a compressed long-term profile plus raw recent working memory.
• Update and forgetting separate a demo from a system that can operate for years. The emerging principles are “merge before adding” and “keep raw records separate from compressed views.”
• Do not put too much faith in benchmarks. Letta's plain filesystem setup with GPT-4o-mini scored 74.0% on LoCoMo, above the 68.5% reported for Mem0's best graph variant. Evaluate with your own data.

Let us begin with a fact that is easy to overlook but shapes this entire field: a large language model is stateless.

An LLM call is essentially a tokens-in, tokens-out function. Anything you teach it in one conversation—your name, your preference for pnpm over npm, or the pitfall you hit last week—disappears from the next call unless an external system puts that information back into the context window. “An agent has memory” therefore does not mean that the model truly remembers. It means that a surrounding system feeds the right part of the past back at the right time.

A larger context window does not remove the problem. Mem0 reported a full-context baseline with a p95 latency of 17.12 seconds and roughly 26K tokens per request. A memory-based approach used about 1,764 tokens of selected facts with roughly 0.2 seconds of search latency, saving around 90% of the tokens and 91% of the latency.

A memory system is fundamentally an engineering workaround for a finite context window. It exchanges roughly 1.8K tokens of selected facts for 26K tokens of raw history and balances capacity, cost, and latency.

All later design questions—what to save, where to save it, how to retrieve it, and how to forget it—reduce to one parent question: with a limited context budget, how can an agent behave as if it remembered everything?

A Conceptual Map of Memory ​

Short-Term vs. Long-Term Memory ​

Short-term memory, also called working memory, is the context inside the current conversation or thread: recent messages, tool results, intermediate reasoning, and unfinished plans. Its lifetime is normally the lifetime of that session.

Long-term memory persists across sessions and threads. It lives in a database, vector store, graph, or file and supports personalization and continued learning.

LangGraph makes this boundary explicit through two separate persistence mechanisms:

Confusing short-term and long-term memory is one of the most common architecture mistakes in agent systems. Put a user preference in a thread checkpointer and it vanishes when the user starts a new session. Put transient scratch work in a long-term store and that store quickly decays into noisy, misleading context.

Short-term memory preserves continuity within a conversation. Long-term memory preserves continuity across conversations. They need different storage media, write triggers, and cleanup policies.

CoALA's Four Categories—and the Disagreement Around Them ​

The 2023 Princeton paper Cognitive Architectures for Language Agents (CoALA, arXiv:2309.02427) popularized a four-way classification:

This vocabulary draws on cognitive psychology, including Tulving's distinction between episodic and semantic memory, Squire's work on procedural memory, and Baddeley and Hitch's working-memory model. IBM, MongoDB, LangChain, Letta, and Mem0 all use variants of it.

By late 2025, however, the field had started to split. Letta's Sarah Wooders rejected the brain analogy: an LLM is a tokens-in, tokens-out function, not a brain, so memory should be designed around information entering and leaving the context window. A December 2025 survey, Memory in the Age of AI Agents (arXiv:2512.13564), proposed a simpler taxonomy: factual, experiential, and working memory.

There is no need to choose a camp. CoALA gives teams a shared vocabulary; the engineering view reminds us that the decisive question is not which label a memory receives, but whether the right information enters the context at the right moment.

The Memory Lifecycle: A Unifying Framework ​

The most useful way to compare systems is through a memory's complete lifecycle:

Extraction → Storage → Retrieval → Update/Forgetting

• Extraction decides what is worth remembering.
• Storage decides where it lives and in what representation: vector, graph, KV, relational database, or text file.
• Retrieval decides when and how it returns to the model: semantic search, keywords, graph traversal, or direct prompt injection.
• Update/forgetting resolves conflicts, merges duplicates, and expires or decays stale information.

Frameworks differ because they make different choices in these four stages. Mem0 extracts facts passively; Letta lets the agent edit memory through tools. Graphiti invests heavily in update and forgetting through a bi-temporal model. Once a product is placed into these four boxes, its real trade-offs become easier to see.

Stage One: Extraction—What to Remember, and Who Writes It ​

Extraction answers two questions: who writes memory, and what gets extracted. Information lost here cannot be recovered by a stronger retriever later.

Four Ways to Write Memory ​

Direct append is cheap and loses no information, but raw logs eventually become too large to feed back into the model. LLM extraction is the database-oriented mainstream: a separate call distills a conversation into structured facts, triples, insights, or experience entries.

Agent self-editing gives the main model tools such as core_memory_append, memory_replace, memory_insert, and memory_rethink. It makes nuanced decisions with full context, but every memory operation spends inference tokens. Manual files offer the highest control and quality but do not scale.

Real systems combine modes. Codex layers human-authored AGENTS.md, LLM-distilled ~/.codex/memories/, and append-only session logs. Claude Code combines CLAUDE.md with model-written auto memory.

The matrix has two revealing empty corners. File systems rarely use unfiltered append because files are meant for humans to read. Databases rarely rely entirely on human maintenance because people cannot fill machine-scale retrieval stores. Both sides therefore converge on LLM extraction.

What Different Systems Extract ​

Mem0 extracts general personal facts and preferences. Its FACT_RETRIEVAL_PROMPT turns dialogue into short, self-contained statements such as Name is John. Newer defaults lean toward ADD-only extraction, retaining separate details instead of aggressively merging them.

Codex extracts four high-signal categories: user preferences, reusable knowledge, failure shields in the form symptom → cause → fix, and repository maps. It deliberately underindexes assistant claims and overindexes user statements and code evidence. Missing a fact is preferable to storing the agent's speculation as truth.

AutoGen learns insights from failure. Its task-centric memory repeats a task and, after an incorrect answer, asks an LLM to distill an insight from that failure. This resembles procedural memory more than a passive record.

Passive Extraction vs. Active Self-Editing ​

Neither approach is universally better. Passive extraction offers consistency and low cost for unattended consumer products. Active editing offers the nuance needed by autonomous agents that should understand a user better over time.

Stage Two: Storage—Where Memory Lives ​

Storage is not merely a database choice. The representation determines how large memory can grow, who can inspect it, how errors are corrected, and how difficult migration becomes.

Three Storage Families ​

Vector stores excel at fuzzy semantic matching. Temporal graphs represent entities and their changing relationships; Graphiti uses episodic, semantic-entity, and community subgraphs, combining semantic search, BM25, and graph algorithms. Files are deceptively simple: the model edits Markdown and the runtime loads or searches it.

These families are not mutually exclusive. Mem0 can combine a vector database, an optional graph database, and a conventional metadata store. cognee feeds relational, vector, and graph layers through one pipeline.

The More Fundamental Divide: Databases vs. Files ​

Database memory makes humans pay the cost: vectors and graph nodes are difficult to inspect, review, and migrate. File memory makes capacity the constraint. It scales through index-then-fetch: keep a compact index resident and load detailed files only when needed.

This also explains a clear trend. Personal and coding agents increasingly choose files—Codex, Claude Code, OpenCode, Pi, Hermes, OpenClaw, Cursor rules, and Copilot instructions—because transparency, editability, Git history, and repo portability matter. Memory middleware such as Mem0, Zep, Graphiti, and cognee chooses databases because multi-user capacity and retrieval quality matter more than direct human inspection.

The reason is the audience. Developers want memory they can inspect, manage, version, review in a pull request, trace with git blame, and roll back. A CLAUDE.md or AGENTS.md fits the workflow they already use; a vector index does not. Memory middleware has the opposite priorities: large capacity, semantic retrieval, and tenant isolation are the product, while direct human readability is secondary.

Backend Choices at a Glance ​

Even inside the database camp, two styles emerge. Mem0 assigns responsibilities to specialized stores, gaining flexibility while operating several systems. cognee consolidates three layers into LanceDB to reduce dependencies. LangGraph pragmatically reuses PostgreSQL and pgvector, leaving memory design to the developer but introducing no unfamiliar infrastructure. The choice is between a higher capability ceiling from specialized backends and simpler operations from a general-purpose one.

Stage Three: Retrieval—Putting Memory Back Into Context ​

Storage and retrieval are two sides of the same coin. Pick a vector store and the retrieval path will probably be semantic search; pick Markdown and it will probably be index-then-fetch.

Database Memory: From One Semantic Path to Hybrid Retrieval ​

Database-backed systems begin by embedding the query and retrieving the top-K nearest memories. Pure semantic search is weak at exact entity names, rare proper nouns, numbers, and dates—an embedding may consider pnpm and npm nearly identical—and it does not know whether a fact is still current. The common direction is therefore hybrid retrieval.

Graphiti is the most complete: vectors capture semantic similarity, BM25 finds exact terms, and graph BFS follows entity relationships. RRF combines the three lists before a cross-encoder reranks them. Mem0 follows a similar design and adds temporal selection for questions such as whether a role belongs to last year or today. crewAI turns retrieval into a feedback loop: below a confidence threshold, it feeds the current results to an LLM, reformulates the query, and searches again.

Hybrid retrieval improves recall, but every additional path, fusion stage, and reranker adds latency and engineering complexity. Mem0's roughly 0.2-second memory-search p95 is dramatically cheaper than its 17.12-second full-context comparison, yet the trade-offs among hybrid paths still require engineering.

File Memory: Layered Index-Then-Fetch Retrieval ​

File-backed systems load a compact index every session, search it, open the relevant detail file, and return to raw evidence only when verification is needed.

In Codex, the sequence is explicit:

1. Inject memory_summary.md into the system prompt as a resident index.
2. Search MEMORY.md with keywords to locate a relevant entry.
3. Follow its pointer and open the detailed topic file.
4. Return to the original rollout JSONL when the distilled claim needs evidence.
5. Keep the lookup within a four-to-six-step budget so the agent does not browse memory without limit.

Claude Code's auto memory uses the same pattern: the first 200 lines or 25 KB of MEMORY.md act as the per-session index, and topic files are loaded on demand.

This is how file memory competes on capacity. The library may grow to hundreds of files while the resident context cost stays fixed; detailed content enters the window only after a relevant grep hit.

A Useful Exception: Hermes Uses Full-Text Search ​

Hermes stores historical messages in SQLite and searches them with FTS5, including a trigram CJK index, rather than semantic embeddings. This deliberately gives up semantic similarity in exchange for zero LLM or embedding cost, the exact original message instead of a second-hand summary, and a retrieval path whose code explicitly says no summary LLM path. When the goal is to recover precisely what someone said, plain full-text search can be faster, cheaper, and more accurate than vectors.

Retrieval May Matter Less Than We Think ​

Letta's filesystem experiment is a counterintuitive reminder: a GPT-4o-mini file agent with almost no prompt tuning scored 74.0% on LoCoMo, above specialized memory systems including Mem0's reported 68.5% graph result. The key may be how an agent manages context, not the retrieval primitive itself. What matters is when it retrieves, how much it retrieves, and how it uses what it found.

Stage Four: Update and Forgetting—Keeping Memory From Rotting ​

If extraction, storage, and retrieval determine whether memory works at all, update and forgetting determine whether it keeps working. A store that only grows will eventually contain duplicate facts, conflicting preferences, and obsolete information. Many projects implement the first three stages and effectively leave this one blank, which is why update and forgetting are the true boundary between a toy and a durable system.

Deduplication and Merging ​

Most systems do little meaningful deduplication. AutoGen assigns monotonically increasing IDs and appends; MetaGPT only detects exact object duplicates. They assume retrieval ranking can work around dirty data after it has already entered the store.

The systems that treat consolidation as a first-class concern follow three main strategies.

1. Similarity gating plus an LLM decision at write time. crewAI first performs vector search for similar memories. If the top similarity is at least its default consolidation_threshold of 0.85, an LLM produces a ConsolidationPlan with keep, update, and delete actions and decides whether insert_new is still necessary. Mem0's classic DEFAULT_UPDATE_MEMORY_PROMPT similarly retrieves the top ten memories and chooses ADD, UPDATE, DELETE, or NONE. Mem0 maps UUIDs to consecutive integers before showing them to the LLM to reduce identifier hallucinations.

The defining idea is simple: merge or reject bad data before it enters the store, not after retrieval has to route around it.

2. Deduplication through graph identity. cognee derives node IDs with uuid5(NAMESPACE, normalized_name), so repeated mentions of the same normalized entity collapse into one node; an LLM then merges its accumulated descriptions. Graphiti first narrows candidates with MinHash, LSH, and Jaccard similarity, then lets an LLM decide. Its prompt explicitly forbids merging items whose numbers, dates, or critical qualifiers differ—“the 2024 budget” and “the 2025 budget” are similar text but different facts.

3. Offline LLM reorganization. Letta's sleep-time agent reads transcripts in the background and rewrites memory blocks with memory_rethink so they remain comprehensive, readable, and current. Codex hard-codes no-op preferred and aggressively merge, using the Git workspace diff to route memory that needs consolidation. Hermes performs “umbrella merging” under the maxim that one skill per session is a failure: session-specific fragments should be folded into reusable skills instead of becoming a log.

These approaches all refuse to rely on the main conversation agent to maintain memory while doing another task. They create a two-speed system: the conversation records evidence quickly; a dedicated process merges, promotes, rewrites, or archives it.

There is an important reversal in newer Mem0 snapshots. Its default V3 path has moved from aggressive LLM UPDATE decisions toward ADD-only extraction, linked_memory_ids, and MD5 exact deduplication. “Owns a dog named Max” and “went camping with Max” remain two linked memories rather than being crushed into one. Even Mem0 is recalibrating the trade-off: merge too aggressively and independently useful facts become impossible to recover. “Merge before adding” is a dial, not a command to erase detail.

Forgetting Without Destroying History ​

Most memory systems do not truly forget. Mem0, cognee, and AutoGen generally grow monotonically; Letta's archival storage “persists indefinitely” unless an API call explicitly deletes something. Capacity management is pushed to retrieval and context truncation instead of the store itself. The projects that do forget use four distinct approaches.

1. Soft invalidation and versioning. Graphiti gives every fact edge four bi-temporal timestamps:

When a new fact contradicts an old one, the old edge is not deleted. It gets expired_at=now and invalid_at equal to the new fact's valid_at. This makes both current-state and point-in-time queries possible. If a user moved from npm to pnpm, the system can use pnpm now without erasing the history that npm was previously correct. AutoGPT's newer memory layer reuses Graphiti's model.

2. Usage-count-driven retention. Codex stores usage_count and last_usage in stage1_outputs; consolidation can ignore entries whose last use falls outside max_unused_days. OpenClaw likewise promotes and retains memory according to real retrieval traffic. This is more defensible than an “importance” score assigned when the memory was written: repeated use is evidence, while predicted importance is a guess.

3. Time decay and state machines. OpenClaw applies exponential decay with different half-lives, such as 30 and 14 days, for different layers. crewAI includes decay = 0.5^(age_days/30) in retrieval scoring. Hermes marks a skill STALE after 30 unused days and moves it into a recoverable .archive/ after 90 days; it never physically deletes the skill.

4. Tombstone-based logical forgetting. OpenHands models conversation history as an append-only event log. When context must be compressed, its condenser writes a Condensation tombstone saying that a group of events has been replaced by a summary. A replayed View filters those events and shows the summary, but the original evidence remains physically present. One dataset supports both a compact model-facing view and a complete audit view.

The common pattern is striking: mature systems prefer logical forgetting over physical deletion. Physical deletion permanently destroys evolution and audit history, which are often the only way to explain why an agent acted as it did. The scarce capability is not clean deletion; it is reversible forgetting.

Do not expect the main conversation agent to maintain memory casually between tasks. Reliable systems either gate writes or assign consolidation to a constrained, dedicated process. Memory maintenance must be a first-class workload.

Four Memory-Writing Philosophies ​

The table contains four core tensions.

Cost versus control. Passive extraction gives a separate, relatively cheap LLM call responsibility for deciding what matters. It is predictable and does not spend the main agent's tokens, but any signal outside the extraction prompt is lost. Letta's active editing makes the best-informed model decide, but every core_memory_append or memory_replace spends inference tokens and memory quality becomes model-dependent.

When to resolve conflict. Mem0 decides ADD, UPDATE, DELETE, or NOOP before data enters the store. Letta rewrites a core-memory block later. A plain file can simply overwrite the old value. Zep/Graphiti alone preserves both sides through bi-temporal invalidation. Historical fidelity increases along roughly overwrite < LLM decision < temporal invalidation, but so does mechanism weight.

Auditability versus operations. Files appear to offer both the best auditability and the lowest operations cost, but their hidden price is context-bound capacity. Index-then-fetch moves the scaling problem into layered retrieval. Zep pays for Neo4j operations to obtain temporal reasoning and point-in-time audits.

There is no silver bullet, only scenario fit. Mem0 suits unattended user memory for an existing agent; Letta suits autonomous agents built from scratch; Zep suits changing or regulated enterprise facts; files suit coding agents.

Letta's filesystem experiment reinforces the point: a plain filesystem with GPT-4o-mini reached 74.0% on LoCoMo, above Mem0's reported 68.5% best graph variant. How the agent manages context can matter more than whether retrieval uses vectors or graphs.

Where Mainstream Products Are Converging ​

ChatGPT, Gemini, Claude, and coding agents are converging on a compressed long-term profile plus raw recent working memory. The profile is compact and persistent; recent conversation provides a delta because summaries are inevitably stale.

ChatGPT optimizes for frictionless, always-on personalization. Saved memories are explicit, user-editable facts such as names, preferences, and goals. They appear with timestamps in the system prompt's Model Set Context. Reference chat history, launched on April 10, 2025, maintains a rolling history and implicit profile rather than literally searching every old chat. It is not directly inspectable. The benefit is zero-wait personalization; the cost is less predictable provenance and auditability.

Gemini optimizes for a single source of truth. Personal Context, launched in August 2025, centers on one structured user_context document. Each short fact carries a Statement, Rationale, and timestamp, then recent raw turns provide the delta. Timestamps make conflict resolution straightforward: today's “I joined a new company” can replace a year-old current role, while “considered moving to San Francisco in June 2024” remains a time-bounded event rather than a permanent identity. Despite Google's enormous data advantage, the design is conservative about what becomes personalization.

Claude optimizes for transparent, work-oriented control through two separate systems. Search and reference chats, introduced in September 2025, performs on-demand RAG through conversation_search and recent_chats, linking results back to the original conversation. Generate memory from chat history synthesizes an editable profile refreshed every 24 hours; edits apply immediately, and the feature expanded to free users in March 2026. Project isolation gives each project its own memory and summary, while the cross-chat summary explicitly excludes project chats. Claude.ai chat memory, the API /memories tool with create/str_replace/insert, and Claude Code's cascaded CLAUDE.md files are three different systems and should not be conflated.

Coding agents have converged even further: codebase index, rules files, and automatic memories are becoming a standard stack. Cursor semantically splits code by function and class, stores embeddings in Turbopuffer, and uses a Merkle tree to check hashes every 5–10 minutes so only changed files are reprocessed; raw code is not uploaded with the embeddings. Its v1.0 Memories sidecar proposes project-scoped entries for developer approval. GitHub Copilot separates repository facts—with citations back to supporting code and validation against the current branch—from cross-repository user preferences. Unused facts can expire after 28 days, while successful use resets the timer. CLAUDE.md, AGENTS.md, .windsurfrules, and copilot-instructions.md tie the ecosystem together as a de facto file-based standard.

Three shared principles emerge: long-term profiles must be compressed, recent raw conversation must compensate for stale summaries, and conflict resolution increasingly depends on time—timestamps, validation-before-use, or refresh cycles.

Do Not Trust Benchmarks Too Much ​

LoCoMo and similar benchmarks can be inflated by aggressive retrieval: provide enough likely answers in the candidate context and the score rises, even if the memory system has little real understanding.

Letta used a plain filesystem, GPT-4o-mini, and minimal prompt tuning to score 74.0% on LoCoMo—higher than the 68.5% reported for Mem0's best graph variant.

Cross-vendor numbers are rarely comparable because evaluation models and retrieval settings differ. Even Mem0's own pages report LoCoMo/LongMemEval values of 92.5%/94.4% in one place and 91.6%/93.4% elsewhere, with the hosted version including proprietary optimizations unavailable to the open-source release. Treat public scores as hints. Evaluate on your own real data.

Four Practical Routes—No Silver Bullet ​

Whatever route you choose:

• Do not adopt a graph database too early. A vector store plus LLM extraction covers most cases; graphs pay off only when relationship reasoning is central.
• Timestamp every memory. Time is the foundation of conflict resolution and temporal queries. Early OpenAI memory entries lacked timestamps and performed poorly on LoCoMo temporal questions: Mem0's graph variant reported 58.13% versus OpenAI's 21.71%. A timestamp is nearly free; without one, “what was I doing last Wednesday?” cannot be answered reliably.
• Separate user, session, and project scopes. This is necessary for relevance and safety.
• Use benchmarks as references, not truth.

Codex's two-stage maintenance pipeline is a strong integrated example:

Phase one distills historical sessions into structured SQLite records. Phase two runs a dedicated consolidation agent inside a networkless, approval-free sandbox and uses the baseline-to-worktree diff in ~/.codex/memories/.git as the signal for ingestion and forgetting. Memory maintenance becomes constrained, incremental, reversible, and informed by usage counts instead of being an unmanaged append operation.

Conclusion: Converging Principles and Two Unsolved Gaps ​

Across open-source systems and mainstream products, two principles are converging.

First, merge before adding. Aggressive append slowly corrupts a memory store with duplicates, obsolete facts, and contradictions. Some systems gate at write time; others consolidate offline. Both recognize that the main conversation agent should not be expected to maintain memory casually.

Second, separate raw records from compressed views. OpenHands keeps events behind tombstones, Codex separates rollout logs from distilled summaries, and Graphiti invalidates edges without deleting them. Daily retrieval uses the compact view; audits and corrections can still return to original evidence.

Two gaps remain.

The first is human review before writing. None of the surveyed systems provides a complete confirmation gate before memory enters the store. AutoGPT uses a two-step confirmation for deletion—find candidates, ask the user, then execute—but writes remain automatic. Hermes can emit a curator dry-run REPORT.md for approval, but only for skill organization. The industry has chosen prompt constraints and automated safeguards over systematic human approval in order to preserve unattended operation.

The second—and more serious—gap is security. Memory systems are naturally vulnerable to indirect prompt injection. Malicious instructions hidden in ordinary-looking content can persuade an agent to read or write memory without the user's awareness. Once poisoned memory is stored, it can influence future behavior repeatedly and invisibly.

Memory can make an agent smarter the longer it is used, but it can also make one incident of poisoning persist. The field is converging on sound engineering principles, yet these gaps remain open—which is precisely why now is such an interesting time to build.

✨Article Summary (AI Generated)

This article uses a 15,400-token login bug fix scenario to dissect the context compaction strategies of three mainstream CLI agents: Codex CLI, Claude Code, and OpenCode. Codex CLI takes a single-layer "handoff summary" approach; Claude Code employs a three-tier progressive mechanism — tool result trimming, Prompt Cache-friendly strategies, and a 9-section structured LLM summary; OpenCode implements "stepped governance" through non-destructive timestamp-based message hiding paired with a 5-heading LLM summary. The article reveals a core insight: the best context management isn't about endlessly expanding memory capacity, but learning to forget with precision.

When using AI agents for deep programming tasks, you've surely encountered this predicament: at first, the AI is sharp and responsive, hitting every mark; but as conversation turns accumulate, it seems to get progressively dumber.

When context is nearly exhausted, the AI rushes to finish, leading to poor results — the community calls this Context Anxiety, and just like us humans, haste makes waste. To keep the conversation going, the agent must shed some memories (compress — compact). But how to shed, what to shed, and how to recover afterward have become key markers of whether an agent runtime is truly mature.

Today, let's tear apart three mainstream CLI agents — Codex CLI, Claude Code, and OpenCode — and see how each performs their "compression magic" when facing the same login bug.

Note: This analysis is based on the open-source repository logic of Codex CLI and OpenCode, along with community reverse engineering and leaked source code verification of the Claude Code runtime.

Scenario Replay: A 15,400-Token Login Bug Fix ​

Imagine you're fixing a login endpoint returning 401 Unauthorized. You summon your AI agent and embark on an intense debugging session.

Here's the complete conversation log:

Just 26 messages, yet they've consumed roughly 15,400 tokens. The five bolded tool results (#5, #8, #13, #21, #26) alone total about 12,500 tokens — 81% of the total. This data was critical during debugging, but once the bug is fixed, it becomes dead weight in the context. If left uncleaned, the next conversation turn might overflow the window and lose the system prompt or the user's core requirements.

Codex CLI: Writing a Crisp "Handoff Memo" ​

OpenAI's Codex CLI (source, Rust implementation) takes an approach that feels very intuitive to humans: summarize and replace.

Its core idea can be captured in one sentence: hand the entire conversation to an LLM to write a "handoff summary," then replace the original history with that summary.

Dual-Path Design ​

Codex offers two compression paths:

1. Local path (compact.rs): The client calls an LLM to generate the summary, compatible with any model provider.
2. Remote path (compact_remote.rs): Directly calls OpenAI's internal API endpoint responses/compact, letting the server handle compression. OpenAI models only.

Note that "local" and "remote" here don't refer to whether an LLM call is needed — both paths require LLM involvement. The difference is where the core "generate summary" step runs. In the local path, the client constructs the summarization prompt (loaded from the built-in template templates/compact/prompt.md), streams the LLM API call via ModelClientSession, and processes the returned result — the entire orchestration happens on your machine, so it works with any model provider. In the remote path, the client sends the prepared conversation history and tool definitions to OpenAI's compact_conversation_history endpoint for server-side summary generation — but the client is far from hands-off. Before the call, it trims overly long function call histories and builds the complete prompt object including tool specifications and system instructions; after the call, it filters the results (e.g., discarding stale developer role messages, keeping only genuine user and assistant content), restores ghost snapshots for /undo functionality, and recalculates token usage.

In short, the remote path only outsources the "compression" step to OpenAI's server — pre-processing and post-processing remain client-side. The advantage is that OpenAI's server likely has dedicated optimizations for this endpoint (such as more economical models or internal caching) that aren't available through the generic API. This reflects OpenAI's vertical integration of their own infrastructure.

The Compression Flow in Detail ​

When taking the local path, Codex first extracts recent user messages (hard-capped at ~20,000 tokens), then sends a brief summarization prompt to the LLM. This prompt has just 4 core points:

You are performing a "context checkpoint compression." Generate a handoff summary for another LLM that will continue the task, including: current progress and key decisions, important constraints and user preferences, remaining TODOs, and critical data needed to continue work.

The keyword is "handoff" — it's not writing meeting minutes, but a briefing that lets the next person (model) hit the ground running.

Applied to our login bug scenario:

Breaking it down:

Notice the before-and-after — all messages collapse to just 4. Codex deeply respects "user intent": it physically deletes all assistant replies and tool-related messages, but preserves all user messages verbatim (#2 and #10).

It then inserts a fabricated assistant message containing a structured handoff summary. This summary includes the task objective, completed items, key architectural decisions, and remaining TODOs. For the new model, it doesn't need to see those massive file dumps and test stack traces — it just needs to know "the tests are fixed" and that's enough.

Auto-Trigger and Fallback ​

When token usage approaches the model's context window limit, Codex automatically triggers compression (no need for manual /compact). If space is still insufficient after compression, it falls back to more aggressive "head trimming" — chopping from the earliest messages to ensure the conversation can continue.

The biggest advantage of Codex's approach is its intuitiveness: the handoff summary concept is something every working professional understands. The downside is it's rather "all-or-nothing" — all AI replies and tool results get replaced by a summary, and if that summary misses a critical detail, it's truly gone forever.

Claude Code: Three-Tier "Precision Forgetting" ​

Anthropic's Claude Code takes a more nuanced approach. Rather than pursuing one-shot physical deletion, it designs three progressively stronger cleanup mechanisms — from light to heavy, avoiding LLM calls whenever possible.

Note: Claude Code is not open source. The following analysis is based on community reverse engineering and public materials; actual implementation may vary across versions.

Layer 1: Tool Result Trimming (Zero LLM Cost) ​

This is the most frequent and lightweight layer. No LLM call needed — it's purely a local rules engine that runs automatically before every request.

The logic is simple:

• Always protect the results of the most recent tool calls (active data can't be deleted)
• Older tool results beyond the protection window → replaced with [Old tool result content cleared] placeholder

Applied to our scenario:

This approach is remarkably clever: it maintains the AI's "flow state." The AI remembers it searched the code (#4's tool_call is still there), and remembers it read files (#7's tool_call is still there) — it just doesn't remember what the search returned or what the file contained. If it truly needs to look again, it will simply re-issue a read_file.

I find this layer's design exquisitely elegant — it achieves "selective amnesia" rather than "total forgetting." Like remembering you read a great book last year but forgetting the specifics — you can always flip through it again when needed.

Layer 2: Cache-Friendly Strategy (Prompt Cache) ​

This is Claude Code's signature move, and the unique differentiating advantage among the three.

Anthropic's API supports Prompt Cache — if the prefix of your message to the API matches the previous request, the server can reuse prior computation results, dramatically reducing cost and latency.

What does this mean? When cleaning messages, Claude Code strives to avoid modifying the first half of the message sequence. It takes a "surgical" approach: trimming only at the tail, ensuring the beginning of the message sequence remains absolutely consistent. The trade-off is slightly lower cleanup efficiency, but the payoff is maximized cache hit rate.

Applied to our scenario. Suppose after Layer 1 cleanup, the message sequence is #1-#26 (tool results already replaced with placeholders). Context still exceeds the limit and needs further trimming. A "naive" approach would delete from the earliest messages — but Claude Code doesn't do that:

The naive strategy on the left deletes the oldest messages, which seems reasonable, but the cost is that the entire prefix changes — the API cache completely invalidates, and the next request must compute from scratch. Claude Code's strategy on the right is the opposite: it would rather delete less, as long as the prefix portion of the message sequence remains identical to the previous request, allowing Anthropic's Prompt Cache to hit.

For long-running tasks (like having the AI help you refactor an entire module), this strategy yields significant cost savings — because most of each API request's content can hit cache, and you only pay for the newly added tail content.

Layer 3: 9-Section Structured LLM Summary (Last Resort) ​

When the first two layers can't prevent context from continuing to grow, the system triggers a final full summary. According to the source code, the auto-compaction threshold is effective context window - 13,000 tokens (where effective window = model context window - min(max output tokens, 20,000)).

However, even when the threshold is reached, the system doesn't jump straight to an LLM summary. When auto-compaction triggers, the system first tries Session Memory Compact — leveraging structured information already in session memory to substitute for a full LLM call. This means most auto-compactions don't even need an LLM call. Only when the session memory path is unavailable or insufficient does the system fall back to the traditional LLM summary flow, generating a structured summary with 9 fixed sections:

1. User's original intent
2. Core technical concepts
3. Files and code of interest
4. Errors encountered and how they were fixed
5. Problem-solving logic chain
6. Summary of all user messages
7. TODO items
8. What's currently being worked on
9. Suggested next steps

This summary has extremely strict requirements — the prompt demands the model directly quote key phrases from the original text rather than paraphrasing everything. This prevents "context drift" (the model subtly diverging from the original meaning during retelling).

Applied to our scenario:

After compression, Claude Code performs a series of post-processing steps — what I call "state reconstruction":

• Injects a lead-in at the beginning of the new conversation ("This session continues from a previous conversation...")
• Automatically re-reads recently edited files (up to 5 files, total budget 50,000 tokens, 5,000 tokens per file), ensuring the AI has the latest code
• Re-declares tool and skill definitions
• Project specifications in CLAUDE.md, as part of the system prompt, remain permanently resident and are unaffected by compression

Users can also attach custom instructions during manual compaction, such as /compact Focus on API changes, to steer the compression toward a specific focus.

Additionally, there's a passive fallback path: when the API returns a prompt_too_long error, the system automatically initiates a reactive compression and retries the request, ensuring users don't encounter an abrupt error from context overflow. To prevent infinite loops from repeated compression failures, automatic compaction pauses after 3 consecutive failures.

Claude Code's approach is the most complex of the three, but also the most "cost-effective" — most of the time it only needs to execute the Layer 1 rules engine cleanup, or complete compaction via the Session Memory path, requiring no extra LLM calls at all.

OpenCode: Prune First, Summarize Later — "Stepped Governance" ​

The open-source newcomer OpenCode (source, TypeScript + Effect-TS implementation) offers a more balanced strategy. In session/compaction.ts, it implements a stepped governance flow: use low-cost measures to free up space first, and only call on the LLM when truly necessary.

Step 1: Prune (Mark as Hidden, Not Physical Deletion) ​

OpenCode's first move isn't deletion — it's "marking." The rules are crystal clear:

• Only execute when pruning can free more than 20,000 tokens (minor cleanups aren't worth the hassle)
• Always preserve the most recent 40,000 tokens as a "safety cushion" (active work can't be touched)
• skill type tool outputs are never pruned (they contain operational instructions)
• Protect the full content of the last 2 user turns

Key design: Unlike Claude Code's placeholder replacement, OpenCode's pruning is not physical deletion. Instead, it stamps old messages with a compacted = Date.now() timestamp, making them "invisible" in subsequent requests. The data is still in the database — just hidden.

Key point: The data isn't truly lost. This leaves room for future history traversal features — if developers need auditing, or if the agent triggers some rollback logic, this data can be pulled back into context. This is a very forward-thinking design.

Step 2: LLM 5-Heading Summary ​

If things are still too bloated after pruning, OpenCode uses a hidden, dedicated agent (without disturbing the user's current interaction) to call an LLM and generate a summary. This summary follows a fixed 5-heading structure:

OpenCode has a wonderfully thoughtful design after summarization: it automatically replays the last user message. This ensures the agent's most recent memory point stays on the user's latest instruction, rather than on some cold summary text. The user is completely unaware that compression happened — your last message gets resent, the AI continues responding, as if nothing happened.

Another highlight: OpenCode follows the user's language. If you've been communicating in Chinese, its summary will also be in Chinese. For non-native English-speaking developers, this is a very friendly design.

I think OpenCode's approach is the most "developer-friendly" of the three — fully open-source code (TypeScript), modern architecture (Effect-TS), and the non-physical deletion design leaves ample room for extension. If you want to deeply customize compression behavior, OpenCode is the easiest to get started with.

The Three Musketeers Face Off ​

Let's put all three approaches side by side:

Input: 26 messages, ~15,400 tokens (same "fix login bug" scenario)

Differences Worth Expanding On ​

On whether to preserve user messages verbatim: Codex chooses to keep user messages intact, compressing only model responses. The advantage is that the AI can always look back at what you said, but the cost is reduced compression efficiency when user messages themselves are lengthy. Claude Code and OpenCode choose to compress everything into summaries — more aggressive but more space-efficient.

On caching: This is Claude Code's most unique advantage. The other two see significant changes in API request content after compression, essentially invalidating previous caches. Claude Code deliberately maintains prefix stability, allowing post-compression requests to still reuse prior caches. For long-running tasks, this translates to meaningful cost savings.

On non-physical deletion: OpenCode's timestamp marking approach is a forward-thinking design. While the current version doesn't implement history traversal, the data isn't truly lost, leaving the door open for future possibilities. Both Codex and Claude Code's compressions are irreversible.

Final Thoughts ​

If I were to describe these three with an analogy:

• Codex CLI is like a senior employee writing a handoff memo. They tear up the previous drafts and hand you a clear status report — blunt but effective.
• Claude Code is like a scholar with the ability to forget with precision. They first erase minor annotations in their books, and only when the bookshelf truly can't hold any more do they condense entire books into one-page outlines. They care deeply about the efficiency of flipping through books (caching).
• OpenCode is like a pragmatic stepped governance administrator. They first box up old files and label them (hide), and only summarize when absolutely necessary. Their most thoughtful touch? After summarizing, they remind you: "The last thing you said was about this, right?"

Ultimately, in 2026, the best context management isn't about endlessly expanding an LLM's memory capacity, but learning how to forget with precision. After all, an agent that remembers everything is often the one most easily distracted by noise.

References:

• Codex CLI: openai/codex (see codex-rs/core/src/compact.rs)
• Claude Code community resources:
  • Claude Code System Prompts
  • Sam Saffron's Gist
  • Claude Code's Compaction Engine
  • Leaked source repository (unofficial leak, used for verification)
• OpenCode: anomalyco/opencode (see packages/opencode/src/session/compaction.ts)

✨Article Summary (AI Generated)

The author shares his experience of distilling Harness Engineering knowledge into a reusable Agent Skill. After systematically studying sources from Anthropic, OpenAI, Martin Fowler, LangChain, and others, he identified seven core layers of harness design: project setup, context engineering, constraints & guardrails, multi-agent architecture, evaluation & feedback, long-running tasks, and diagnostics. The resulting harness-engineering skill covers three scenarios — new project scaffolding, diagnosing poor agent behavior, and continuous improvement — using a progressive-disclosure architecture. Quantitative evaluation showed 100% assertion pass rate with the skill vs. 83% without. The key insight: 80% of agent quality issues stem from harness gaps, not model limitations.

Why I Wrote This ​

Over the past two years, while using various AI coding assistants (Claude Code, Cursor, Copilot, etc.), I kept running into the same problem: Agent performance was inconsistent. While things have generally improved as models get better, the journey has been full of ups and downs.

Sometimes the code it writes perfectly matches the project style. Other times it acts like an intern on their first day — unaware of the project structure, ignoring conventions, and forgetting decisions we'd already agreed on.

I started applying Prompt Engineering techniques like structured prompts, few-shot examples, and demonstrations to stabilize AI output. Then I moved on to Context Engineering to enrich the agent's context and further stabilize its performance.

In recent weeks, a more systematic term has emerged: Harness Engineering.

When an agent performs poorly, 80% of the time the issue isn't the model — it's the Harness. — Anthropic

What is a Harness? Simply put:

• Model = CPU (the compute itself)
• Context Window = RAM (working memory)
• Harness = Operating System (scheduling, constraints, feedback, file system — all the infrastructure that makes the CPU work effectively)

You wouldn't expect a CPU to run efficiently on bare metal without an operating system. Similarly, you shouldn't expect a model to produce consistent output in a project without a Harness.

What I Learned ​

I systematically read articles from the following sources:

• Anthropic — Building effective agents, multi-agent research systems, harness design for long-running agents
• OpenAI — AGENTS.md design patterns, Context Engineering best practices
• Martin Fowler — The engineering philosophy of Harness Engineering ("Relocating Rigor")
• LangChain — Taxonomy of agent frameworks vs. runtimes vs. harnesses
• philschmid — The importance of Agent Harness in 2026
• Independent developers — Hermes Agent's self-evolution, Vue Lynx's design-notes-driven development
• Academic papers — Formalization of natural language agent harnesses

After reading everything, I found that while these articles approached the topic from different angles, their core ideas converged into seven layers:

So I Built a Skill ​

After reading all these articles, I realized these patterns are entirely reusable. Whether your project is a React frontend, Python backend, or Rust CLI tool — the design principles of a Harness are universal.

So I distilled this knowledge into an Agent Skill called harness-engineering.

What It Does ​

This skill has three core use cases:

Scenario 1: New Project Setup

When you start a new project and tell the agent "help me set up Harness engineering," it will:

1. Assess your project type, tech stack, and team size
2. Create an AGENTS.md (a table-of-contents-style agent navigation file)
3. Set up a docs/ directory (architecture, conventions, data models, etc.)
4. Configure the constraints layer (lint rules, type checking, pre-commit hooks)
5. Set up evaluation and feedback mechanisms

Scenario 2: Diagnosing Poor Agent Performance

This is the most interesting scenario. When you start complaining —

• "Why does it keep making the same mistake?"
• "It completely ignores our conventions!"
• "The code quality is terrible"

This skill gets triggered and guides the agent to diagnose gaps in the Harness layer, rather than blaming the model:

Scenario 3: Continuous Improvement

Every time a new reusable Harness pattern is discovered, update it in the skill so other projects can benefit too.

How It's Organized ​

The skill uses a progressive disclosure architecture:

harness-engineering/
├── SKILL.md              # Entry file (<60 lines), routes to specific references
└── references/
    ├── 01-project-setup.md       # Project setup
    ├── 02-context-engineering.md  # Context engineering
    ├── 03-constraints.md          # Constraints & guardrails
    ├── 04-multi-agent.md          # Multi-agent architecture
    ├── 05-eval-feedback.md        # Evaluation & feedback
    ├── 06-long-running.md         # Long-running tasks
    └── 07-diagnosis.md            # Diagnostics

SKILL.md itself is very concise — it acts like a router, directing the agent to read the relevant reference document based on the current scenario. This follows a core principle of Harness Engineering itself: progressive disclosure, load on demand.

Patterns That Impressed Me ​

A few patterns particularly resonated with me. Let me discuss them individually.

"Give a Map, Not a Manual" ​

I saw this concept in a tweet. The traditional approach is to write detailed step-by-step instructions (a manual) for the agent, but this makes the agent fragile — any deviation leaves it lost.

A better approach is to give the agent a map:

# Bad approach (manual)
Step 1: Open src/auth/login.ts
Step 2: Find the handleLogin function
Step 3: Add ... at line 42

# Good approach (map)
Auth system is in src/auth/. Login flow: login.ts → validate.ts → session.ts.
Rate limiting middleware is in src/middleware/rateLimit.ts — follow its pattern.
Every auth change needs a test in src/auth/__tests__/.

A map lets the agent navigate autonomously; a manual turns it into a fragile execution machine.

"Every Mistake → Add a Rule" ​

This pattern comes from cross-referencing multiple articles. The core idea:

1. The agent makes a mistake
2. You fix the mistake
3. Then you add a rule to permanently prevent that type of mistake from happening again

This rule can be a lint rule, a type constraint, a test case, or simply a convention in the documentation. Over time, the Harness accumulates more and more rules, and the agent's error rate for known patterns approaches zero.

This is exactly what Martin Fowler calls "Relocating Rigor" — moving the quality gates that humans enforce through code review, experience, and intuition into automated checks. The agent runs freely within the checked boundaries.

Harness = Dataset ​

This perspective comes from Anthropic. Every agent interaction is a training signal:

• What it tried
• What succeeded
• What failed
• What the fix was

These traces are your competitive advantage. They're the data that makes your Harness better over time — not fine-tuning the model, but optimizing the operating system.

Skill Evaluation: Does It Work? ​

Following the skill-creator process, I conducted a quantitative evaluation of this skill. I designed 3 test scenarios, running with-skill and without-skill versions for each:

The with-skill version passed all assertions in every scenario. The without-skill version had the most gaps in the "new project setup" scenario — it didn't know to create AGENTS.md, didn't know how to organize docs/, and couldn't set up the progressive disclosure context architecture.

Of course, a 17% difference isn't enormous. But the key point is: with the skill, the agent's output was consistent and complete; without it, results were hit or miss. For an engineering practice skill, consistency is more valuable than occasional brilliance.

How to Install ​

This skill can be installed via GitHub:

npx skills add 10xChengTu/harness-engineering

After installation, when working in Claude Code, OpenCode, or other agents that support Skills:

• Starting a new project → the skill auto-triggers to guide Harness setup
• Encountering agent quality issues → the skill intervenes for diagnosis when you start complaining
• Proactive inquiry → "Help me improve this project's Harness"

Final Thoughts ​

Harness Engineering is still a very early field. Models are getting stronger, and constraints needed today might be unnecessary tomorrow — so this skill itself follows a core principle: build for deletion.

If you're also doing development with AI Agents, try adding a Harness to your project. Start simple — an AGENTS.md file, a few lint rules, a progress.md. Then observe how the agent's performance changes.

You'll most likely feel the same way I do: It's not that the model isn't good enough — we just haven't given it a good working environment.

All referenced articles and the complete skill source code can be found in the GitHub repository.