Harnessing Coding Agents: From Brief Glory to Sustained Excellence

Coding agents often start strong, executing tasks with precision and writing tests diligently. However, over time, their performance deteriorates, leading to errors and inefficiencies. The solution isn't a more advanced model but a robust harness that includes a constitution, autonomy boundaries, context maintenance, durable memory, a review panel, and post-edit hooks. Let's explore how these components can transform a coding agent from a fleeting success to a long-term asset.

Why the Harness Matters More Than the Model

The core of a coding agent's success is not the model itself but the surrounding infrastructure—the harness. This includes prompts, tools, context policies, hooks, sandboxes, sub-agents, and observability. The effectiveness of a coding agent over time is significantly influenced by the harness rather than the model alone. Consider the example from the Terminal-Bench 2.0 benchmark, where LangChain improved a coding agent's performance significantly by optimizing its harness while keeping the model unchanged. By focusing on components such as system prompts, tools, and middleware features like self-verification and loop detection, the harness can elevate the agent's capabilities without altering the underlying model.

Addressing Decay with Specific Artifacts

Four main areas of decay affect coding agents: discipline, context, memory, and review. Each of these can be addressed with specific artifacts:

• Discipline: A constitution file (CLAUDE.md / AGENTS.md) outlines the non-negotiable aspects of the agent's operation, including workflow loops and quality standards.
• Context: A tiered CONTEXT.md helps maintain relevant information about the repository and modules, ensuring agents don't have to relearn the same things.
• Memory: Durable memory files and a tiny knowledge graph help agents remember important facts across sessions.
• Review: A multi-lens agent panel provides diverse perspectives on changes, ensuring thorough reviews.

Implementing a Constitution

The constitution file is a lean document that contains workflow loops and quality requirements. It's essential for maintaining discipline and should be concise, as it is read in every session. This is where you define the boundaries of the agent's autonomy, preventing it from executing irreversible actions like pushing to a remote or deploying. By establishing a clear set of rules, the constitution helps ensure that the agent adheres to the desired workflow and quality standards throughout its operation.

Maintaining Context with Tiered Files

To prevent context decay, a tiered system of context files is necessary. These files provide a structured way to store information about the repository and individual modules. The root CONTEXT.md serves as a map, while per-module files provide detailed information relevant to specific parts of the codebase. This system ensures that context is always up-to-date and accessible, reducing the need for agents to relearn information. The rule that stale context is a bug and should be fixed in the same change that invalidated it helps maintain the integrity and relevance of the context information.

Building Durable Memory

Durable memory files and a knowledge graph allow agents to retain important facts across sessions. Each file contains one fact, indexed for easy access. The knowledge graph helps agents understand the relationships between different parts of the code, answering questions like "what breaks if I change this?" This is particularly important for maintaining consistency and preventing regressions, as the memory and knowledge graph provide a comprehensive view of the code's dependencies and interactions.

Establishing a Review Panel

A review panel consists of sub-agents focused on specific lenses, such as product, architecture, infrastructure, security, and test engineering. Each sub-agent evaluates changes based on its expertise, providing a comprehensive review that catches issues a single reviewer might miss. Only high-confidence BLOCKER or MAJOR issues gate the process, ensuring that the review is both thorough and efficient. This multi-lens approach allows for a more nuanced assessment of changes, identifying potential problems from multiple perspectives and enhancing the overall quality of the codebase.

Using a Post-Edit Hook

The post-edit hook ties everything together by firing after every edit and providing a checklist to update context, memory, and decision logs. This ensures that changes are reviewed before they become part of the codebase, maintaining the consistency and accuracy of information across the project. By prompting the agent to make necessary updates in the same diff, the post-edit hook helps prevent silent rewrites and ensures that all changes are transparent and reviewable.

Applying These Practices in Real Projects

In my experience working with GitHub’s infrastructure for agentic development, implementing a robust harness can significantly improve the reliability of coding agents. For a newsroom CMS platform like the one I work on, where timely and accurate content delivery is critical, these practices ensure that the system remains efficient and error-free, even as complexity grows. By focusing on the harness rather than the model, developers can create systems that are not only effective but also resilient to the challenges of long-term operation.

{
  "permissions": {
    "allow": ["Read","Edit","Write","Grep","Glob",
              "Bash(git add:*)","Bash(git commit:*)",
              "Bash(npm test:*)","Bash(npm run:*)","Bash(npm ci:*)","Bash(pytest:*)"],
    "ask":   ["Bash(git push:*)","Bash(npm publish:*)","Bash(aws:*)","Bash(terraform apply:*)","Bash(kubectl apply:*)"],
    "deny":  ["Bash(git push --force:*)","Bash(git push -f:*)","Bash(*--no-verify*)"]
  }
}

Limitations and Considerations

While these practices can greatly enhance the performance of coding agents, they are not without limitations. For instance, the reliance on durable memory and context files requires diligent maintenance to avoid becoming outdated. Additionally, the review panel, while comprehensive, can introduce overhead in terms of time and resources. It's crucial to balance the thoroughness of reviews with the need for efficiency.

Furthermore, in environments where multiple agents operate concurrently, coordination can become challenging. As discussed in my earlier post about Node.js job tracking, ensuring that agents do not interfere with each other's tasks is vital. This requires clear ownership maps and a well-maintained decision log to avoid conflicts. By coordinating through shared state rather than direct communication, agents can operate efficiently and avoid conflicts, much like insect colonies that coordinate complex tasks through environmental changes rather than verbal communication.

Example: Implementing a Durable Memory System

Consider a scenario where an agent repeatedly encounters a bug that has been previously fixed. By using a durable memory system, each time the bug is corrected, a 'fact' file is updated with the details of the fix. This could include the date, the nature of the bug, and the solution applied. When the agent encounters a similar issue in the future, it can reference this durable memory, quickly identifying the correct course of action without re-discovering the solution independently. This not only saves time but also prevents the introduction of previously resolved bugs, maintaining the stability and reliability of the codebase.