DeepSeek Harness: Why I'd Test Its Plugin-First Agent Design

DeepSeek Harness makes the agent loop itself a plugin, alongside the router, tool registry, and session log. That is the change I care about: the framework treats orchestration as replaceable behavior rather than a fixed core surrounded by extensions. According to the supplied MindStudio article, DSH is open-source, MIT-licensed, and works with other model providers and OpenAI-compatible endpoints, not only DeepSeek models.

I would evaluate it as an orchestration framework, not another reason to switch models. My interest is whether I can change how an agent works without maintaining a fork—and whether those changes remain understandable when something goes wrong.

I'm working from MindStudio's explanation, not reporting a deployment or an independent benchmark reproduction. That distinction matters, especially for the performance claims.

Why I care about replacing the agent loop

I find the plugin boundary more interesting than the model compatibility. If tools are configurable but the decision loop is fixed, I still have to accept the framework's assumptions about how work progresses. DSH makes that loop replaceable too.

The underlying framework is Cordis. The source describes two properties: removing a plugin reverses its registrations, hooks, and behavior changes; declaring dependencies lets the runtime activate or deactivate components as those dependencies appear or disappear. Cordis calls these temporal and spatial composability.

I read that as a lifecycle contract worth testing, not a blanket promise that every consequence of an agent's work disappears. Removing a tool registration and reversing an action performed through that tool are different requirements. I would not assume the latter from this description.

For the editorial and AI-assisted workflows I work on, that distinction is important. I'd want to remove an experimental capability without leaving its hooks active. I would separately require controls around anything that changes content or publishing state. Plugin cleanup would not replace those controls.

How I'd inspect a DeepSeek Harness configuration

DSH boots a plugin tree assembled from configuration layers. The source describes bundles containing plugins and their configuration, profiles patching a YAML configuration, a home-level patch layer, and command-line overrides. Layers can target plugins by ID, change configuration, or insert plugins.

My first step would be inspecting the resolved tree, not editing a convenient-looking file. The article mentions a configuration-dump command but does not supply its exact syntax, so I would check the project documentation rather than invent a command here.

I would save that resolved configuration with each evaluation run. Otherwise, I could be comparing two supposedly identical setups while a home-level override changes one of them. My acceptance criterion would be simple: another engineer should be able to explain which plugins ran and why.

The four UI modes described in the source—standard, PGC, minimal, and creator—are different plugin bundles. Creator mode can take a plugin description in chat, write the plugin, install it, and verify that it is live.

I would try that in an isolated evaluation environment. Generating a capability and activating it during a session is useful for exploration, but I would want reviewable changes and an explicit approval boundary before adopting that workflow around production credentials.

What do the DeepSeek Harness benchmark numbers prove?

I take the benchmark as a reason to test the harness, not as a procurement decision. MindStudio reports that Frontier Harness Eval ran Kimi K3 through nine harnesses across twelve configurations, using the same 30 tasks and runtime.

In that evaluation, Codex reached a 66.7% pass rate. DSH's creator configuration and Claude Code tied at 63.3%. The reported cost per successful task was about $3.28 for DSH and $18.34 for Claude Code.

Those are results for a particular configuration and task set, not a general price comparison between products. With 30 tasks, one task changes the pass rate by roughly 3.3 percentage points, as the source also cautions.

My takeaway is narrower: I should hold the model constant while evaluating orchestration. Otherwise, I cannot tell whether an improvement came from the model, the harness, or a configuration change. I would also measure review effort. A task marked successful in an evaluation is not automatically a change I'd merge into one of my repositories.

What I'd borrow from Super Agent

Super Agent is the source's concrete example of this architecture. It watches GitHub and Feishu activity, starts work when asked, and returns with a pull request. DeepSeek describes it as a research prototype, not an official product. Its behavior is assembled from DSH plugins and a worker preset rather than a separate fork.

I like the separation between communication and execution. A long-running router session handles human communication and distributes work. Isolated workers perform the coding and report through the router. Once a worker owns a ticket, follow-up comments on that conversation can reach it directly.

The source reports that disabling this direct route increased router decisions by roughly 30%. I would treat that as a useful design clue: after assigning ownership, I would avoid routing every follow-up through the coordinator unless it needs to make another decision.

I also like the deterministic gate before model evaluation. Super Agent checks authorization and whether it was addressed using code before asking a model whether the message represents work. Below is the kind of small gate I would unit-test in an experiment. This is my illustrative JavaScript, not a DSH API or a complete authorization system.

function shouldEvaluateTrigger(trigger, authorizedUserIds) {
  return (
    authorizedUserIds.has(trigger.actorId) &&
    trigger.addressedToAgent === true
  );
}

const authorizedUserIds = new Set(['reviewer']);

console.assert(
  shouldEvaluateTrigger(
    { actorId: 'reviewer', addressedToAgent: true },
    authorizedUserIds
  ) === true
);

console.assert(
  shouldEvaluateTrigger(
    { actorId: 'outsider', addressedToAgent: true },
    authorizedUserIds
  ) === false
);

I would require the event adapter to establish actor identity and address detection before this function runs. I would not populate those fields from an untrusted message's own claims. Passing this gate would permit evaluation, not grant permission to execute arbitrary instructions.

Evaluating DeepSeek Harness on a real repository

I would start with one repository and a deliberately narrow set of changes. My evaluation would separate lifecycle behavior, coding quality, and operational control rather than collapse everything into a single success score.

1. Freeze a baseline. I would record the model, resolved plugin tree, task instructions, and allowed tools. Then I would change one component at a time.
2. Exercise plugin removal. I would register a disposable capability, remove its plugin, and check that the tool and associated hooks are gone. I would also test what happens to components depending on it.
3. Constrain worker authority. I would define what each task may read, modify, and propose. Super Agent's mandate records who requested work, the applicable rules, and its scope; I would inspect how those boundaries are enforced.
4. Test follow-up instructions. I would send corrections while work is active and inspect which session receives them. Super Agent backgrounds shell commands running longer than five seconds; I would check interruption and completion behavior rather than assume background execution solves coordination.
5. Review the result manually. I would track correctness, cost, elapsed time, and the work needed to make the patch acceptable. My preferred setup would minimize total effort, not merely model spend.

On a newsroom CMS, I would begin with an isolated maintenance task, not a publishing action. That would let me inspect patches and permissions without making editorial output the experiment.

Where DeepSeek Harness would not be my choice

I would not adopt DSH just because everything can be replaced. More configuration freedom gives me more combinations to validate. If my current assistant handles a narrow workflow adequately, I would need a concrete limitation before taking on another framework.

I would also distinguish Cordis lifecycle cleanup from operational rollback. The source explains removal of registrations, hooks, and patches; it does not establish rollback guarantees for external effects. I would design and test those separately.

Super Agent's worker preset removes clarifying questions, plan mode, web fetch, and sub-agent spawning. I would consider that a deliberate execution policy, not a universally better agent. For ambiguous requirements, I would want uncertainty resolved before assigning work to that preset.

I would be cautious with the claim that the router gets more efficient under load. The reported stress test showed larger batches and lower time per decision as trigger volume increased, while individual calls became slower. I would test my own arrival patterns and waiting times before translating that into a capacity assumption.

Finally, the source says DeepSeek's core repository has stopped accepting outside pull requests and directs contributors toward plugins. I would check whether my required changes fit that contribution model. If I needed a runtime fix rather than an extension, that policy would weigh against adoption.

I'd run a bounded DSH trial if replacing orchestration is already a real requirement. Before expanding it, I would require a reproducible plugin tree, evidence that removal behaves as expected, and a permissions boundary I can explain without referring to the model's judgment.