Harneloop
An artifact-aware framework for making task-specific AI agent harnesses improve through real attempts, trace-backed diagnosis, isolated candidates, and evidence-gated promotion.
Agents can finish a task without understanding the artifact.
An agent saying “done” is not evidence that the result is good.
While developing ViperMesh, the recurring failure was not only that an agent made a wrong move. It was that the agent could produce a poor Blender scene, stop, and claim success without a reliable way to inspect the artifact, trace the mistake, or test a better approach.
Generic evaluation loops often reduce a task to a final score. That hides the operational cause of the failure. Harneloop puts the produced artifact, the action trace, the environment, and the evaluation evidence inside the improvement loop so the harness can be changed for a reason.
Improve the harness around the agent.
Harneloop is a development and versioning layer for agent harnesses. It is not another model runtime and it is not only an evaluation dashboard. The operating agent attempts the real task, captures artifacts and traces, diagnoses the gap, proposes a candidate change, and tests that candidate against the same evidence contract.
A candidate can change instructions, context, tools, retrieval, observers, validators, memory, or the environment. The working harness stays unchanged until the evidence supports promotion. That makes improvement inspectable, reversible, and portable across the task environments where the bottleneck actually appears.
Every improvement has to earn its way into the harness.
- 01
Describe the capability
Define the real task, success conditions, required environment, and evidence that can prove the result.
- 02
Map the environment
Record the tools, files, services, models, permissions, and reset path the agent actually depends on.
- 03
Run a real attempt
Let the agent perform the task and retain its actions, state transitions, logs, traces, and produced artifacts.
- 04
Inspect the artifact
Evaluate the output itself, including visual, structural, behavioral, and task-specific checks.
- 05
Trace the failure
Connect the observed issue to the action, context, tool, validator, retrieval, or environment decision that caused it.
- 06
Test a candidate
Create an isolated harness candidate and rerun the same evidence-bearing task before changing the working harness.
- 07
Promote or reject
Promote only an improvement that survives the relevant checks. Preserve rejected candidates as useful evidence.
Portable workspaces for one capability at a time.
Each task-specific workspace is a harness unit: an isolated environment that can be stored, paused, moved, exported, and continued without reconstructing the entire research context.
harness-unit/
target/
environment/
harness/
experiments/
evidence/
regressions/
promoted/
unit.yamlViperMesh made the thesis measurable.
The first published application was a Harneloop-guided ViperMesh harness for Blender spatial reasoning and scene construction. The agent struggled with spatial relationships, so the harness was developed around artifact inspection, scene validation, deterministic tools, and evidence from the actual rendered output.
The benchmark used the same OpenAI GPT 5.5 High model for both harnesses. It compared ViperMesh with the Anthropic x Blender MCP server baseline, keeping the model constant so the measured difference was about the harness and tool surface. Visual evaluations were produced by a neutral, non-biased LLM judge rubric and should be read as scoped benchmark evidence, not a universal claim.
| Target task | Blender spatial reasoning and scene construction |
|---|---|
| Acting model | OpenAI GPT 5.5 High for both harnesses |
| Comparison | ViperMesh versus the Anthropic x Blender MCP server baseline |
| Evidence | Artifacts, traces, renders, scene checks, and benchmark results |
| Framework status | Harneloop v0.0.2 public alpha |
Harness first, when the bottleneck is outside the weights.
Harneloop challenges the reflex to fine-tune or change model weights before understanding the task environment. When the bottleneck is context, tools, feedback, validation, retrieval, memory, or artifact awareness, changing the harness can be more efficient and easier to inspect than changing the model.
This is a research position, not a claim that harness changes always beat fine-tuning. Harneloop exists to make the comparison testable: preserve the environment, isolate the candidate, repeat the evidence-bearing task, and keep the result with the unit.
Useful infrastructure, still an early framework.
Harneloop is currently a public alpha. The core lifecycle, unit structure, CLI workflow, repository schemas, and tests are in place, while broader validation across task families is still ahead. The ViperMesh result is the first evidence that the framework can support meaningful harness evolution, not the final proof of a general law.
Next work includes more task-specific units, stronger artifact validators, regression suites, clearer candidate diffs, and repeatable comparisons across different agent environments.