Harness Engineering: Why I Stopped Blaming the Model
The model is not your bottleneck. After building two production apps with zero human-written code, I'm convinced the harness — the constraints, verification loops, and enforcement around the model — is the whole game.
Three engineers. Five months. Roughly one million lines of production code, none of it written by hand. That’s OpenAI’s internal Codex experiment, and the number everyone quotes.
The number isn’t the interesting part. What those engineers were doing is. They weren’t writing code. They weren’t even reviewing most of it. They were building the harness — the constraints, the verification loops, the architectural enforcement that let agents produce reliable code at scale. When the team grew to seven engineers, throughput went up, because each new person added more constraints, better linting rules, tighter feedback loops.
I believe that number because I’ve lived a smaller version of it. Fuellytics — a fleet fuel card fraud detection platform with Stripe Issuing, Twilio, Claude Vision OCR, and a five-validator fraud pipeline — took 55 commits across 5 active days. 22 bounded contexts. Zero human-written source code. MetricFlow, a cross-platform ad analytics tool with six data source integrations, took 40 commits over 13 days.
I did not get those results by picking a better model. I got them by spending a year building the thing that wraps the model.
Most people are optimizing the wrong layer
When an AI coding tool produces garbage, the reflex is to blame the model. Claude hallucinated. GPT got confused. So you switch models. You upgrade. You wait for the next benchmark winner and hope it fixes your workflow.
It won’t. The same model that produces slop in a bare chat window produces architecturally consistent, tested, working code when it’s wrapped correctly. The difference isn’t intelligence. It’s structure.
Three evolutions, and only one of them enforces anything
The industry got here in three phases, each subsuming the last:
Prompt engineering taught us to communicate with models. Write better instructions, get better output. But a prompt is a suggestion. The model can ignore it. There’s no enforcement, only hope.
Context engineering taught us to inform models. Karpathy’s framing — what the model sees matters more than how you ask. Instead of hoping it remembers your coding standards, you inject them. Real progress. The model can still ignore what you show it.
Harness engineering teaches us to constrain models. A custom linter doesn’t ask the agent to respect your architecture — it rejects code that violates it. A test suite doesn’t hope the logic is right — it verifies. As OpenAI’s Codex team put it: “Not documented. Enforced.”
That’s the whole philosophical core. Models are probabilistic. Rules are deterministic. Use the deterministic thing to bound the probabilistic thing.
What enforcement actually looks like
Every major tool now supports a rules file — CLAUDE.md, .cursor/rules/, AGENTS.md. Those are the starting point, not the harness. They’re still context engineering wearing a config file.
OpenAI’s team went further, with a strict layered architecture and a directional dependency chain:
Types → Config → Repo → Service → Runtime → UICode could only depend forward. A Service could import from Repo; Repo could never import from Service. And that rule wasn’t in a README the agent might skip — it was enforced by custom linters (themselves agent-generated), structural tests, and CI jobs that blocked violating PRs.
CodeMySpec, the harness I build, does the same thing a different way. It’s 368 modules across 46 bounded contexts, and the dependency rules are enforced at compile time by Elixir’s Boundary library. Contexts export schemas only — never repositories, never queries. BDD specs are constrained to test through the web layer, so an agent can’t quietly reach around the interface to make a test pass.
I deliberately constrained the harness to one stack — Elixir, Phoenix, LiveView — because there’s basically one idiomatic way to do most things in that ecosystem, and because the environmental complexity stays inside the application. No Kubernetes, no Docker orchestration, no microservice mesh. Fewer degrees of freedom is the point. The agent can’t wander into a bad decision if the bad decision doesn’t compile.
Verification is the part everyone skips
When OpenAI published their harness engineering write-up, Birgitta Böckeler responded on Martin Fowler’s site with the sharpest critique I’ve read: the piece was all about internal code quality and maintainability, and said almost nothing about “verification of functionality and behaviour.”
The code was clean, layered, documented, structurally sound. But did it work?
This is the gap that has cost me the most time, and I have receipts. My agents built Potemkin villages. They’d hit a FunctionClauseError, wrap the call in a try/catch, render a green “Success!” flash message, and move on. Then the QA agent would load the page, see the flash, and mark the scenario as passing. Two agents collaborating in good faith to produce a passing test over broken functionality.
Nobody lied. Every instruction was followed. The harness just measured the wrong thing.
Anthropic hit the same wall from a different angle: Claude would mark features “complete” without verifying they worked, until the harness explicitly required browser-based end-to-end testing. Once they added browser automation, “the agent was able to identify and fix bugs that weren’t obvious from the code alone.” They also had to add strongly-worded instructions like “It is unacceptable to remove or edit tests” — otherwise agents would weaken the test suite to match a buggy implementation instead of fixing the implementation.
The principle underneath all of this: agents take the path of least resistance, and the verification standard you set is the verification standard you get. If your harness only demands unit tests, unit tests are what you get, and they’ll pass. If it demands a browser clicking through the real app and asserting on outcomes, you get that instead.
The fix that finally worked for me was making QA assert on outcomes, not UI elements. Don’t check that a success message rendered. Check that the row exists in the database, that the charge posted, that the flag cleared.
Four layers, and skipping one ships that class of bug
Every testing layer catches a different category of failure, and none of them is a superset of the others:
- Unit tests catch implementation errors. They have no idea what the user wanted.
- BDD specs catch requirement misunderstandings. They don’t test the running app.
- Story QA catches bugs in the real environment. It doesn’t test cross-feature paths.
- Journey QA catches seam bugs between contexts — the ones that only appear when features meet.
Skip one and that entire class of bug ships. Fuellytics had a fraud vulnerability where a flagged driver could clear their own flag without submitting the required photos. The BDD spec for that story passed cleanly. The QA agent caught it by walking the actual flow. Across that build, outcome-based QA surfaced over 100 issues that every other layer had signed off on.
Harnesses are fractal
The thing that made all of this click for me is that harnesses stack. They compose:
Level 0: The raw model
Stateless. No tools. No memory. Pure text generation.
Level 1: Claude Code wraps the model
Agent loop, tool use, context management.
Level 2: CLAUDE.md + hooks + MCP servers wrap Claude Code
Project rules, verification, external integrations.
Level 3: A platform wraps the configured agent
Specs, lifecycle orchestration, multi-phase verification.
Level 4: CI/CD wraps the platform
Deployment gates, integration tests, production monitoring.Every level adds constraint and verification. Someone running Claude Code with no rules file is operating at Level 1. A team with a full stack is at Level 3 or 4. Both of them will tell you they’re “using Claude.” They are using fundamentally different systems, and they will get fundamentally different results.
But my codebase is fifteen years old
Böckeler’s other fair criticism: everything OpenAI described was greenfield. Constraints from day one. What do you do with a decade-old codebase with no architectural rules, inconsistent tests, and patchy docs? Turning on enforcement there is like running a static analyzer for the first time and drowning in ten thousand alerts.
That’s real, and the answer is incremental. Start with what already exists — pre-commit hooks, whatever linters you have, whatever CI already runs. Pick one boundary that actually matters and enforce that one mechanically. Then use the agent to build the next piece of harness, which is the genuinely fun move: having the agent generate the linters and structural tests that will constrain the agent in future sessions.
And be patient about it. OpenAI spent five months on their harness. I’ve spent about a year on mine.
The model is the commodity
HumanLayer said it best: “Agents aren’t hard; the harness is hard.”
Model intelligence is getting commoditized fast, and every lab is racing to hand you more of it for less money. The harness is the part that’s yours. It’s the difference between a demo that impresses your team and a system that ships to customers.
Generation is trivial now. Managing the velocity is the hard part. CodeMySpec can produce an entire bounded context — schema, repository, LiveView, tests, BDD specs — in minutes. Let it run unchecked and you get 100,000 lines that compile, pass their own tests, and don’t work.
If you’re using an AI coding tool today and you haven’t configured its rules file, haven’t set up verification hooks, and haven’t structured your project so an agent can navigate it, you’re running at Level 1 in a world where Level 3 is sitting on the shelf.
The model isn’t your bottleneck. It hasn’t been for a while.
I write about harness engineering, spec-driven development, and making AI-assisted development actually reliable over at CodeMySpec, where this argument is part of a longer series on the anatomy of agentic coding systems.
Sources
- Harness Engineering — OpenAI
- Unlocking the Codex Harness — OpenAI
- Effective Harnesses for Long-Running Agents — Anthropic
- Harness Engineering — Birgitta Böckeler, on martinfowler.com
- The Anatomy of an Agent Harness — LangChain
- Skill Issue: Harness Engineering for Coding Agents — HumanLayer
- The Third Evolution — Epsilla
- OpenAI’s Agent-First Codebase Learnings — Alex Lavaee
- Context Engineering — Simon Willison, on Karpathy