Harness Engineering: Why AI Writes Faster but Development Doesn’t Speed Up – And How We Fixed It

Core question this article answers: If AI can write code faster than ever, why hasn’t our overall delivery velocity improved proportionally – and what can we do about it?

Over the past two years, AI coding has made a staggering leap. It went from “producing runnable code” to “autonomously writing entire features.” But when we dropped this capability into real business scenarios, multi‑person collaboration, and legacy systems, we hit a strange paradox: the faster AI wrote code, the slower our overall development rhythm seemed to move.

We tracked a widely cited metric – “percentage of AI‑generated code” – which kept climbing. Yet when we looked at actual release cadence, the efficiency gains were far less impressive. Between code output rate and real throughput, a chasm had opened.

This wasn’t unique to us. The OpenAI Codex team, in their well‑known blog on Harness engineering, repeatedly emphasised one observation: “Early progress was slower than expected, not because Codex lacked capability, but because environmental specifications were unclear.” The whole industry seems to be converging on the same realisation: we need to build a stable “working environment” for our models.

That engineering layer has recently been named Harness Engineering. It is not about teaching the model how to answer – it’s about designing how the model works. In this post, I’ll share our team’s journey: the pitfalls we stumbled into, the trade‑offs we made, and the problems we still haven’t solved.

Why Do We Even Need Harness Engineering?

Core question this section answers: If our AI models are already so smart, why do we need to wrap them in such a heavy engineering framework?

A widely cited equation captures it neatly:

Agent = Model + Harness

Mitchell Hashimoto, who coined the term, gave an even more grounded definition: “It is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”

In plain terms, we shift our engineering focus from “did the model say the right thing this time?” to “does the model do the whole job reliably?”

This is actually the third wave in a natural evolution of AI engineering:

Phase Timeframe Focus
Prompt Engineering 2022–2024 Single‑call quality – how to phrase a request to get a better output.
Context Engineering 2025 Per‑step information – what to feed the model, in what form.
Harness Engineering 2026 End‑to‑end task reliability – execution environment, tool coordination, state management, feedback loops, constraints, and verification.

These are not replacements; they stack. Prompt teaches the model how to talk. Context ensures it has the right information on the job. Harness gives it a stable workplace to keep working. The arrival of Harness Engineering signals that the first two layers are now mature – the remaining bottleneck has moved outside the model.

A Concept Born from Practice, Not Theory

Interestingly, “Harness Engineering” wasn’t invented by someone in a lab. It emerged organically:

  • From August 2025, the OpenAI Codex team validated in agent‑first internal experiments that environmental design, context organisation, tool abstraction, feedback loops, and control systems were as critical as model capability.
  • In February 2026, Mitchell Hashimoto named these practices “harness engineering.”
  • LangChain later formalised the boundary as Agent = Model + Harness.
  • Thoughtworks decomposed it into guides and sensors.
  • Academia began using the ETCLOVG seven‑layer taxonomy for systematic analysis.

There is no single standard definition yet, but a clear practical path exists: build an engineering environment that is executable, constrainable, verifiable, and feedback‑driven for your Agent.

The Efficiency Gap We Experienced

Back in our own team: many developers now rely on AI coding daily. A small independent module can go from idea to runnable code in the time it takes to drink a coffee. But when we measured output, we found the same oddity: the “AI‑written code percentage” was high, but release velocity wasn’t keeping pace.

We traced this to three root causes:

1. Development is never just “writing code.” Brooks’ No Silver Bullet distinguished accidental complexity (syntax, tools, platforms – the “translation cost”) from essential complexity (conceptual structure, alignment with the real world, volatile requirements). AI cuts accidental complexity, but essential complexity remains untouched – in fact, with more code produced, downstream alignment, review, and maintenance become heavier.

2. Local acceleration merely shifts the bottleneck. If you make “writing” ten times faster but leave understanding, alignment, verification, and consolidation untouched, the total cycle time is still dictated by the untouched parts. The bottleneck simply moves from “write” to “review/test/maintain.”

3. AI can’t see our implicit engineering constraints. Team conventions, domain knowledge, historical dependencies – if not explicitly fed in, AI is blind to them.

In other words: when AI drives the cost of “writing code” to near zero, the real bottlenecks become visible – and they were never about writing. They are about understanding, aligning, tracing, consolidating, and verifying. And those are precisely the areas where current AI tools perform worst.

Translated into the Harness terminology: we were not solving Prompt (models are smart enough); we were not solving Context (retrieval and long‑context tools are mature). We were hitting the Harness layer – how to make AI verifiable, feedback‑driven, fixable, iterative, and sustainable inside our own engineering ecosystem.

Our Practical Framework: Two Tracks and One Long‑Term Memory

Core question this section answers: What engineering system do we need to build so that AI delivers stable, predictable results in real business scenarios?

Our goal, in one sentence: “AI‑driven full‑cycle development – human proposes requirements → AI understands → AI executes → human confirms.” This covers requirement clarification → design → implementation → testing → deployment → archival, with parallel frontend/backend flows across DEV/TEST/OPS, plus online operations and alerting.

We structured the whole system as 2 Tracks + 1 Long‑Term Memory:

  • Track 1: End‑to‑end development delivery – handles “pre‑release.”
  • Track 2: Online operations – handles “post‑release.”
  • Long‑term memory (Knowledge Base) – makes AI truly “understand” our business, systems, and online quality.

Track 1: End‑to‑End Delivery – SpecWorker Project Implementation

The core problem for end‑to‑end delivery is: no matter who uses it, or which project it runs on, AI’s output quality must be stable and predictable. We tackle this across three layers.

Protocol Layer: Input/Output Contracts for Every Step

What it does: defines exactly what the input and output of every AI step must be.

Why we need it: There is no tacit understanding between you and AI. You think you’ve said it clearly; it thinks it understands; only at the end do you discover the mismatch. Humans can rely on unspoken conventions; humans and AI must rely on explicit contracts. The Protocol Layer is that contract.

It mandates four things:

  • Each step must produce documents in a specified format.
  • Documents must follow standard templates.
  • After writing, machines automatically check compliance.
  • Every change is recorded as a delta, preserving full history.

Result: AI no longer “freestyles.” The format is fixed, content is verifiable, history is traceable. When something breaks, we can pinpoint which step caused it.

Pipeline Layer: Standardising the “Requirements → Release” 6+1 Stages

The Pipeline layer standardises the entire workflow so that when AI runs the long chain from requirements to release, it doesn’t lose context, evidence, or discipline.

We define 6 core stages plus one optional precursor: P0 brainstorming (optional) → P1 requirements → P2 design → P3 implementation → P4 e2e‑test → P5 deploy → P6 archive.

P1 Requirements: TAPD Pull + Measurable AC + Test‑Cases from the Same Source

Core problem: The “understanding and alignment” phase is where AI‑driven development most easily collapses. AI writes the feature, but no one can later recount why it was written that way. The key pain point: requirement scope must be nailed down at P1, otherwise everything downstream goes off‑track.

Our approach:

  1. TAPD as the single source of truth: Pull the official requirement description from TAPD as the “original scope” paragraph in requirements.md. AI is not allowed to paraphrase user words; it can only quote from TAPD.
  2. Acceptance Criteria (AC) must be testable: Each requirement is broken into WHEN (precondition) → THEN (system SHALL…) , with AND clauses. Vague statements like “performance should be good” are prohibited.
  3. test-cases.md is generated from the same AC list: P1 produces both requirements.md (for P2) and test-cases.md (for P4) from the same set of acceptance criteria. Downstream P4 does not re‑interpret requirements – it simply runs the pre‑defined test cases.

Trade‑off: P1 does not solve “what the user really wants” – that must be done by humans. We only solve “how AI does not distort what has already been expressed.”

P2 Design: Contract‑First + sandbox_mode + D‑x Change Points

Core problem: Traditional design.md is written for humans – background, rationale, architecture diagrams. AI cannot read such documents effectively. It needs a machine‑readable contract: API signatures, error codes, state machines, mandatory fields. If P2 doesn’t pin these down, P3 implementation will invent its own – and downstream code reviewers have nothing to compare against.

Our approach:

  1. Contract‑first: API signatures, data models, and mandatory fields are written as Markdown tables + Mermaid sequence/data‑flow diagrams. design.md is a contract, not an explanation.
  2. sandbox_mode field for frontend: Top of the frontend design.md has a mandatory sandbox_mode: true/false field – when true, P3 writes changes into a sandbox directory (off the main path); when false, it writes directly. This guides AI on whether to isolate changes.
  3. D‑x change‑point breakdown: A list of D‑1, D‑2, D‑3… change points, each annotated with “file:line @ function” + “purpose” + “implementation” + “key code snippet.” P3 works through this list one by one, and the code reviewer also checks against it.

Trade‑off: design.md is not required to be “perfect” – only “machine‑readable.” Any field left for “we’ll decide during implementation” must be explicitly marked as “to be clarified” or “to be confirmed” – no hidden assumptions.

P3 Implementation: D2C + 5‑Round UI Calibration (95% threshold) + 3‑Tier Code Review

Core problem: Implementation is where things most often go wrong. AI writes fast, but whether it writes correctly, consistently, and stably depends entirely on downstream safeguards.

Frontend D2C + UI calibration:

  • We break “restoring UI from Figma” into 3 serial Skills (plus a fixer subagent):

    1. specworker-d2c-get-figma – pulls design specs, localises assets, generates pages.config.json.
    2. specworker-d2c-refactor – routes by slug, supports full/incremental, sandbox/direct write, Hippy/generic Web templates, and multi‑slug parallelism.
    3. specworker-ui-calibration – runs automatically after refactor.

UI calibration self‑healing loop:

  • Take screenshots → compute pixel‑difference + SSIM.
  • If either metric < 95%, trigger a fix loop (max 5 rounds).
  • Each round invokes specworker-ui-calibration-fixer-fe to apply local modifications based on diff + current DOM.
  • Re‑compare after each round.
  • If after 5 rounds the threshold is not met, fallback and output a summary – human decides next steps (additional rounds, design adjustment, or skip).

Backend 3‑tier contract review:

  • After each P3 implementation group, specworker-code-reviewer SubAgent is called automatically. It compares the implementation against design.md and outputs three tiers:
Tier Meaning Action
Critical Contract violation / inconsistent API signature / missing error code Must be reviewed and signed off by a human – AI cannot auto‑pass.
Important Non‑critical but must be marked – known deviation + reason Explicitly logged in evaluation.
Suggestion Style, naming, comments – discretionary Free to ignore.

Key rule: The reviewer reads git diff first, not full files. It does not handle code style – that’s left to linters. It only checks the contract.

P4 Integration Testing: End‑to‑End + Backend API Tests (Dual‑Flow with Self‑Healing)

Core problem: Testing is the stage where AI most easily “fakes completion” – it says “passed” without real validation. Frontend and backend testing are fundamentally different: frontend requires browser/mini‑program UI interaction and screenshot comparisons; backend requires HTTP requests, log inspection, and database checks. Both flows are necessary, but their capability stacks are independent.

Frontend testing:

  • Dispatches by project type:

    • Web automation: specworker-web-automator-test (Playwright) runs end‑to‑end cases and takes screenshots.
    • Mini‑program automation: uses cloud‑based real‑device testing.
    • Generic automator: converts test-cases.md into executable scripts.

Backend testing (self‑healing):

This is the part worth expanding – the difficulty is not in running tests, but in automatic root‑cause analysis when they fail. The full chain:

  1. specworker-api-test Skill generates Node.js (native fetch) test scripts from api_test_cases.md produced at P2.
  2. It collects environment variables (host/token/test accounts) from deploy.md / .env.test and executes the tests.
  3. On failure, specworker-api-test-debugger SubAgent takes over:

    • Extracts the trace-id from the failed request.
    • Automatically queries CLS logs, MySQL data rows, and Redis key states using a fixed SOP.
    • Produces a diagnostic report: “root cause + suggested fix.”
  4. The diagnostic report is attached to the implementation Agent, which fixes the code and re‑runs the specific case.
  5. If re‑run passes, the case is closed. If the same case fails after 3 diagnosis rounds, it stops with a “requires human intervention” flag – the developer then invokes specworker-debugging Skill for manual root‑cause analysis.

Trade‑off: The debugger cannot handle cases where “logs were never printed.” Such situations must fall back to human‑assisted debugging.

P5 Deployment: Frontend Git Conventions / Backend deploy.md Dynamic Parsing (Dual‑Flow)

Core problem: Deployment is the hardest stage to guard – once it goes live, the cost of errors jumps from “rework” to “production incident.” Frontend and backend deployment are also different: frontend pushes build artifacts to CDN/test environments; backend releases images to K8s plus database changes. Both require a discipline‑layer score threshold: total_score ≥ 95, with at most 3 remediation rounds (consistent across P1–P6).

Frontend P5:

  1. Git commit convention: commit message must follow <type>(<scope>): <subject>, with type limited to feat / fix / refactor / style / test / chore.
  2. Test environment deployment + status polling: after push, trigger deployment per the knowledge‑base CI/CD spec. Poll for status every 30 seconds for up to 10 minutes. If timeout, output a warning and ask user to decide – not auto‑fail.
  3. Deployment artifact saved: produce {change_dir}/frontend/deploy.md following the standard template.

Backend P5:

  1. deploy.md task parsing: P2 already wrote a deployment plan. P5 parses it into three task types: database changes / pipeline releases / others.
  2. SQL changes require explicit human confirmation: no DDL/DML is executed automatically – the generated SQL diff must be reviewed and confirmed by the user with a yes/no. This applies even to Critical fixes.
  3. Pipeline release + polling: after invoking the release API, poll until success/failure. On failure, automatically pull K8s pod logs and attach them to the deployment report.
  4. Scorecard ≥95 gate: P5 score must be ≥95, with at most 3 rounds of fixes. (In fact, the entire P1–P6 pipeline enforces the same ≥95 score.) P5 is highlighted because it’s the last gate before release – no “archive later” is allowed if deployment hasn’t succeeded.

Trade‑off: P5 does not handle canary strategies or rollback decisions – those must be decided by SRE. AI only “executes the plan” and “reports failures.” The mandatory SQL confirmation may seem slow, but it’s a hard rule we established after production incidents – better slow than wrong.

P6 Archive: changes‑sync + knowledge‑sync + specs‑generator

Core problem: Archiving is the stage most often skipped – code merged, tests passed, deployment done – who has patience to write archive docs? But skipping archive costs us later: when a similar requirement comes, AI can’t find the previous solution and starts from scratch; past pitfalls repeat. Archiving is not “documentation for documentation’s sake” – it’s compound interest.

Our mandatory three‑step suite:

  1. changes‑sync: Align the actual git changes with the design/planning descriptions – ensure “what the code did” matches “what the documents said it would do.” If not, force update the docs (or go back to P3).
  2. knowledge‑sync: Distill “repeatedly used design patterns, pitfalls encountered, and agreed contracts” from the current change into the project‑level knowledge base specs/, so that next time a similar requirement arrives, P1 can grep them.
  3. specs‑generator: Based on delta-spec.md (with four markers: ADDED / MODIFIED / REMOVED / RENAMED), incrementally merge into the corresponding sections of specs/[module]/spec.md – avoiding “full copy → knowledge base bloat.”

Delta Spec is the soul of P6: it does not copy the entire change’s documents into specs; it only marks what was added, modified, removed, or renamed. The generator performs an incremental merge accordingly.

Observability on the Pipeline: Traceability → Retrospectability → Measurability

Core question this section answers: When the executor shifts from human to AI, how do we trust that it actually did what it claims?

Observability is not “nice‑to‑have” – it is a prerequisite for trusting AI. We break it into three dimensions:

Traceability – turning “I’m done” into machine‑readable evidence:

  • .phase-metrics.jsonl – one JSON line per phase, recording phase/action/timestamp/duration_ms/token counts/lines added/deleted/files changed, and estimated cost in USD.
  • evaluation.md – an independent evaluation skill scores each phase (dimensions/score/conclusion), with total_score ≥95 required to proceed, max 3 rounds.
  • Report API payload – sends fixed fields as event_type=post:phase:end to a backend service (fire‑and‑forget). Even if upload fails, the local jsonl remains for offline retry.

Retrospectability – from failure to root cause automatically:

Failure Type Retrospective Path Convergence Point
UI rendering deviation Screenshot → pixel + SSIM → <95% triggers fixer loop (max 5 rounds) Pass 95% or escalate to human
API test failure trace‑id → CLS logs → MySQL rows → Redis keys (SOP fixed) “Root cause + suggested fix” report
Cross‑stage retry waste summarize-report aggregates which skills/agents were called, token consumption, and retry steps Write back to knowledge base; mark even successful retries

Key rule: The SOP is hard‑coded – no agent freedom. This explicit SOP converts “implicit human debugging experience” into a fixed search path for the agent.

Measurability – turning “is this working?” into numbers:

We track four metric families:

  • Token / cost – input/output tokens, auto‑converted to estimated cost.
  • Duration – phase start‑to‑end in ms.
  • Retry / failure rate – which phases repeatedly fail, where (from summarize-report).
  • Code change volume – lines added/deleted, files changed.

Discipline Layer: Hard‑Coded Gates That AI Cannot Bypass

Core question this section answers: If AI is so capable, why do we need such strict discipline gates? Because AI has a bad habit – it takes shortcuts.

AI will skip tests and write code directly; guess a fix without root‑cause analysis; say “done” without verification; give itself high scores. These are not occasional – they are inherent tendencies. So we build a defence for each shortcut pattern:

Shortcut Pattern Discipline Gate
Skip tests before coding TDD – force tests first
Guess fixes without analysis Debugging – force root‑cause analysis first
Claim “done” without proof Verification – demand runtime evidence
Code deviates from design Review – compare against contract point‑by‑point
Self‑scoring too high Evaluation – use an independent SubAgent to score

Each gate is hard‑coded into the pipeline – not “suggested.” They are embedded, act as checkpoints, and trigger immediate blocking if violated.

Track 2: Online Operations

Core question this section answers: When an alert fires after release, how does AI reliably fix it?

The development pipeline handles “pre‑release.” The operations track handles “post‑release.” They share the same knowledge base, the same trace‑id retrieval SOP, and the same scoring thresholds – they are dual designs of the same Harness, differentiated only by input trigger.

The complete operations chain has 7 steps:

  1. Alert trigger / automated inspection: monitoring alerts (error‑rate spikes, latency surges, business metric anomalies) + periodic health checks (critical SLIs) feed in from two sources.
  2. Deduplication and correlation: merge duplicates, link alerts by call chain to avoid dozens of alerts from the same root cause.
  3. Evidence collection: automatically pull trace‑id chain, CLS logs, MySQL rows, Redis key states, and relevant change records following a fixed SOP.
  4. Root‑cause analysis: AI produces a “hypothesis + evidence + impact” triple – no pure guesses allowed.
  5. AI / human fix: low‑risk (copy, null pointer, missing field) → AI directly raises a PR; high‑risk (database changes, canary strategy) → requires human sign‑off.
  6. Regression validation: re‑run the original failed case. If pass → close; if fail → escalate.
  7. Archive: write back “how the alert happened, how it was fixed, and under what conditions it would recur” into the knowledge base – so that next time, root‑cause analysis hits directly.

Why separate this as its own track? Because the development pipeline is proactive (human proposes requirements → AI executes), while operations is reactive (system alerts → AI responds). Their entry points, cadence, and discipline points differ – but they are two sides of the same Harness when they share the knowledge base.

Knowledge Base: AI’s Long‑Term Memory

Core question this section answers: How do we prevent AI from having to re‑learn everything from scratch on every new requirement?

Without a knowledge base, every new requirement forces AI to understand the context from zero – the so‑called “compound interest” never materialises. We structure the knowledge base as a set of specifications + an operational lifecycle.

Knowledge Base Specification: Structure / Organisation / Content Requirements

Two knowledge bases co‑exist, each managing its own scope:

  • Project‑level specs/ – long‑term product assets: business rules, technical architecture, API contracts, glossary. Granularity: per product/service.
  • Change‑level knowledge-spec/ (the change directory) – each iteration has its own directory, containing requirements.md, design.md, planning.md, test-cases.md, delta-spec.md, and archive/. Granularity: per change.

They are linked via index.md. During P1 analysis, the system can jump from the change directory to specs to find historical similar changes, and vice versa.

Five‑directory layered design – top‑down, increasingly concrete and volatile:

business/   (business rules, no dependency on any specific frontend/backend)
  ↓
frontend/ + backend/ (tech specifications per side, depend on business/)
  ↓
common/    (API contracts derived from proto files, never depend on implementation)
  ↓
changes/   (requirement evolution, may reference all of the above)

Plus two auxiliary directories: archives/ and issues/. Dependencies are strictly one‑way downward.

Spec quality and granularity:

  • Three‑level granularity: high‑level overview → module/service spec → sub‑page / detail (field‑level request/response).
  • Two‑hop lookup, no global wildcards: Any retrieval must follow index.md → relevant spec. Glob patterns like **/*.md are prohibited.
  • Single Source of Truth (SSOT): Terms are defined only once in glossary.md; interfaces are recognised only from .proto files. No duplicate definitions.
  • Unified chapter structure: All similar documents (e.g., each service’s spec.md) share the same skeleton (overview / directory / data model / cache / API / sequence / dependencies / rules) so that the model can “read at fixed positions.”
  • In‑place incremental updates: All updates are made on the original files and reviewed via Git diff – no timestamp‑based copies.

Knowledge Base Three‑Phase Operation: Initialisation → Evolution → Governance

Phase 1: Initialisation of legacy assets – take inventory. Gather evidence from historical docs, code, and live production; run information collection, analysis, generation, and content validation. But this cannot be fully AI‑driven – legacy systems contain “outdated but still running” code, “deprecated but not deleted” interfaces, and “old fields no longer used.” AI cannot judge these; domain experts must verify and prune. Our empirical approach: AI drafts the initial version, humans remove obsolete parts, humans add critical constraints – only then do we have a usable baseline.

Phase 2: Evolutionary updates – after each iteration, P6 forces the three‑step suite: changes-sync, knowledge-sync, specs-generator. If these are not completed, the next change’s P1 cannot start – enforced by the discipline layer.

Phase 3 (continuous governance) is still in progress.

Context Injection: session‑start Hook + Two‑Hop Lookup + Dual‑Layer Token Accounting

Core question this section answers: How do we ensure that only the right pieces of context are fed to the model at the right time?

Context injection is not “the more, the better” – it’s “only what it needs to see at this step.”

We implement four engineering actions:

  1. Two‑hop lookup via index.md (no global wildcards): every Skill’s precondition explicitly forbids **/*.md globbing.
  2. Dual‑layer token accounting (parent Skill / SubAgent independent billing): a counter‑intuitive insight – SubAgents do not save context; they add a separate bill. We discovered this when we designed code‑reviewer to read full files; token consumption blew up unnoticed. Fix: rewrite to read git diff + key snippets first, and only read full files when diff is huge or context is insufficient.
  3. All SubAgents prioritise git diff: this is now a unified convention across P3’s reviewer, P4’s debugger, and P6’s sync.
  4. Keep file length moderate – avoid bloated files.

When protocol (contracts), pipeline (stages), discipline (gates), and long‑term memory (knowledge base) work together, they all do the same thing: move what AI cannot see to where it can definitely see it.

Four Engineering Principles We Learned the Hard Way

Core question this section answers: What reusable principles can we extract from our Harness journey?

The single biggest takeaway: Engineering AI coding is fundamentally about systematic management of uncertainty. The model is probabilistic, attention attenuates, context gets compressed, outputs self‑rationalise – these are not bugs; they are the “physical constants” of LLMs. Harness Engineering works precisely because we accept that these constants cannot be eliminated – we can only build a deterministic skeleton around them.

Principle 1: Pursue Determinism Over Free‑Form Creativity

Adopt Fixed Flow combined with adversarial, programmatic quality gates. Four concrete practices:

  • Persistent state: each step’s input, output, and status are written to a shared persistent file – never pass context directly between agents.
  • Programmatic gate checks: critical steps and artefacts are hard‑checked; if they fail, roll back to the previous stage – do not rely on AI’s self‑judgement.
  • Input quality constraints: standardised templates (requirements, design) enforce input quality.
  • Adversarial discipline: behavioural iron rules (TDD / Debugging / Verification) + independent evaluation + “self‑rationalisation” alarms.

Reflection: “Letting AI be creative” sounds appealing, but the engineering cost is multiplying uncertainty for downstream. Fixed Flow does not limit AI’s capability – it anchors that capability on a verifiable track.

Principle 2: Control Context Aggressively

When context grows too long, CodeBuddy compresses it – which harms SKILL effectiveness. This is the most fundamental fact in context engineering.

Practical actions:

  • Embed critical rules into rules (so they are not compressed in long sessions).
  • Use new sessions for unrelated tasks.
  • SKILLs should read files on‑demand, not scan everything.
  • Keep file lengths under control – avoid super‑long files.

Reflection: Context is not a “window” – it is a scarce resource. What determines AI performance is not window size, but the density of critical information within that window. Controlling context means controlling the signal‑to‑noise ratio.

Principle 3: Optimise Token Costs

  • Choose the right model for the task – not “the strongest is always better.”
  • Control context length.
  • Start a fresh session when not needed – don’t keep one session forever.

Reflection: Cost optimisation is not about equating “expensive” with “good.” A cheaper model + compact context + clean session often outperforms “strongest model + everything in one pot” – task‑fit is the primary factor, model size is secondary.

Principle 4: Implement Deterministic Processes as Scripts

LLMs are stochastic – although they can achieve results after many iterations, that wastes tokens and time. For deterministic, repeatable flows, bake them into scripts: scripts/run_test.js, etc.

We also prefer SKILLs over MCP. MCPs have problems: they permanently occupy context length, cannot be flexibly selected, and too many degrade model performance. Our interim approach: CodeBuddy IDE → SKILL Script → MCP Server, using progressive disclosure for authentication and invocation.

Reflection: Use AI where it excels; use scripts for the rest. Draw this boundary clearly, and AI’s value amplifies; blur it, and randomness devours you.

Four Recurring Problems and Our Solutions

Core question this section answers: Which issues keep cropping up, and how have we systematically addressed them?

These four problems are not isolated – they are different manifestations of the same underlying fact (LLMs are probabilistic) in different stages.

Problem 1: Instruction Following

  • Issue: AI skips critical steps, causing flow deviation; quality gates are not strictly enforced.
  • Causes: Context compression loses information; attention decay reduces focus on distant instructions.
  • Solutions:

    • TODO file driven: write core steps into a TODO file; AI executes and updates progress one by one.
    • Break into SubAgents: reduce per‑call context to improve instruction adherence.
    • Progressive disclosure: load context on‑demand.

Reflection: AI often doesn’t “disobey” – it simply doesn’t see the instruction. Long‑session compression and attention decay render explicit instructions inaudible later. So improving instruction following is not about repeating “behave” – it’s about putting instructions where AI can’t miss them (TODO, SubAgent entry, current frame of progressive disclosure).

Problem 2: Requirement Ambiguity

  • Issue: Natural language is inherently ambiguous; AI easily misinterprets requirements.
  • Solutions:

    • Multi‑round clarification: before execution, force AI to ask questions and get confirmation.
    • Structured requirement specification: convert all requirements to GIVEN‑WHEN‑THEN format.

Reflection: Ambiguity is not AI’s fault – it’s a physical property of natural language. Humans also misunderstand, but they use common sense to compensate; AI does not. So instead of expecting AI to “understand better,” write requirements in a format it cannot misinterpret – structured + explicit clarification, turning “understanding” into “matching.”

Problem 3: Design Restoration

  • Issue: AI’s UI restoration from Figma is mediocre – layout, assets, and styling often deviate.
  • Solutions:

    • Introduce intermediate artefacts (html + css + assets): AI is better at generating code from structured intermediate forms.
    • Multi‑round UI calibration: screenshot comparison, iterate to approach the design.

Reflection: AI is bad at going directly from “image” to “code,” but good at going from “intermediate structure” to “code.” This suggests a general method: when A → B fails, insert a middle layer A → C → B – making each segment a transformation that AI truly excels at.

Problem 4: Ensuring Output Reliability

  • Issue: LLMs are probabilistic and hallucinate – generated code varies each time.
  • Solutions:

    • Self‑validation loop: write → run → test → fix → re‑validate.
    • UTDD (Unit Test‑Driven Development): generate test cases first, then implementation – use tests to constrain code quality.
    • Review Agent gate: key artefacts are cross‑reviewed; only pass if they meet criteria.

Reflection: Reliability is not “make AI write correctly the first time” – it is “admit it won’t, but wrap it with mechanisms that catch errors.” Self‑validation, UTDD, and Review Agent all share one thing: none of them trust AI’s single‑shot output; they all rely on a “output + verification” dual track. That is the essence of Harness: given the model, engineering’s job is not to remodel the model, but to remodel its environment.

The Road Ahead: We’ve Only Just Mapped the Territory

Core question this section answers: What are the unsolved challenges in Harness Engineering, and where should we go next?

We are increasingly clear that Harness Engineering is not a “theory‑first, then implement” methodology – it emerged from the trenches of teams like Anthropic, Codex, and ours, and was later retroactively named and systematised. The value of this system is not in “how much we got right,” but in acknowledging how much we still have wrong.

At least six items remain open:

  1. The coupling between scoring and downstream real‑world impact is not yet closed. A P2 that scores 90 but causes P3 failures is not yet fed back into the scorer – it still judges only the current phase’s artefacts.
  2. Automatic knowledge‑base governance is still evolving. The three‑step suite solves how to archive, but we have no mechanism for how to age out or retire archived knowledge.
  3. The operations track alert‑closure loop is not fully complete. The main chain works in sibling projects, but cross‑project SOP reuse and knowledge‑base sharing are still being tested.
  4. Multi‑model evaluation, cross‑project knowledge transfer, and agent self‑evolution (the AHE frontier) – we are only at the doorstep.
  5. Adapting high‑complexity legacy projects – these often have “submerged icebergs” of implicit constraints. Currently, domain experts must accompany the AI through significant initialisation. Reducing this “co‑pilot cost” so that legacy projects can enter AI‑native rhythm as quickly as new ones is our most pressing challenge.
  6. AI test reliability – while left‑shift testing, E2E, API testing, and self‑healing have caught many issues, AI‑generated test cases may still suffer from insufficient coverage, weak assertions, and verifying only “runs” not “business correctness.” Next steps include test‑case quality evaluation, negative‑case generation, risk‑based coverage mapping, and a secondary review mechanism for “is the test itself trustworthy.”

Throughout this journey, we invented no new concepts – we took the engineering language that leading companies have forged at the Harness layer and layered it into our own system, with every piece backed by real pitfalls, real solutions, and real artefacts.

But that’s what makes Harness Engineering most interesting: it is not a final framework – it is a map that reality forces us to keep updating. Each time we run a real business scenario, the map advances one step further. We’ve just reached the point where the map is being drawn. There is still a vast blank ahead.

If you’re doing similar work in your own team, we welcome your feedback, your questions, and your collaboration to push this frontier together.

One‑Page Summary – Quick Reference

Dimension Key Takeaways
Harness Definition Agent = Model + Harness. Not “how to answer” but “how to work.”
Why Needed AI writes faster, but overall velocity doesn’t keep pace – essential complexity remains, bottlenecks shift, implicit constraints are invisible.
Core Architecture 2 tracks (dev delivery + online ops) + 1 long‑term memory (knowledge base).
Protocol Layer Every AI step has a fixed I/O contract – format, verifiability, traceability.
Pipeline Layer P1→P6 standard stages: requirements → design → implementation + review → test with self‑healing → deployment gates → archive for compound interest.
Discipline Layer TDD, Debug, Verify, Review, Evaluate – hard‑coded gates, not suggestions.
Knowledge Base Project‑level specs/ + change‑level directories, two‑hop lookup, no global wildcards, SSOT.
Observability Traceability (evidence), retrospectability (root cause), measurability (cost/duration/retries).
Four Principles Fixed Flow for determinism, context control, token cost optimisation, scripts for deterministic tasks.
Open Challenges Scoring–downstream coupling, automated knowledge governance, legacy‑project adoption, AI test reliability.

Practical Action Checklist

If you want to start implementing Harness Engineering in your team, here are the most critical levers:

  1. Lock the requirements entry: pull from your project‑management tool as the sole source of truth; make all AC testable; generate test cases from the same AC list.
  2. Make design a machine‑readable contract: use structured tables and diagrams; mark every “to‑be‑clarified” field explicitly.
  3. Close the feedback loop: require an independent score (≥95) before moving to the next phase; on failure, automatically trace from trace‑id to logs/database/Redis.
  4. Enforce non‑negotiable discipline: TDD before code; SQL changes require human approval; P6 archiving must complete before the next change starts.
  5. Optimise context and cost: prefer git diff over full files; treat SubAgent context as separate billing – not a free lunch.

Frequently Asked Questions

Q1: What’s the difference between Harness Engineering and Prompt Engineering?
Harness Engineering focuses on the entire working environment for an Agent to complete a multi‑step task, while Prompt Engineering focuses on single‑call input/output quality. They are not replacements – they stack.

Q2: Why doesn’t faster AI coding lead to faster overall development?
Because development is never just “writing code.” AI reduces accidental complexity but leaves essential complexity untouched. Local acceleration merely moves the bottleneck from “write” to “understand, align, verify, and consolidate.”

Q3: Why is P6 archiving so important?
Skipping archiving means the next similar requirement forces AI to start from zero; past pitfalls repeat. Archiving is not “documentation” – it’s compound interest.

Q4: Why must SQL changes require manual confirmation?
We learned this from production incidents. Database changes are high‑risk; AI auto‑executing DDL/DML can turn “rework” into “incident.” Deployment should be slow rather than wrong.

Q5: Doesn’t using SubAgents save context? Why emphasise dual‑layer token accounting?
Counter‑intuitively, SubAgents do not save context – they add a separate bill. They read files independently, and the main Agent is unaware of their token consumption. That’s why we prioritise git diff over full files.

Q6: How do we onboard legacy projects into this Harness system?
Legacy projects often have “submerged” implicit constraints – outdated docs, abandoned interfaces. Currently, domain experts must co‑pilot the initialisation: AI drafts, humans remove obsolete parts, humans add critical constraints – only then do we have a usable baseline.

Q7: How do we ensure the quality of AI‑generated tests?
While self‑healing catches many issues, AI‑generated tests may still have coverage gaps and weak assertions. Next steps include test‑case quality evaluation, negative‑case generation, risk‑based coverage mapping, and a secondary review of “is the test itself trustworthy.”

Q8: Is this system suitable for small teams?
The core ideas – lock requirements, make design a contract, enforce gates, build a knowledge base – can be scaled down. Small teams can start with the “three basics” (requirements entry, design contract, independent scoring gate) and expand incrementally.