A Practical Guide to AutoHarness: Auto-Generating Code Harnesses for LLM Agents Using Tree Search and Sandboxing

Getting large language models (LLMs) to behave is hard. In agent architectures, models routinely generate illegal actions or break the environment. We used to write manual rules to constrain them, which is tedious and full of blind spots. AutoHarness, a Rust library, takes a different approach: it uses tree search combined with Thompson sampling to automatically synthesize and optimize code harnesses for LLM agents. In testing, it hits a 100% legal action rate in just 14.5 iterations on average.

Why Do We Need Auto-Generated Code Harnesses?

A harness constrains an LLM agent with safe and legal action boundaries. Hand-coding these constraints constantly lags behind business logic changes. AutoHarness automates this process, letting a small model paired with a solid harness outperform a large model without one.

AutoHarness supports three harness modes, each covering a different constraint scenario:

  • Filter: Intercepts actions before execution. If the LLM makes an illegal move, the filter blocks it outright, preventing environment state pollution.
  • Validator: Checks the state itself. Once the environment changes, the validator inspects whether the current board or logic follows the game rules.
  • Policy Harness: Goes beyond interception and validation. It directly participates in generating and recommending actions, guiding the LLM toward a legal strategy space.

These three modes cover the full pipeline from simple interception to complex strategy guidance. I initially thought auto-generated constraint code would be too rigid and block valid actions by mistake. But after seeing it hit a 100% legal action rate across 145 TextArena games, this dynamic tree-search generation approach clearly works.

How to Get AutoHarness Running in Five Minutes

The worst part of deploying dev tools is dependency hell. AutoHarness keeps installation simple, offering both a one-click script and standard Cargo integration.

For macOS Intel (x86_64) users, just run the one-click install script in your terminal:

# One-click install (Recommended)
curl -fsSL https://raw.githubusercontent.com/gyc567/AutoHarness/main/install/install.sh | bash

# Or use jsDelivr CDN (Faster)
curl -fsSL https://cdn.jsdelivr.net/gh/gyc567/AutoHarness@main/install/install.sh | bash

# Verify
autoharness --version

If the GitHub raw link is slow, switch to the jsDelivr CDN for better download speeds. After the script runs, verify the installation with autoharness --version. By default, the binary goes to ~/.local/bin/autoharness. Make sure to add this path to your system environment variables, or your terminal will throw a “command not found” error:

export PATH="$HOME/.local/bin:$PATH"

If you prefer not to use the script, or if you are on Linux x86_64 and Windows x86_64 (which require manual compilation), you can clone the repo and install manually. macOS Apple Silicon (ARM) currently lacks a native package but runs fine using the x86_64 compatibility layer.

Manual installation commands:

git clone https://github.com/gyc567/AutoHarness.git
cd AutoHarness/install
chmod +x install.sh
./install.sh

The install script supports a few common parameters. Run ./install.sh uninstall to remove it, or ./install.sh --help if you forget the commands.

If you are working inside OpenCode or CloudCode, just copy and paste this prompt to let the system auto-design the harness architecture:

Now use the AutoHarness CLI: https://github.com/gyc567/AutoHarness to design a Harness engineering system for this project.

For Rust developers, you can also add it directly to your project’s Cargo.toml:

[dependencies]
autoharness = "0.1.0"

How to Integrate AutoHarness into a Rust Project

Integration boils down to defining your state, actions, and an evaluator, then feeding them to the synthesis engine. The engine takes your initial code snippet, mutates and searches through it, and spits out an optimized harness.

Here is a complete basic usage example. This code simulates a board game environment, defining the game state and four movement actions:

use autoharness::core::{State, Action, Harness, HarnessType};
use autoharness::engine::{CodeSynthesisEngine, SynthesisConfig, Evaluator};
use autoharness::sandbox::{SandboxExecutor, SandboxConfig};

// Define your state
#[derive(Debug, Clone, serde::Serialize)]
struct GameState {
    board: Vec<Vec<i32>>,
    score: i32,
}

impl State for GameState {
    fn to_prompt(&self) -> String {
        format!("Board: {:?}, Score: {}", self.board, self.score)
    }

    fn validate(&self) -> autoharness::core::Result<()> {
        Ok(())
    }
}

// Define your actions
#[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)]
enum GameAction {
    MoveUp,
    MoveDown,
    MoveLeft,
    MoveRight,
}

impl Action for GameAction {
    fn to_string(&self) -> String {
        format!("{:?}", self)
    }

    fn from_string(s: &str) -> autoharness::core::Result<Self> {
        match s {
            "MoveUp" => Ok(GameAction::MoveUp),
            "MoveDown" => Ok(GameAction::MoveDown),
            "MoveLeft" => Ok(GameAction::MoveLeft),
            "MoveRight" => Ok(GameAction::MoveRight),
            _ => Err(autoharness::core::HarnessError::action_parse("Unknown action")),
        }
    }
}

In this code, GameState implements the State trait. Its to_prompt method formats the current board and score into a string so the LLM understands the environment. GameAction implements the Action trait, handling serialization and deserialization. If the input string isn’t one of the four predefined actions, from_string throws a parse error.

Next comes the evaluator and the engine call. The evaluator is the judge of the system, scoring the generated harness code:

// Create custom evaluator
struct GameEvaluator;

impl Evaluator for GameEvaluator {
    fn evaluate(&self, code: &str) -> autoharness::engine::Result<f64> {
        // Evaluate harness code
        // Return a score between 0.0 and 1.0
        if code.contains("is_legal_action") {
            Ok(0.8)
        } else {
            Ok(0.2)
        }
    }
}

// Synthesize harness
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = SynthesisConfig::new()
        .with_max_iterations(20)
        .with_convergence_threshold(0.95);

    let mut engine = CodeSynthesisEngine::new(config);
    let evaluator = GameEvaluator;

    let initial_code = r#"
        def is_legal_action(state, action):
            # TODO: Implement validation logic
            return True
    "#;

    let optimized_code = engine.synthesize(initial_code, &evaluator)?;
    println!("Optimized harness:\n{}", optimized_code);

    Ok(())
}

The logic in GameEvaluator is straightforward: if the generated code contains the string is_legal_action, it scores 0.8; otherwise, it scores 0.2. The engine uses this score as a guide, leveraging tree search to adjust the code until the score hits the configured convergence threshold of 0.95, or it runs out of its 20 iterations.

How Do AutoHarness’s Tree Search and Sandbox Mechanisms Work?

AutoHarness achieves efficiency and safety through the engine module’s tree search strategy and the sandbox module’s strict isolation. The search algorithm finds the right direction, while the sandbox provides a safety net. You need both.

The architecture consists of four core modules:

  • Core module: Defines basic data models and interfaces like State, Action, and Harness.
  • Engine module: The core code synthesis engine with integrated tree search.
  • Sandbox module: Provides a safe code execution environment where all mutations are run.
  • Feedback module: Collects execution feedback from the sandbox and feeds it back to the engine to guide the next search round.

Core Interfaces and the Synthesis Engine

The Core module defines three key traits. State requires serialization, cloning, and thread-safe passing, with core methods to_prompt and validate. Action also needs serialization and thread safety, centering on the to_string and from_string conversion methods. The Harness trait is the unified interface for all constraint types, requiring implementations for harness_type, evaluate, and propose_actions.

CodeSynthesisEngine orchestrates the search process. It holds a SearchTree, SynthesisConfig, and SynthesisStats. When you call the synthesize method with initial code and an evaluator, the engine starts the tree search. After searching, get_best_code retrieves the historically optimal code node.

Thompson sampling is key to the engine’s adaptive optimization. It balances exploring unknown code spaces with exploiting known high-scoring code. This mechanism prevents brute-force exhaustion, allowing it to converge in an average of 14.5 iterations.

The Safety Baseline of Sandbox Execution

Running LLM-generated code directly on a host machine is dangerous. AutoHarness’s SandboxExecutor locks all code execution into an isolated process.

SandboxConfig provides granular resource control options:

pub struct SandboxConfig {
    pub memory_limit_mb: u64,          // Default: 256
    pub time_limit_ms: u64,            // Default: 5000
    pub max_file_descriptors: u32,     // Default: 64
    pub max_output_size: usize,        // Default: 10MB
    pub enable_network: bool,          // Default: false
    pub working_directory: Option<PathBuf>,
    pub environment_variables: HashMap<String, String>,
}

I once ran LLM-generated code without memory limits, and an infinite memory allocation loop crashed the entire test server via OOM. AutoHarness defaults to 256MB of memory and a 5-second execution time, limits file descriptors to 64, and disables network access by default. These defaults are the safety baseline. Beyond resource limits, the sandbox implements system call filtering, only allowing necessary calls, killing timed-out processes, and validating inputs before execution.

How to Tune Synthesis and Sandbox Configurations for Your Use Case

Default configs handle most cases, but if you are running complex logic or need extreme convergence precision, manual tuning is required. Configuration is split into synthesis engine and sandbox settings. The goal is balancing effectiveness against time consumption.

Synthesis Engine Parameter Tuning

The basic engine config is conservative, suitable for routine tasks:

use autoharness::engine::SynthesisConfig;

let config = SynthesisConfig::new()
    .with_max_iterations(20)
    .with_convergence_threshold(0.95)
    .with_max_depth(10);

If business rules are highly complex and the basic config fails to produce a perfect harness, switch to an advanced config:

use autoharness::engine::SynthesisConfig;

let config = SynthesisConfig::new()
    .with_max_iterations(50)
    .with_convergence_threshold(0.99)
    .with_max_depth(15)
    .with_mutations_per_node(5)
    .with_exploration_constant(2.0)
    .with_adaptive_sampling(true)
    .with_target_iterations(30)
    .with_min_improvement(0.005)
    .with_max_nodes(2000);

The advanced config pushes max iterations to 50 and the convergence threshold to 0.99. Setting max_depth to 15 lets the search tree dig deeper, ideal for complex logic with tricky edge cases. Mutations per node increase from the default 3 to 5, and the exploration constant is set to 2.0 to encourage the engine to try new things. Adaptive sampling is enabled with a target iteration count of 30. To prevent the search tree from growing indefinitely, the node limit is capped at 2000, and the minimum improvement threshold is set to 0.005. If optimization gains drop below that, the engine stops early to save compute.

Sandbox Environment Customization

Sandbox config can also be adjusted based on the expected behavior of the generated code. For instance, if the code needs to process large datasets, 256MB might not be enough:

use autoharness::sandbox::SandboxConfig;

let config = SandboxConfig::new()
    .with_memory_limit(512)
    .with_time_limit(10000)
    .with_max_file_descriptors(128)
    .with_max_output_size(20 * 1024 * 1024)  // 20MB
    .with_network(false);

Here, memory is relaxed to 512MB, execution time is extended to 10 seconds, file descriptors are doubled to 128, and max output size is allowed up to 20MB. Network access remains disabled, as harness generation doesn’t require internet connectivity.

How Does the Project Handle Autonomous Code Improvement and Testing?

Writing the code isn’t the end. AutoHarness introduces a GOAL.md pattern for autonomous code improvement, backed by strict testing and quality scoring mechanisms.

Run the test suite with standard Rust commands:

cargo test

To run specific tests for synthesis or sandbox logic, specify the test name:

cargo test test_synthesis
cargo test test_sandbox

For project maintenance, AutoHarness uses the GOAL.md pattern to drive autonomous code improvement. Run the ./scripts/score.sh script to check the current code quality score. The project currently scores a perfect 100: format checks 20, clippy static checks 20, test coverage 25, docs 15, maintainability 20, and safety 7 (out of 10). The only deduction is in safety, likely because some extreme edge cases still need polishing.

The project also maintains several key files: GOAL.md defines improvement goals, CLAUDE.md is an action guide for the Agent, template/GOAL.md provides a template, and the docs/goal-md/tutorial-cn/ directory contains a complete tutorial index and a 5-minute quick start.

Practical Summary / Operations Checklist

  • Use the one-click script for installation: curl -fsSL https://cdn.jsdelivr.net/gh/gyc567/AutoHarness@main/install/install.sh | bash.
  • Remember to add ~/.local/bin to your system PATH after installation.
  • Apple Silicon uses the x86_64 compatibility layer; Linux and Windows require manual compilation.
  • Add autoharness = "0.1.0" to your Cargo.toml for Rust project integration.
  • You must implement the State and Action traits to provide serialization for the engine.
  • When writing an Evaluator, return a score between 0.0 and 1.0 based on the generated harness code features.
  • For complex tasks, increase max_depth and mutations_per_node in SynthesisConfig rather than blindly maxing out max_iterations.
  • Ensure SandboxConfig memory and time limits match your business needs before execution; keep network access disabled unless absolutely necessary.
  • Run cargo test for testing and ./scripts/score.sh for code quality scoring.

One-Page Cheat Sheet

  • Tool Purpose: A Rust library for auto-synthesizing code harnesses for LLM agents.
  • Core Algorithm: Tree search combined with Thompson sampling; converges in 14.5 iterations on average.
  • Three Modes: Filter, Validator, Policy Harness.
  • Four Modules: Core, Engine, Sandbox, Feedback.
  • Safety Mechanisms: Sandbox isolation, resource limits, system call filtering, forced timeouts.
  • Performance Result: Small model + harness > Large model without harness.
  • Test Score: Code quality composite score 100/100.

Frequently Asked Questions (FAQ)

1. What operating systems and architectures does AutoHarness support?
Currently, macOS Intel (x86_64) has native support, and macOS Apple Silicon (ARM) uses the x86_64 compatibility layer. Linux and Windows x86_64 architectures require manual compilation from the source.

2. Where is the default installation path? How do I make the system recognize the command?
It installs to ~/.local/bin/autoharness by default. You need to add export PATH="$HOME/.local/bin:$PATH" to your shell configuration file, or run that command directly in your current terminal.

3. Is it safe to run LLM-generated code directly in the sandbox?
It is fairly safe. AutoHarness’s sandbox applies multiple isolation layers, including a default 256MB memory limit, a 5-second forced timeout, system call filtering, and disabled network access. This blocks most malicious or abnormal code.

4. How many iterations does the engine need to generate a usable harness?
According to project testing data, it achieves a 100% legal action rate in an average of 14.5 iterations across 145 TextArena games. The default max iteration count is set to 50.

5. What are the requirements for the score returned by the evaluator?
The custom Evaluator must return a floating-point number between 0.0 and 1.0. A higher score indicates better harness code quality, and the engine uses this score to guide the tree search direction.

6. What should I do if the search process takes too long?
Check your SynthesisConfig. You can lower max_depth or mutations_per_node, or raise the min_improvement threshold so the engine stops early when optimization gains diminish.

7. How does the project evaluate code quality?
The project uses the GOAL.md pattern to drive improvements. Run the ./scripts/score.sh script to view the code quality score, which covers format, clippy, tests, docs, maintainability, and safety.

8. How do I quickly start this in OpenCode/CloudCode?
Just paste this prompt into your workspace: “Now use the AutoHarness CLI: https://github.com/gyc567/AutoHarness to design a Harness engineering system for this project.” The system will handle the rest.