Production-Grade AI Agent Architecture: The Hermes Runtime That Keeps Agents Alive

高效码农

4 weeks ago

Production-Grade AI Agent Architecture: A Deep Dive into the Hermes Runtime

Treating an AI Agent as nothing more than a ReAct loop wrapped around an LLM works fine in a demo. Ship it to production, and it’ll collapse within three days. It’s not the model’s fault—it’s the missing runtime.

The Real Question: What Is an Agent Framework Actually Solving?

Over the past two years, I’ve built my fair share of Agent applications. I started with quick LangChain prototypes—basic chatbots that barely held context. But eventually, I needed something that could run inside Feishu (Lark), read local files, call internal APIs, and act like an actual “digital employee.”

The hardest lesson wasn’t about prompt engineering. It was this: how do you keep an Agent alive and stable in the messy real world?

Consider what the real world looks like. Users send messages from Telegram, Discord, Slack, Feishu, or the CLI. Every platform has a different payload structure. The same user might fire off three rapid-fire messages. Or they might start a long-running task and then hit “stop” halfway through. If two messages from the same session trigger two separate Agent loops, you get corrupted context and choked resources at best—a full-on system stall at worst.

This is what a framework actually solves. It’s not about “making LLMs smarter.” It’s about making LLM-driven programs as reliable as a proper backend service.

What I find striking about Hermes is that it barely invents anything new. It takes state machines from frontend dev, thread pools from backend engineering, and event loops from operating systems—then plugs LLM calls into the critical junctions.

Let’s walk through the layers.

Layer 1: The Entry Point—Raw “Semi-Products,” Not Messages

A user sends a message. In Telegram, it’s an Update object. In Discord, it’s a Message. In the CLI, it’s a raw string. In Feishu, it’s a webhook JSON payload.

These inputs share one critical trait: they’re platform-specific, semi-structured, and nowhere near ready to feed into an Agent.

Telegram update
Discord message
CLI input
Webhook payload

This layer doesn’t do any heavy lifting—it just passes the raw data inward. But it highlights a fact that’s easy to overlook: the boundary of an Agent framework isn’t “receiving a message”; it’s “receiving a platform event.” If your framework only handles one input format, you’ll rewrite half your logic to switch platforms.

Layer 2: The Adapter—Washing the Dirty Data

The Adapter layer does one thing clearly: translates platform-specific inputs into a unified internal MessageEvent.

Specifically, it:

Identifies who sent it (user ID)
Routes it to the right conversation (chat/session ID)
Extracts the raw text
Checks for attachments (images, files, audio)
Generates an internal routing key: the session_key

This session_key is the most important concept in the entire architecture. Think of it as the session’s ID card. It determines where all subsequent work gets routed and defines the granularity of your concurrency control.

The Adapter also manages a crucial “gatekeeping” mechanism:

active_sessions[session_key] = busy guard
pending_messages[session_key] = next round / interrupt slot

If a session already has an active Agent loop, new messages don’t immediately trigger another loop. Instead, they enter a “pending” state. This design rests on a simple, pragmatic judgment: running two independent Agent loops simultaneously in the same session does nothing but waste tokens and create chaos.

Layer 3: The Event Bus—It Doesn’t Think, It Coordinates

This layer is probably the most misunderstood.

Some people see “Event Bus” and assume it’s a message queue. Others think it schedules Agent execution. Neither is quite right. The Event Bus has a single, narrow responsibility: coordinating the lifecycle flow of events throughout the runtime.

Its job includes:

Publishing events (publish)
Managing subscribers (subscribe)
Firing lifecycle hooks
Async dispatching
Handling interrupt signals and pending messages

A simplified lifecycle looks like this:

on_message_received → on_processing_start → on_agent_run → 
on_tool_event → on_response_sent → on_processing_complete

Here’s the key point: the Event Bus does not own the thread pool, cache the AIAgent, run the ReAct loop, or maintain remote LLM sessions. Its role is more like a PA system: “Something happened. Who needs to know? What hooks should fire? Should we trigger the typing indicator, log this, emit a metric, or run cleanup?” All these “side effects” are walled off from the core loop at this layer.

If you’re familiar with Node.js, the pattern clicks immediately:

Node.js	Hermes
EventEmitter	Event Bus
Express Middleware	Lifecycle Hooks
Event Loop	Async Scheduler
libuv Worker Pool	ThreadPoolExecutor
Route Handler	GatewayRunner + AIAgent

This isn’t a one-to-one mapping, but the philosophy is the same: the event loop handles scheduling, the thread pool handles blocking work, and business logic lives in the handlers.

Without this layer, every platform adapter would have to reimplement typing indicators, interrupt handling, progress updates, and cleanup logic. The codebase would turn into a tangled mess. The Event Bus extracts these cross-cutting concerns from the core ReAct loop.

One more nuance: order vs. concurrency.

Within a single message, stages are ordered: Received → Start → Agent Run → Response → Complete.
Across sessions, events run concurrently: Session A can be processing while Session B starts and Session C waits for a thread.
Within a single session, Hermes enforces a hard constraint: only one active run at a time. New messages during an active run go into pending/interrupt/queue states.

The core trade-off: allow cross-session concurrency, protect the single session from conflicting Agent loops.

Pseudo-code for the logic:

async def handle_raw_platform_message(raw):
    event = adapter.to_message_event(raw)
    session_key = event.session_key

    await publish("message_received", event)

    if adapter.is_session_active(session_key):
        adapter.store_pending_or_interrupt(session_key, event)
        await publish("message_pending", event)
        return

    await publish("processing_start", event)

    try:
        response = await gateway_runner.handle_message(event)
        await publish("response_sent", response)
    finally:
        await publish("processing_complete", event)

Layer 4: GatewayRunner—The Real Dispatch Master

If the Event Bus is the “coordination layer,” the GatewayRunner is the “execution layer” dispatch center.

It owns and coordinates these runtime resources:

ThreadPoolExecutor
running_agents (active run tracking)
agent_cache (AIAgent objects cached by session_key)
Session store
Queued events
Model/provider configs
Progress and interrupt monitors

How the Thread Pool Works

A crucial design decision: GatewayRunner does not create permanent worker threads per session. Instead, it submits blocking Agent work to a shared thread pool.

Session A borrows Worker 1
Session B borrows Worker 2
Session C waits until a slot frees up

Once the Agent finishes, the thread returns to the pool. Worker threads are not owned by any session.

Why this design? agent.run_conversation() might do any of these things:

Call a remote LLM API (network I/O)
Execute local tools (read/write files)
Wait for a subprocess
Stream output

All of these are blocking operations. You absolutely don’t want them clogging your async message-processing loop. So GatewayRunner essentially does this:

await loop.run_in_executor(
    shared_thread_pool,
    run_sync,
)

Inside run_sync, the final call is:

agent.run_conversation(message, conversation_history=history)

Here’s a nuance worth noting: the core AIAgent loop is synchronous and runs inside a thread. The surrounding orchestration uses async/await, but the heavy lifting happens in threads. This sacrifices some theoretical concurrency efficiency in exchange for drastically lower code complexity. For an Agent running on a user’s machine where concurrency pressure is moderate, this is a completely reasonable trade-off.

What Happens When the Thread Pool Fills Up?

If all workers are busy, new tasks queue up in the executor. That’s classic backpressure:

Free thread available → run immediately
Pool is full → wait in line

It’s important to note: this does not mean a session “owns” a thread. It just means the task is waiting for a shared resource.

Single-Session Active Run Enforcement

GatewayRunner and the Adapter work together to ensure that only one active Agent run exists per session_key at any given time.

What happens when a new message arrives for an active session? Hermes can:

Store it as a pending next-round message
Treat it as an interrupt signal
Route it as a command
Push it into an explicit FIFO queue

The default leans toward “protect the session, note that something came in, interrupt if necessary”—rather than “dump everything into a global FIFO.” This matters because many people assume Agent frameworks should work like message queues. In reality, when a user fires off multiple messages in a rapid sequence, you often want the Agent to finish its current work first and then decide how to handle what’s queued.

Pseudo-code for the gateway:

async def gateway_handle_message(event):
    session_key = event.session_key

    mark_running_or_pending(session_key)

    def run_sync():
        history = load_history(session_key)
        config = resolve_model_and_tools(event)

        agent = agent_cache.get(session_key, config)
        if agent is None:
            agent = AIAgent(config=config, session_key=session_key)
            agent_cache[session_key] = agent

        return agent.run_conversation(
            event.text,
            conversation_history=history,
        )

    result = await run_in_executor(shared_thread_pool, run_sync)

    persist_result(session_key, result)
    clear_running(session_key)
    drain_pending_if_any(session_key)

    return result.final_response

This flow makes the three-way relationship clear:

GatewayRunner handles scheduling.
ThreadPoolExecutor provides temporary execution threads.
AIAgent executes the ReAct loop.

Layer 5: AIAgent—The Local ReAct Controller

AIAgent is where the actual work happens. It’s a local runtime object that encapsulates:

Which model/provider to use
Which tools are available
How to construct messages
How to manage conversation history
How to run the ReAct loop
How to handle interruptions
How to execute tool calls
How to produce final responses

Inside the ReAct Loop

Here’s the code that shows it best:

while not done and iteration_count < max_iterations:
    assistant_message = call_llm(messages, tools=tool_schemas)

    if assistant_message.tool_calls:
        tool_results = execute_tools(assistant_message.tool_calls)
        messages.append(tool_results)
        continue

    return assistant_message.final_text

In practice:

Call the LLM with the message history and tool schemas.
The model returns a final answer → stop.
The model returns tool_calls → execute the tools → append observations → call the LLM again.

This is the ReAct pattern in code: Reason → Act → Observe → Repeat. The loop is local; the intelligence comes from the remote model. But the control structure—the “thinking”—stays on your hardware.

The Remote LLM Is Stateless

This is a critical point that trips up a lot of newcomers: the remote LLM does not remember your session.

Every time Hermes calls the model, it has to send all the necessary context:

System prompt
Conversation history
Latest user message
Tool schemas
Tool observations
Runtime instructions

From the outside, the model seems to “remember” what you talked about. In reality, Hermes is rebuilding the complete context from scratch and sending it every single time. All “state” lives in the local runtime. The remote only does a stateless inference based on the current input.

The implication: session state management is entirely local; the remote does stateless inference only. This makes horizontal scaling easier (no session stickiness headaches), but it means longer contexts burn more tokens. That’s a trade-off every Agent framework has to face.

Layer 6: Tools—The LLM’s Hands and Feet

Tools are local capabilities exposed to the model via structured schemas:

Read files
Write files
Run terminal commands
Query memory
Search context
Call external APIs
Generate media

The LLM doesn’t execute these directly. It only emits structured tool-call requests. The local runtime handles:

Validating the call
Executing the tool
Converting the result into an observation
Appending the observation back into the message list

Flow:

LLM emits tool call
  → AIAgent validates it
  → Local runtime executes it
  → Result becomes an observation
  → Observation is sent back to the LLM

A hill I’ll die on: make your tool schemas painfully precise. Models will find creative ways to fill parameters wrong. “File path” might come in as an absolute path, a relative path, or even with rogue spaces and newlines. Local execution error-handling often saves you more than fancier prompts ever will.

End-to-End: A Message’s Full Journey

Here’s how the entire pipeline works from input to output:

A user sends a message on some platform.
The platform adapter receives the raw payload.
The adapter normalizes it into a MessageEvent + session_key.
The Event Bus fires lifecycle events (message_received).
The runtime checks if this session_key is already active.
GatewayRunner prepares the session history, configuration, tools, and Agent instance.
GatewayRunner submits the blocking Agent work to the shared thread pool.
A worker thread runs agent.run_conversation().
AIAgent enters the ReAct loop.
The remote LLM is called with messages/history/tool schemas.
If the LLM returns a tool call, the local runtime executes it and appends the observation.
The loop repeats until a final response is produced or a stop condition triggers.
GatewayRunner persists the result and clears the running state.
The Event Bus fires response_sent and processing_complete.
The adapter sends the final reply back to the user.

Every step in this chain is observable and interceptable. That’s the line between “industrial-grade” and “demo-grade”: you don’t just make it work; you make it inspectable and controllable at every layer.

Ownership and Shared Resources: A Quick Reference

Component	Owned By	Shared?	Lifecycle	Responsibility
GatewayRunner	Process/Runtime	Yes, process-global	Long-lived	Main scheduler
ThreadPoolExecutor	GatewayRunner	Yes	Long-lived	Runs blocking Agent work
Worker Thread	Thread pool	Yes	Per-task	Executes a single submitted run
session_key	Runtime/Session system	No	Persistent identifier	Routes session work
active_sessions[session_key]	Adapter/Runtime	Per-session	During active run	Prevents duplicate runs
pending_messages[session_key]	Adapter/Runtime	Per-session	Until consumed	Holds next-round/interrupt data
running_agents[session_key]	GatewayRunner	Per-active session	During current run	Tracks active Agent for interrupts
agent_cache[session_key]	GatewayRunner	Per-cached session	Cross-round until invalidated	Reuses local AIAgent instances
AIAgent	Runtime-created object	Bound to cached session	Cross-round reuse	Local ReAct controller
Remote LLM API	Provider	Shared external service	Stateless per call	Generates assistant messages/tool calls
Tools	Local runtime	Shared definitions	Tool-dependent	Executes operations

Some Real-World Judgments

You might look at this architecture and think it’s comprehensive. Here’s my take—based on real scars.

Where does this architecture shine? Long-running, multi-platform, stability-critical “digital employee” applications. The complexity here doesn’t go into “making a single inference smarter.” It goes into “keeping the Agent from making a mess in a production environment.” If that’s your use case, this is a blueprint to copy.

Is it over-engineered? If you’re building a single-turn chatbot, absolutely. But if you have users who might interrupt a long-running task mid-flight, if you need to sync with multiple IM platforms, if a single session could run for minutes—this complexity isn’t decorative; it’s survival.

Thread pool vs. pure async coroutines? In an ideal Python world, pure coroutines could handle higher concurrency. But Hermes uses a thread pool for a practical reason: the existing LLM SDKs and tool libraries are mostly synchronous. Wrapping them in coroutines often introduces subtle, hard-to-debug issues. Using a thread pool for blocking work trades a bit of efficiency for broad compatibility and easier reasoning. This is an engineering trade-off, not a technical “best” solution.

Is a single pending slot enough? The default only keeps one pending message—not a FIFO queue. If a user fires off three messages while the Agent is busy, only the latest one survives. That’s a feature in some scenarios (the user’s last instruction is the most relevant) and a bug in others (strict task ordering). Whether you need a full queue depends entirely on your use case, and the framework won’t guess that for you.

Actionable Takeaways

A few conclusions you can apply immediately:

Agent = Harness + Model. Don’t obsess over the model alone. The harness is what determines whether your Agent survives production.
The session_key is your north star. All routing, caching, and concurrency controls hinge on this key.
Walled-off side effects. Push logging, metrics, interrupts, and progress updates out of the core ReAct loop via an event bus. Don’t let them pollute your reasoning logic.
Thread pools for blocking work. The async/sync hybrid approach is more stable and compatible than a pure async rewrite of everything.
State is local; the remote is stateless. Rebuilding context is an architectural choice, not a flaw. Embrace it and design around the token costs.
Enforce single active runs per session. This prevents a tidal wave of concurrency bugs. The trade-off is you have to design a coherent pending-message strategy.

One-Page Visual Summary

Entry (Feishu / TG / CLI)
  ↓
Adapter Layer: Normalize to MessageEvent + session_key
  ↓
Event Bus: Lifecycle coordination, hooks, interrupts
  ↓
GatewayRunner: Scheduling, thread pool, Agent cache
  ↓
AIAgent: ReAct loop (Reason → Tool Call → Observe → Repeat)
  ↓
Remote LLM: Stateless inference, no session memory
  ↓
Local Tools: Read files, run commands, call APIs
  ↓
Response sent back to user

FAQ

Q: What’s the core difference between Hermes and LangGraph?
A: LangGraph leans toward graph-based state machine orchestration. Hermes leans toward runtime scheduling. One is about “how to organize logic”; the other is about “how to keep an instance running stably.”

Q: Will two concurrent messages in the same session break things?
A: Not necessarily, but Hermes prevents two runs by default. If you need FIFO semantics, you can implement it on top of the pending_messages slot.

Q: Why not use pure async coroutines instead of a thread pool?
A: Existing tool libraries are mostly synchronous. Forcing them into coroutines introduces subtle bugs. The thread-pool approach is more reliable and easier to debug.

Q: What’s cached inside agent_cache?
A: Fully initialized AIAgent instances—model config, tool registrations, and session histories. Reusing them avoids expensive re-initialization.

Q: How long-running can tasks be in Hermes?
A: It depends on your thread-pool size. For multi-minute tasks, configure the pool limits carefully so one long task doesn’t starve other sessions.

Q: Does a single pending slot mean I’ll lose messages?
A: By default, if multiple messages arrive during an active run, only the latest one is preserved. You’ll need to customize this for strict FIFO requirements.

Q: Can this architecture scale horizontally?
A: You’ll need to externalize the session store and agent_cache to a shared Redis or similar. The in-memory single-node version is clean; scaling requires moving those components out.