Why gateway.run Keeps Restarting Automatically: Understanding Hermes Process Replacement, CPU Usage, and Practical Troubleshooting

If you are running a Hermes-based Agent or Gateway service, you may eventually encounter a confusing problem:

“

You only started the program once, but the logs suggest it keeps restarting.

Or perhaps:

“

Why did CPU usage suddenly spike?
What exactly does gateway run --replace do?
Is the service crashing?

This article walks through a real diagnostic log and explains what is actually happening when Hermes appears to restart repeatedly. The goal is simple: make the behavior understandable and provide a practical troubleshooting path without unnecessary complexity.

The analysis below is based entirely on the log and technical discussion already provided, without introducing external assumptions.

The Problem: “Why Does Hermes Keep Restarting?”

Consider this log output:

WARNING gateway.run: Shutdown diagnostic — other hermes processes running:
  dapen       1570 92.6  2.0  94260 83524 ?        Ss   20:12   0:01 /home/dapen/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run --replace

Many users immediately interpret this as:

“

The application crashed and is automatically restarting.

However, that is often not what the log means.

A more accurate interpretation is:

“

Hermes detected another running Gateway process and triggered a replacement behavior.

In other words:

The issue is usually not “automatic restart.”
The real issue is often:

“

Multiple Hermes processes competing with each other, or a supervisor repeatedly relaunching the process.

Understanding this distinction makes troubleshooting much easier.

What the Log Is Actually Telling You

The most important part of the command is this:

gateway run --replace

This flag matters.

The behavior implied by the log is typically:

  1. A Gateway process starts.
  2. Hermes detects an existing process.
  3. The new process attempts to replace the old one.
  4. The old process exits.
  5. Another mechanism relaunches the previous process.
  6. The cycle repeats.

So the real technical question becomes:

“

Why are processes repeatedly replacing one another?

Rather than:

“

Why is Hermes restarting?

Breaking Down the Log Line

Let’s inspect this portion:

dapen 1570 92.6 2.0 94260 83524 ? Ss 20:12 0:01

This follows the familiar ps aux output structure.

The columns are:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

Here is what each value means:

Field Value Meaning
USER dapen User account
PID 1570 Process ID
%CPU 92.6 CPU utilization
%MEM 2.0 Memory utilization
VSZ 94260 Virtual memory
RSS 83524 Resident memory
TTY ? No terminal attached
STAT Ss Process status
START 20:12 Start time
TIME 0:01 Accumulated CPU time
COMMAND python … Executed command

A common question is:

“

How do you know CPU usage is 92.6%?

Because:

92.6

appears in the %CPU column.

That means the process was consuming approximately:

“

92.6% CPU

at the time of inspection.

Why High CPU Usage Matters

Two fields are especially revealing:

START 20:12
TIME 0:01

This tells us:

  • ❀
    The process started recently.
  • ❀
    Total CPU time is very short.
  • ❀
    Yet CPU usage is extremely high.

This pattern often signals one of several situations.

Scenario 1: Rapid Restart Loop

The process repeatedly cycles through:

Start → Exit → Restart → Exit

This can consume CPU aggressively because the system is continuously spawning and terminating processes.

Scenario 2: Busy Loop

The application may be repeatedly checking:

Is another process running?
Should replacement happen?
Should I terminate?

This creates high-frequency internal activity rather than productive work.

Scenario 3: Watchdog Relaunch Cycle

Another service may continuously relaunch Hermes:

Process exits
↓
Supervisor detects exit
↓
Supervisor restarts process
↓
Replacement occurs again
↓
Loop repeats

This often results in elevated CPU usage.

The Most Common Reasons Hermes Appears to Restart

Let’s examine the major causes one by one.

Cause 1: A Supervisor Keeps Restarting Hermes

This is one of the most overlooked explanations.

Many users assume:

“

I started Hermes manually.

But in reality, another background service may already be managing it.

Examples include:

  • ❀
    systemd
  • ❀
    supervisord
  • ❀
    pm2
  • ❀
    Docker restart policy
  • ❀
    shell restart loops

These systems monitor processes and automatically restart them after exit.

When combined with:

gateway run --replace

the behavior can become cyclical.

The sequence may look like this:

New process starts
↓
Detects existing Hermes process
↓
Uses replace behavior
↓
Old process exits
↓
Supervisor interprets exit as failure
↓
Supervisor relaunches process
↓
Loop repeats

To the user, it looks like:

“

Hermes keeps restarting forever.

How to Check for systemd

Run:

systemctl --user status hermes

or:

systemctl status hermes

You can also inspect running processes:

ps aux | grep hermes

If background management exists, it may explain the repeated relaunch behavior.

Pay Attention to Restart=always

A particularly important configuration is:

Restart=always

This setting means:

“

Restart the process whenever it exits.

Combined with --replace, it can unintentionally create an infinite replacement loop.

Cause 2: Multiple Gateway Instances Are Running

This happens more often than people realize.

For example:

First terminal:

gateway run

Later, another terminal:

gateway run --replace

Now two competing instances exist.

The second instance detects the first:

Existing process found
↓
Replacement triggered
↓
Original process terminated

If a supervisor relaunches the original process, the conflict continues.

The result:

“

Multiple Gateway processes repeatedly replacing one another.

How to Confirm Multiple Instances

Run:

pgrep -af hermes

If you see entries such as:

gateway run
gateway run --replace

then multiple Hermes processes are likely involved.

Cause 3: Stale PID Files or Socket Files

Many CLI-based systems create runtime files such as:

  • ❀
    PID files
  • ❀
    lock files
  • ❀
    socket files

These help the application determine:

“

Is another instance already running?

However, after an abnormal exit, these files may remain behind.

Hermes may incorrectly assume:

“

Another process still exists.

As a result, replacement behavior may trigger unnecessarily.

How to Inspect Hermes Runtime Files

Run:

ls -la ~/.hermes

Pay attention to items resembling:

.pid
.lock
gateway.sock

These may indicate stale state.

How to Clean Up

First stop Hermes:

pkill -f hermes

Then remove temporary artifacts:

rm -rf ~/.hermes/tmp/*
rm -rf ~/.hermes/*.lock

Finally, restart Hermes.

Cause 4: Docker Restart Policies

If Hermes runs inside a container, restart behavior may come from Docker.

A configuration like:

"RestartPolicy": {
  "Name": "always"
}

can produce a loop:

replace terminates process
↓
container exits
↓
Docker relaunches container
↓
cycle repeats

This again appears as:

“

Continuous restarting.

Cause 5: Internal Watchdog Behavior

Some Agent frameworks implement their own monitoring systems.

A possible structure:

Parent process
↓
Monitors child process
↓
Child replaces parent
↓
Parent relaunches child
↓
Loop forms

This effectively creates:

“

A watchdog loop.

How to Inspect Process Hierarchy

Run:

pstree -ap | grep hermes

You may see something like:

hermes-agent
 └─python
    └─python -m hermes_cli.main gateway run --replace

If multiple nested Python layers appear:

python
 └─python
     └─python

that may indicate recursive process management.

Why CPU Reaches 92.6%

Another common concern:

“

Is 92.6% CPU usage a bug?

Not necessarily.

Look again:

START 20:12
TIME 0:01

This suggests:

“

The process started recently but consumed significant CPU immediately.

That pattern commonly aligns with:

Restart loops

Repeated launch cycles.

Fork loops

Repeated process creation.

Watchdog loops

Processes repeatedly relaunching one another.

What Happens If You Ignore It?

If restart loops continue unchecked, several problems may emerge.

Log Explosion

Repeated messages such as:

other hermes processes running

can rapidly flood logs.

File Descriptor Exhaustion

Repeated process creation may gradually consume system resources.

Socket Conflicts

New instances may fail to bind resources properly.

Sustained High CPU Usage

Your machine may become noticeably slower.

Memory Growth

Especially if child processes are not cleaned up correctly.

Recommended Troubleshooting Order

If Hermes appears stuck in a restart cycle, use this sequence.

Step 1: Check for Multiple Processes

Run:

pgrep -af hermes

Look for:

gateway run
gateway run --replace

simultaneously.

Step 2: Inspect the Process Tree

Run:

pstree -ap | grep hermes

Check for:

  • ❀
    nested Python processes
  • ❀
    watchdog loops
  • ❀
    recursive launching behavior

Step 3: Inspect systemd

Run:

systemctl --user | grep hermes

Determine whether a background service manages Hermes.

Step 4: Inspect the .hermes Directory

Run:

ls -la ~/.hermes

Look for stale:

  • ❀
    PID files
  • ❀
    lock files
  • ❀
    sockets

A Simple Recovery Method

If the root cause is unclear, a clean reset is often effective.

Terminate Hermes:

pkill -9 -f hermes

Verify no process remains:

pgrep -af hermes

No output means cleanup succeeded.

Then start Hermes only once:

gateway run

Importantly:

“

Do not use --replace initially.

If stability returns, the problem was likely:

“

Multiple instances competing for control.

A Common Misunderstanding About --replace

Many users assume:

“

--replace is the safer startup option.

However, based on observed behavior, its purpose is closer to:

“

Replacing an already-running instance.

If no prior process exists, using:

gateway run

often simplifies troubleshooting.

Meanwhile:

gateway run --replace

is most appropriate when:

“

You intentionally want to replace an existing instance.

Otherwise, diagnosis becomes harder.

Frequently Asked Questions

Why does Hermes keep restarting if I only launched it once?

Because another Hermes instance may already be running in the background.

The new instance detects it and triggers replacement behavior.

How did you determine CPU usage was 92.6%?

The value appears in the %CPU column of the ps aux output.

What exactly does gateway run --replace do?

It detects an existing Gateway instance and replaces it.

It is not simply a startup command.

How can I verify whether multiple Hermes instances exist?

Run:

pgrep -af hermes

Look for multiple Gateway commands.

How can I monitor CPU usage in real time?

Use:

top -p 1570

or:

htop

for live monitoring.

What is the fastest recovery method?

Stop everything:

pkill -9 -f hermes

Confirm cleanup:

pgrep -af hermes

Then launch only:

gateway run

without:

--replace

initially.

Quick Troubleshooting Checklist

1. Check for multiple processes

pgrep -af hermes

2. Inspect process hierarchy

pstree -ap | grep hermes

3. Check background services

systemctl --user | grep hermes

4. Inspect Hermes files

ls -la ~/.hermes

5. Clean up processes

pkill -9 -f hermes

6. Restart once

gateway run

Avoid:

gateway run --replace

until the system is stable.