How to Troubleshoot a Full 20GB Linux Disk and Hermes Authentication Errors on Tencent Cloud
Running Hermes on a Tencent Cloud server with a 20GB system disk can lead to several problems appearing one after another. In this case, the root partition first reached 100% usage, Hermes later stalled while checking a TTS dependency, the Gateway restart command reported a systemd user-session problem, and the mobile client eventually showed a Provider authentication failure.
The useful part of this incident is that each stage produced concrete command output. That makes it possible to follow the troubleshooting process from disk usage analysis to Hermes runtime and model-provider authentication.
Why did the 20GB system disk become full?
The first step was to check the filesystem usage.
df -h
The server initially reported:
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 20G 28K 100% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 3.8G 24K 3.8G 1% /dev/shm
tmpfs 1.6G 163M 1.4G 11% /run
The important number was the remaining space. The root filesystem had only 28KB available.
At this point, the right move was to identify which top-level directories were consuming the disk instead of deleting files at random.
du -sh /* 2>/dev/null | sort -hr
The result was concentrated in a few directories.
9.3G /usr
9.2G /root
980M /var
475M /opt
163M /run
104M /boot
25M /etc
940K /tmp
56K /home
The largest consumers were /usr and /root. Together they accounted for more than 18GB of the 20GB system disk.
That immediately narrowed the investigation.
What was using 9.2GB under /root?
The next step was to break /root into its major subdirectories.
du -xh /root --max-depth=1 2>/dev/null | sort -hr
The output showed:
9.2G /root
3.7G /root/.hermes
2.3G /root/.cache
2.0G /root/go
808M /root/.npm
387M /root/.rustup
127M /root/.local
20M /root/.cargo
Several directories immediately stood out.
/root/.hermes used 3.7GB.
/root/.cache used 2.3GB.
/root/go used 2.0GB.
/root/.npm used another 808MB.
This was enough to explain a large part of the disk pressure, but each directory still needed to be examined before anything was removed.
Can the 2.3GB under /root/.cache be removed?
The cache directory was inspected first.
du -xh /root/.cache --max-depth=2 2>/dev/null | sort -hr | head -30
The result was:
2.3G /root/.cache
1.5G /root/.cache/go-build
736M /root/.cache/uv
637M /root/.cache/uv/archive-v0
127M /root/.cache/go-build/54
98M /root/.cache/uv/simple-v21
96M /root/.cache/pip/http-v2
96M /root/.cache/pip
The two largest components were the Go build cache and the uv cache.
/root/.cache/go-build 1.5G
/root/.cache/uv 736M
/root/.cache/pip 96M
These are cache directories, so they were the safest targets for immediate cleanup.
The cleanup command was:
rm -rf /root/.cache/go-build
rm -rf /root/.cache/uv
rm -rf /root/.cache/pip
There is an important distinction here. These directories are different from the Python virtual environment used by Hermes.
The cache was stored under:
/root/.cache
Hermes dependencies were installed under:
/root/.hermes/hermes-agent/venv
Removing the former does not automatically remove packages from the latter.
Why did /root/go use 2GB even though the go command was unavailable?
The next investigation focused on /root/go.
du -xh /root/go --max-depth=2 2>/dev/null | sort -hr | head -30
The result was:
2.0G /root/go
1.8G /root/go/pkg/mod
1.8G /root/go/pkg
145M /root/go/bin
4.0K /root/go/pkg/sumdb
Most of the usage was under:
/root/go/pkg/mod
which occupied 1.8GB.
At first, it was tempting to use the normal Go cache-cleaning command.
go clean -modcache
The server responded:
bash: go: command not found
This exposed an important detail. The presence of /root/go did not mean that the Go executable itself was installed or available in the current shell.
For a disk-recovery operation, there was no need to install Go simply to remove a large module cache. The important distinction was between the module cache and the smaller /root/go/bin directory.
The large cache was the obvious cleanup target, while /root/go/bin was left alone because it contained actual binaries and was only 145MB.
Which parts of the 3.7GB Hermes directory should be handled carefully?
Hermes itself occupied 3.7GB.
The directory was expanded with:
du -xh /root/.hermes --max-depth=2 2>/dev/null | sort -hr | head -50
The output was:
3.7G /root/.hermes
1.9G /root/.hermes/hermes-agent
926M /root/.hermes/hermes-agent/venv
577M /root/.hermes/node
530M /root/.hermes/hermes-agent/.git
406M /root/.hermes/state-snapshots/20260811-045212-pre-update
406M /root/.hermes/state-snapshots
327M /root/.hermes/sessions
269M /root/.hermes/hermes-agent/web
119M /root/.hermes/node/bin
65M /root/.hermes/skills
65M /root/.hermes/profiles/stocker
65M /root/.hermes/profiles
63M /root/.hermes/node/include
Several large directories looked tempting to delete, but their role mattered more than their size.
The Hermes Python virtual environment occupied 926MB.
/root/.hermes/hermes-agent/venv
The Hermes Node runtime occupied 577MB.
/root/.hermes/node
Those directories are part of the Hermes runtime environment, so deleting them simply because they are large could damage the installation.
The Git repository under Hermes occupied another 530MB.
/root/.hermes/hermes-agent/.git
That is also different from a cache directory. If Git-based updates are still needed, the repository metadata should be preserved.
The most obvious cleanup candidate was the old state snapshot.
/root/.hermes/state-snapshots/20260811-045212-pre-update
It occupied 406MB and its name indicated that it was created before an update.
The sessions directory was another 327MB target.
/root/.hermes/sessions
Its contents should be inspected before removing anything because session data is different from a disposable cache.
The general rule from this incident was simple. Start with caches and clearly identifiable old snapshots. Leave the Hermes runtime directories alone until their role is understood.
Why did Hermes later stall while starting?
After cleanup, the root filesystem was checked again.
df -h
The server had moved from 100% usage to 94%.
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 19G 1.4G 94% /
The disk was still relatively full, but the immediate 100% capacity condition had been relieved.
The next problem appeared when Hermes was launched.
hermes
The process did not proceed into normal interaction. After Ctrl+C, the traceback showed a dependency check related to TTS.
The important part of the call chain was:
check_tts_requirements()
_import_edge_tts()
_lazy_ensure("tts.edge")
_venv_pip_install(missing)
This showed that Hermes was checking its TTS requirements and then entering its lazy dependency mechanism.
Hermes attempted to install a missing dependency, and the process waited inside the subprocess call until it was interrupted.
The next question was whether the edge-tts package had actually disappeared.
Was edge-tts really missing?
The installed package was checked directly.
/root/.hermes/hermes-agent/venv/bin/python -m pip show edge-tts
The result was:
Name: edge-tts
Version: 7.2.8
Summary: Microsoft Edge's TTS
Location: /root/.hermes/hermes-agent/venv/lib64/python3.11/site-packages
Requires: aiohttp, certifi, tabulate, typing-extensions
Required-by: hermes-agent
So edge-tts was present.
It was installed inside the Hermes virtual environment:
/root/.hermes/hermes-agent/venv/
The Python package manager was also present.
/root/.hermes/hermes-agent/venv/bin/python -m pip --version
The server reported:
pip 23.3.1 from /root/.hermes/hermes-agent/venv/lib64/python3.11/site-packages/pip (python 3.11)
This made one explanation less likely. The earlier removal of pip and uv caches had not removed the installed edge-tts package.
Was PyPI connectivity the reason Hermes waited?
The server’s ability to reach PyPI was then tested.
curl -I --max-time 10 https://pypi.org
The server returned:
HTTP/2 200
This confirmed that PyPI was reachable from the server.
At this point, the available evidence showed three things.
The edge-tts package existed.
The Hermes virtual environment had its own working pip installation.
The server could reach PyPI.
That left Hermes’s dependency detection and runtime environment as the next things to inspect.
The important point here was to avoid assuming that a pip install attempt automatically meant the package was absent.
Why did hermes gateway restart show a systemd error?
A separate issue appeared while trying to restart the Gateway.
hermes gateway restart
The command returned:
✗ User systemd not reachable:
Linger was enabled, but the user D-Bus socket did not appear.
systemctl --user cannot reach the user D-Bus session in this shell.
This message describes a user-level systemd and D-Bus session problem.
It is a different layer from the TTS dependency check.
The command was trying to reach a user systemd session through systemctl --user, but that user session was not available in the current shell.
The suggested alternatives from Hermes were to log out and log back in, reboot, or run the Gateway directly with:
hermes gateway run
The foreground command has an obvious operational limitation. The Gateway stays attached to the current terminal and remains active until the terminal is closed or the process is stopped.
There was another important observation later in the incident. The mobile client was already receiving Hermes responses. That meant the Gateway was in fact processing requests, so the current failure had moved beyond basic Gateway connectivity.
What does “Provider authentication failed” mean in the mobile client?
The mobile screenshot showed:
Provider authentication failed.
Check the configured credentials; raw provider details are in the gateway logs.
A second message showed:
Primary model failed — switching to fallback:
minimax-m2.5-highspeed via custom
A third message reported:
Authentication failed and could not be refreshed
These messages describe a model-provider problem.
The request reached Hermes.
Hermes attempted to call the primary model.
The provider authentication failed.
Hermes switched to the configured fallback model.
The fallback also failed authentication.
The visible message therefore points to the model-provider credential path rather than a mobile connection problem.
What should be checked for the custom provider?
The most useful phrase in the mobile message was:
minimax-m2.5-highspeed via custom
This indicates that the fallback model was being accessed through a custom provider configuration.
At this stage, the investigation should focus on the provider configuration rather than the model runtime itself.
The relevant configuration areas are the provider name, model name, base URL, API key, and authentication method.
The exact reason for the authentication failure still needs to come from the Gateway logs.
A direct search across the entire Hermes directory is too broad because the installation contains a full Node runtime and its headers. A search for words such as custom, provider, and api_key can therefore return unrelated matches from Node.js source files.
A more targeted search is better.
First, locate configuration files while excluding the Node runtime and Python virtual environment.
find /root/.hermes -maxdepth 3 -type f \
\( -name "*.json" -o -name "*.yaml" -o -name "*.yml" -o -name "*.toml" -o -name "*.env" -o -name "*.conf" \) \
-not -path "*/node/*" \
-not -path "*/venv/*" \
| sort
Then search only the Hermes logs for authentication-related errors.
grep -RniE "401|403|unauthorized|authentication failed|invalid.*key|invalid.*token|refresh|credential|minimax" \
/root/.hermes/logs 2>/dev/null | tail -100
This is much more useful than searching every file under /root/.hermes.
Where does the troubleshooting stand now?
The incident moved through several distinct stages.
The root filesystem originally had only 28KB available.
The largest disk consumers were /usr and /root.
Inside /root, Hermes, cache data, Go module data, and npm data accounted for most of the usage.
The removable cache directories were identified and cleaned.
The root filesystem recovered to 1.4GB of available space.
Hermes’s edge-tts package was verified inside its own virtual environment.
PyPI connectivity was verified with an HTTP 200 response.
The Gateway restart command reported a user-level systemd and D-Bus session problem.
The mobile client was still able to reach Hermes and received a model-provider authentication error.
The remaining investigation therefore centers on the provider credentials and the Gateway logs that contain the raw provider error.
The next command worth running is the targeted log search.
grep -RniE "401|403|unauthorized|authentication failed|invalid.*key|invalid.*token|refresh|credential|minimax" \
/root/.hermes/logs 2>/dev/null | tail -100
That output is the piece most likely to distinguish between an invalid credential, an expired authentication token, a failed refresh operation, or another provider configuration issue.
