I Ran a 2.78 Trillion Parameter LLM on 8GB RAM with No GPU: Kimi K3 in C

The short version, upfront: This isn’t marketing fluff about “running a 2.78T model in 8GB.” I actually ran the inference engine on a standard laptop with only 8 GB of RAM, got the correct first token, and verified that it produces byte-for-byte identical outputs compared to a 224 GB server rack. The catch? Each token takes about half a minute.

This post is a hands-on log of setting it up—downloading the 1.56 TB checkpoint, packing the trunk, running the same prompts across different memory caps, and validating the outputs. No abstract theory, just the actual commands, the actual bottlenecks, and the actual trade-offs.

What This Actually Is

This is a pure C99 inference engine for Moonshot AI’s Kimi K3 (2.78 trillion parameters, Mixture of Experts). No BLAS, no PyTorch, no ONNX, no GPU—just a single ~176 KB binary compiled with GCC or Clang.

The defining feature isn’t speed; it’s memory agility. The engine doesn’t require the full model to reside in RAM.

The project README provides a concrete ladder:

Machine	RAM	Time per Token
Average Laptop	8 GB	26.5 s
High-end Laptop	32 GB	24.2 s
Desktop	64 GB	19.8 s
Heavy Workstation	128 GB+	5.6 s

Same prompt, same checkpoint, different memory caps. The outputs are byte-identical across every row. Speed changes; correctness doesn’t.

Prerequisites: The Hard Truth About Storage

Before you look at RAM, look at storage. The checkpoint is 1.56 TB. If you don’t have that much free space, stop here.

Other requirements are surprisingly modest:

OS: Linux x86_64 (uses O_DIRECT, posix_memalign, getrusage)
CPU: AVX2 + FMA (AVX-512 is not required)
Toolchain: GCC ≥ 9 or Clang ≥ 10
RAM: 8 GB and up (literally)
Storage: ~1.7 TB free (1.56 TB model + 109 GB packed trunk)
Python: 3.9+ for tooling (download, pack, analysis)

Crucial warning: Keep the packed trunk on a fast local NVMe drive. The engine’s I/O pattern is large, random O_DIRECT reads. Running this over a network volume can turn a 10s/token run into 70s/token, and none of that time is CPU-bound.

Quick Validation: Don’t Download 1.56 TB Yet

Run the entire test suite first. It loads a tiny 13-layer fixture (same tensor graph as the full model) from the repo. No network, no checkpoint download.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j
make test

The output should end with:

GATE 1  teacher forcing : 32/32 positions match tf_pred
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental     : 20/20 generated tokens match full_ids

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY
ALL WEIGHTLESS TESTS PASSED

These three gates verify teacher-forcing logits, single-pass greedy decode, and incremental decode with KV cache carried forward. If these pass, the computational graph is structurally sound. Only then should you commit to the 1.56 TB download.

Fetching the 1.56 TB Checkpoint

Get your Hugging Face token and run:

export HF_TOKEN=hf_your_token_here
./scripts/download-model.sh ~/k3model

The script supports resumption. Once finished, it runs three validations:

Shard count: 96
Total bytes: 1560936091448
Per-shard byte counts (individual verification)

Why per-shard? If shard 37 is corrupt, you only re-download that 17 GB file, not the entire 1.56 TB. The script catches the case where two shards are wrong in opposite directions but the total still matches.

If verification fails, treat it as a hard stop. Partial or corrupt weights produce plausible-sounding garbage, not subtle degradation.

Why Pack the Trunk?

The model has 93 layers. 92 of them are MoE layers. The “trunk” weights (attention, norms, routers, gates) are scattered across 96 safetensors files, interleaved with the 82,432 routed experts.

If you read directly from the shards, fetching layer L means jumping across multiple 17 GB files. That’s slow and inefficient.

The pack-trunk.sh script rewrites the 93 dense layers into a single 109 GB trunk.bin, stored contiguously by layer index. Layer L lives at a known offset. Reading it requires exactly one pread call.

./scripts/pack-trunk.sh ~/k3model ~/k3trunk

This takes about four minutes. Before copying, the packer checks that each layer’s tensors are contiguous within the original shard. If there’s a gap (which would pull expert bytes into the trunk file and bloat it to ~400 GB), the script halts with an explicit error.

Resulting directory structure:

~/k3model/      1.56 TB  96 shards + config.json + tokenizer files
~/k3trunk/       109 GB  trunk.bin + trunk.json (put this on your fastest disk)

The First Run

The minimal command for a laptop:

./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--preset laptop translates to --trunk-gb 3.0 --cache-gb 1.0. Peak RSS sits around 8.24 GB. At this preset, zero trunk layers are pinned; everything streams off disk. The expert cache is ~1 GB.

Actual output from my run:

--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS: 8.24 GB

The first token is Paris. Correct. The trailing quote and “The Eiffel” continuation aren’t a bug—this is a base model with no chat template. It’s continuing the text, not generating a dialogue reply.

--incremental matters here. Without it, the engine recomputes the entire prefix for every new token, costing O(T²). With it, step 0 pays for the prompt once; every subsequent step runs exactly one token’s forward pass, carrying KDA state and KV cache forward. Both paths are gated to produce identical outputs, so this is purely a speed toggle.

Picking a Preset

./bin/k3 --list-presets

Output:

laptop          3.0 / 1.0     8.2 GB peak RSS
desktop        16.0 / 10.0    31.9 GB peak RSS
workstation    60.0 / 30.0    95.5 GB peak RSS
server        110.0 / 13.0    ~128 GB peak RSS
max           110.0 / 109.0   ~224 GB peak RSS

The two numbers are trunk-gb and cache-gb (expert LRU cache). ./scripts/k3-doctor.sh checks MemAvailable and recommends one:

≥192 GB → server (~6 s/token)
≥96 GB → workstation (~6-20 s/token)
≥32 GB → desktop (~24 s/token)
≥10 GB → laptop (~27 s/token)

One counterintuitive note: max isn’t measurably faster than server. The extra 96 GB goes entirely to the expert cache, which—as we’ll see below—doesn’t help in steady-state decode for this architecture.

The Memory Ladder: Measured, Not Simulated

Let’s cut to the actual measurements from a 228 GB AMD EPYC workstation with 124 cores.

Total Budget	Pinned Layers	Expert Cache	s/token	GB Read per Token
8 GB	0	0.49 GB	32.69	25.83 GB
16 GB	3	4.39 GB	32.21	25.83 GB
32 GB	11	10.80 GB	31.44	25.83 GB
64 GB	27	23.59 GB	28.60	25.83 GB
96 GB	43	36.39 GB	24.40	18.11 GB
128 GB	60	49.19 GB	29.40	17.51 GB
224 GB	90	108.98 GB	19.21	14.53 GB

Outputs across every row: 17374, 20829, 10, 427, 414, 1008, 606, 142957. Byte-identical.

Look at the GB Read per Token column. From 8 GB to 64 GB, it flatlines at exactly 25.83 GB. The expert cache grows from 0.5 GB to 23.6 GB—a 48x increase—and moves zero additional bytes off disk.

The expert cache is essentially non-participating below ~64 GB total system memory. Kimi K3 uses Quantile Balancing during training to flatten expert usage across the pool. Flat usage is the worst-case scenario for LRU: there’s no “hot” subset to retain. Everything evicts immediately.

The engine’s internal hit counter can mislead here. It counts “requests satisfied from the arena”—but the batch prefetcher reads the expert off disk microseconds earlier. This gives ~100% hits at every cache size. The resident hit rate (experts already present at step start) is the metric that correlates with actual GB read. At 8 GB, that resident hit rate is 0%.

The Split Sweep: Trunk vs. Cache at Fixed Memory

Same machine, fixed 128 GB total. Vary only the split between trunk and expert cache.

Trunk	Cache	s/token
12.3 GB	110.7 GB	28.38
30.8 GB	92.2 GB	25.20
49.2 GB	73.8 GB	25.69
73.8 GB	49.2 GB	18.37
98.4 GB	24.6 GB	19.46
110.0 GB	13.0 GB	16.80

The fastest config (110 GB trunk / 13 GB cache) is 69% faster than the slowest (12.3 GB trunk / 110.7 GB cache)—at identical total memory.

The fastest config reads 25.83 GB/token with 0% cache hit rate. The slowest reads only 14.46 GB/token with 44% hit rate. The winner reads more bytes from disk and still wins.

Why? Pinning one trunk layer deterministically removes ~1.17 GB/token of guaranteed I/O. Giving that same gigabyte to the expert cache, below the knee, removes nothing measurable. The rule is blunt but effective: fill the trunk budget first, then allocate leftovers to the expert cache.

Why the Trunk Isn’t Quantized

Most inference stacks offer int8 or int4 weights to shrink memory. This engine has exactly two weight types: FP32 and BF16. No quantization knob.

The project sampled 31 attention tensors from the checkpoint and measured symmetric per-row quantization error:

Precision	Mean Relative Error
int8	~1%
int4	~17%

The worst int4 rows hit 65% relative error. And the Moonshot AI technical report explicitly states “non-expert components remain in higher precision.” The trunk was never trained for 4-bit inference. Read latency costs seconds per token; lost precision costs correctness, and no amount of RAM buys it back.

Operational Details from Real Runs

Non-ASCII prompts. The shell re-encodes argv. For CJK, emoji, or accents:

# Bad: shell mangles this
./bin/k3 ... --prompt "你好"

# Good: read raw bytes
printf '你好' > /tmp/p.txt
./bin/k3 ... --prompt-file /tmp/p.txt

Omitting --trunk. If you forget --trunk, the engine loads the full trunk resident—~113.5 GB. No preset can bring that down. This is the #1 cause of unexpected memory usage.

KV cache memory. In --incremental mode, the 24 MLA layers cache expanded k and v in FP32: ~2.37 MB per position. The engine estimates this upfront and refuses to start if it exceeds 90% of available memory:

REFUSING: the KV cache for 100000 positions needs 237.00 GB but only 128 GB available.

This check happens before allocation, so you don’t discover the limit an hour into generation.

Environment variables.

OMP_NUM_THREADS: defaults to all cores
K3_TOK_FILES: tokenizer directory
HF_TOKEN: for the download script

Measurement Noise: The Elephant in the Room

Three back-to-back runs of the exact same config (trunk 110 / cache 13):

Run 1: 14.78 s/token
Run 2: 14.67 s/token
Run 3: 20.14 s/token

Range: 33%. The same binary, same prompt, same machine, same minute. The spread is larger than most micro-optimizations.

In another pair of runs with identical configuration (same pinned layers, same cache slots, same 374.99 GB total read), one got 2,709 MB/s from the disk and the other got 5,874 MB/s. The storage device alone accounts for a 2.17x difference.

Takeaway: Only effects clearing 2x the noise floor are meaningful. The 69% trunk-first win clears it. The 11% improvement at 32 GB doesn’t. Byte counts, hit/miss counters, and resident hit rates aren’t stopwatch readings—they hold steady across runs.

Quick Reference Summary

Run make test before downloading 1.56 TB—it verifies the engine without model weights
pack-trunk.sh is mandatory; without it, trunk reads are scattered across 96 files
Every run needs --trunk; otherwise the trunk loads resident (~113.5 GB)
Non-ASCII input: use --prompt-file, never --prompt
Presets are just shorthand for --trunk-gb and --cache-gb; override them individually
Memory priority: pin trunk layers first, then allocate to expert cache
Quoted PEAK RSS is the measured value; the memory plan is a forecast

FAQ

Does this actually run on an 8 GB machine? Yes, but at 26-33 seconds per token. The test environment was 8 GB RAM, no swap, NVMe storage. HDD will be significantly slower; O_DIRECT large reads are not HDD-friendly.

Why does the run report 8.24 GB RSS when the plan said 8 GB? The plan is a budget forecast; RSS is the measured peak. The engine needs slightly more than the plan to run stably. The cgroup test with MemoryMax=8G actually required ~8.24 GB.

What happens without --incremental? The engine recomputes the full prefix for every new token: O(T²). For 8 tokens it’s bearable; for 100 it’s not. --incremental carries KDA state and KV cache forward; outputs are identical across both paths.

Why is the KV cache so large (2.37 MB/position)? The 24 MLA layers expand the latent into 96 heads × 320 floats, stored in FP32. KDA layers use a fixed 626 MB state matrix that doesn’t grow with context length. At long contexts, MLA dominates memory.

Why does my huge --cache-gb show 0% hit rate? LRU performs poorly on uniformly distributed expert access. Kimi K3’s Quantile Balancing flattens expert load intentionally. Below ~64 GB total memory, the true resident hit rate is zero.

Can I quantize the trunk to int4 to save memory? The project explicitly avoids this. Measurements show 17% mean error and 65% worst-case row error for int4 on the trunk. The trunk components were never trained for low-bit inference. You can buy back disk latency with RAM; you can’t buy back lost precision.

What’s the maximum context length? No hard-coded limit. It’s bounded by the MLA KV cache: ~2.37 MB per position. On a 128 GB machine, practical context is around 50,000 tokens before the cache alone consumes available memory. The engine checks this and refuses to start if the estimate exceeds 90% of MemAvailable.

I Ran a 2.78 Trillion Parameter Kimi K3 LLM on 8GB RAM with No GPU: Here’s How It Actually Works