MiniCPM4: Run Powerful Language Models on Your Phone or Laptop

Achieve 128K context processing with 78% less training data using 0.5B/8B parameter models optimized for edge devices

Why We Need On-Device Language Models

While cloud-based AI models like ChatGPT dominate the landscape, edge devices (smartphones, laptops, IoT systems) have remained largely excluded due to computational constraints. Traditional large language models face three fundamental barriers:

1. Compute Overload: Processing 128K context requires calculating all token relationships
2. Memory Constraints: Loading an 8B parameter model demands ~32GB RAM
3. Training Costs: Standard models require 36 trillion training tokens

MiniCPM Team’s breakthrough solution, MiniCPM4, shatters these limitations through four-dimensional innovation:

• 📌 Model Architecture: Trainable sparse attention (InfLLM v2)
• 📌 Training Data: UltraClean filtering + UltraChat v2 instruction dataset
• 📌 Training Algorithms: ModelTunnel v2 hyperparameter search + BitCPM quantization
• 📌 Inference Systems: CPM.cu engine + ArkInfer cross-platform deployment

1. Core Technology Breakdown

1.1 InfLLM v2: Dynamic Sparse Attention

The Challenge: Standard attention mechanisms face quadratic computation growth. Processing 128K tokens requires >16 billion relationship calculations.

Our Solution:

graph LR
A[Input Text] --> B[Split into 128-token blocks]
B --> C[Construct Semantic Kernels] 
C --> D[Dynamically Select Top-K Relevant Blocks]
D --> E[Compute Attention Only on Critical Blocks]

• Semantic Kernels: Overlapping text segments (32 tokens/kernel, stride 16)
• Dynamic Block Selection: Each query token processes only Top-6 relevant blocks
• Local Preservation: Mandatory inclusion of initial tokens + sliding window context

Performance Benchmarks:

On Jetson AGX Orin hardware, outperforms Qwen3-8B by 7× with 80% memory reduction.

🔍 Technical Insight: Kernel optimization avoids token-level computation using uint64 bitmask compression for tree-based decoding.

1.2 UltraClean: High-Quality Training Data Pipeline

The Challenge: Web-crawled datasets contain ads, templated text, and low-value content requiring trillions of tokens for baseline performance.

Our Method:

def UltraClean_workflow():
    # Efficient Validation: Fine-tune near-trained models
    candidate_data = load_corpus('FineWeb') 
    model = load_pretrained('MiniCPM-1.2B')
    perf_gain = evaluate_finetune(model, candidate_data)
    
    # Classifier Training: High-gain data as positive samples
    high_quality_seeds = filter_by_gain(perf_gain, threshold=0.3)
    classifier = train_fastText_model(high_quality_seeds)
    
    # Iterative Optimization: Dynamic sample balancing
    while not stabilized:
        filtered_data = classifier.apply(raw_data)
        perf_gain = validate(filtered_data)
        adjust_classifier(perf_gain)
    return UltraFineWeb

Dataset Comparison:

Achieves superior performance with 78% less data than conventional approaches.

1.3 ModelTunnel v2: Hyperparameter Search Revolution

Innovations:

• ScalingBench Metric: GPT-4o generated reasoning chains replace loss metrics
• μP Architecture: Parameterization scaling enables cross-model hyperparameter transfer
• Multi-Token Prediction: Simultaneous prediction of 4 future tokens

graph TB
S[Small Model Experiments] --> A[Search Learning Rate/Batch Size]
A --> B[Establish ScalingBench Performance Mapping]
B --> C[Transfer to 8B Model]
C --> D[FP8 Mixed-Precision Training]
D --> E[Multi-Token Supervision]

Experimental result: Reduces hyperparameter search cost from 4,600 GPU hours to 110 hours with <0.5% error margin.

1.4 BitCPM4: Ternary Quantized LLMs

Edge Deployment Challenge: 16GB storage needed for FP16 8B models.

Our Approach:

• Two-Stage Quantization:

  1. Full-precision training → 2. 350B token QAT fine-tuning
• Weight Ternary Encoding: Parameters constrained to {-1, 0, +1}
• Full-Precision Activations: Prevents error accumulation

Quantization Results:

💡 Key Technique: Learning rate rewarmup (5e-3) with 128×128 block quantization.

2. Inference Deployment Guide

2.1 CPM.cu: Edge-Optimized Inference Engine

Core Architecture:

class CPM_cu:
    def __init__(self):
        self.fr_spec = FR_Spec(top_vocab=25%)  # High-frequency token pruning
        self.p_gptq = PrefixAwareGPTQ(ignore_first=4)  # Skip initial outliers
        self.sparse_attn = InfLLMv2_Kernel(block_size=128)

    def execute(self, input):
        # Context-aware quantization
        quantized_weights = self.p_gptq.compress(model)
        
        # Frequency-based drafting
        draft_sequence = self.fr_spec.generate(input)
        
        # Sparse attention verification
        output = self.sparse_attn.verify(draft_sequence)
        return output

Speed Comparison (RTX 4090):

2.2 ArkInfer: Universal Deployment System

Cross-Platform Design:

graph LR
A[Unified Tensor API] --> B[Abstract Executor]
B --> C[Adaptation Layer]
C --> D[NeuroPilot-MTK]
C --> E[Genie-Qualcomm]
C --> F[RK-LLM-Rockchip]
C --> G[llama.cpp-CPU]

Key Features:

• Toolchain Validation: Auto-verify tool functionality (Claude-4 generated test cases)
• Constrained Decoding: Enforce JSON/SQL output formats
• Speculative Execution: BiTA algorithm integration

Real-world performance: Snapdragon 8 Gen3 smartphones process 128K context in <8 seconds.

3. Real-World Applications

3.1 MiniCPM4-Survey: Automated Research Assistant

Workflow:

User Query → Outline Planning → Knowledge Retrieval → Multi-pass Writing → RL Optimization

Training Methodology:

1. Supervised Fine-Tuning: 61,684 academic samples
2. Chapter-Level RL: 12-dimensional reward (structure, factuality, novelty)
3. Document-Level RL: Parallel environment interactions

Performance (SurveyEval Benchmark):

3.2 MiniCPM4-MCP: Tool Integration Protocol

Protocol Advantages:

• Standardized Schema: Unified tool description format
• Dynamic Discovery: Docker-containerized tool ecosystem
• Cross-Tool Chaining: Single query triggers multi-tool workflows

Accuracy Comparison:

4. Comprehensive Performance Evaluation

4.1 Benchmark Comparisons

📌 Key Insight: The 8B version outperforms Phi4-14B by 13 points on Chinese reasoning (CMMLU) with 4× less training data.

4.2 Long-Context Processing

Needle-in-Haystack Test (128K context):

• Accuracy: 100% (vs. 99.2% for full-attention baselines)
• Attention Sparsity: Only 5% context blocks activated
• Positional Encoding: 32K training → 128K extrapolation (YaRN technique)

5. Developer Resources

5.1 Open-Source Assets

# Model Access
- HuggingFace Repository:
  - [MiniCPM4-8B](https://huggingface.co/openbmb/MiniCPM4-8B)
  - [MiniCPM4-0.5B](https://huggingface.co/openbmb/MiniCPM4-0.5B)

# Codebase
- GitHub Project:  
  [https://github.com/openbmb/minicpm](https://github.com/openbmb/minicpm)

# Deployment Platforms
- ArkInfer Supported Hardware:
  - NVIDIA Jetson Series
  - Qualcomm Snapdragon
  - Rockchip RK3588

5.2 Edge Deployment Example

# Install ArkInfer
pip install arkinfer

# Load Model
from arkinfer import Executor
model = Executor.load("MiniCPM4-0.5B", accelerator="rk3588")

# Long-Context Inference
result = model.generate(
   input_text="Summarize this 128K document's key arguments...",
   max_length=4096,
   sparse_attention=True
)

Technical FAQ

❓ How to choose between 0.5B and 8B versions?

Deployment Recommendations:

• Mobile/IoT Devices → MiniCPM4-0.5B (0.1GB storage)
• Desktops/Workstations → MiniCPM4-8B (higher accuracy)
• Extreme Memory Constraints → BitCPM4 (ternary quantization)

❓ What are the advantages of on-device deployment?

Comparative Analysis:

❓ How to reproduce research results?

Key Steps:

1. Data Preparation: Apply UltraClean to FineWeb dataset
2. Hyperparameter Search: Execute ModelTunnel v2 workflow
3. Four-Stage Training:

   • Stable pretraining (7T tokens)
   • Annealing phase (1T tokens)
   • Long-context extension (20B tokens)
   • SFT + RL fine-tuning
4. Quantization: Use open-sourced BitCPM4 training scripts

Conclusion: MiniCPM4’s algorithm-data-system co-design enables 80+ benchmark-performing LLMs to run efficiently on edge devices. This breakthrough unlocks new possibilities for mobile AI, privacy-preserving computing, and real-time decision making – accelerating the transition from cloud-centric to edge-native intelligent systems.