Deploying Tencent Hy4 Preview: vLLM and SGLang Setup Notes

Tencent’s Hy4 preview is a 770B-parameter MoE model with 49B active parameters and a 1M context window. The weights are available on Hugging Face, ModelScope, GitCode, and CNB, including an FP8 quantized version.

This guide walks through production deployment using vLLM or SGLang, plus the API calls you’ll need to actually use the model.

Where to Find the Real Value in the README

The official README is straightforward—architecture specs, benchmark numbers, license—but the most actionable content lives in the Inference and Deployment section.

Both vLLM and SGLang are officially supported, and both can leverage the built‑in MTP (speculative decoding) layer. Hy4 preview includes one MTP layer (10B parameters, 0.7B active) specifically to accelerate generation.

Deploy with vLLM (from Source)

The README does not recommend pip install vllm; instead, you build from source.

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

• Python 3.12 is required.
• uv manages the virtual environment and dependencies.
• --torch-backend=auto lets pip pick the correct PyTorch wheel for your platform.

Start the server:

vllm serve tencent/Hy4-preview-FP8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 3 \
  --attention-backend FLASHMLA_SPARSE \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy4-preview

Key arguments:

• --tensor-parallel-size 8 – 8‑way tensor parallelism. A 770B model won’t fit on a single GPU; this is mandatory.
• --speculative-config.method mtp and --speculative-config.num_speculative_tokens 3 – enable MTP with 3 draft tokens.
• --attention-backend FLASHMLA_SPARSE – selects the sparse attention backend that matches Hy4’s Gated DSA attention.
• --tool-call-parser hy_v4 and --reasoning-parser hy_v4 – custom parsers for tool calling and reasoning traces.
• --enable-auto-tool-choice – lets the model decide when to invoke tools.

The model path points to the FP8 quantized version. For the BF16 version, use tencent/Hy4-preview, but expect much higher VRAM consumption.

Deploy with SGLang (from Source)

SGLang’s setup is similar, but its speculative decoding implementation differs.

Build from source:

git clone https://github.com/sgl-project/sglang
cd sglang
pip3 install pip --upgrade
pip3 install "transformers>=5.6.0"
pip3 install -e "python"

Note the transformers>=5.6.0 requirement – Hy4 relies on a recent version.

Launch the server:

python3 -m sglang.launch_server \
  --model tencent/Hy4-preview-FP8 \
  --tp-size 8 \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3 \
  --speculative-algorithm EAGLE \
  --port 8000 \
  --served-model-name hy4-preview

SGLang uses the EAGLE algorithm for speculative decoding, not MTP. The parameters --speculative-num-steps 2 and --speculative-eagle-topk 1 are explicitly given in the README.

A notable difference: vLLM explicitly sets --attention-backend FLASHMLA_SPARSE, while SGLang does not – it likely relies on its own or transformers’ implementation for Gated DSA.

Which one to choose? Both expose an OpenAI‑compatible API and support FP8. vLLM’s configuration is more detailed in the README, so start there. If you hit issues, try SGLang.

API Calling: Two Key Parameters

Once the server is running, use the OpenAI client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Please introduce yourself briefly."},
    ],
    temperature=0.9,
    top_p=1.0,
)
print(response.choices[0].message.content)

Two important notes from the README:

1. Recommended parameters – temperature=0.9, top_p=1.0. This is the tested combination, not a random default.
2. Reasoning mode – The default is "high" (deep chain‑of‑thought), suitable for math, coding, and reasoning. For casual chat, pass extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}} to disable the CoT and get a direct response.

The reasoning_effort parameter is particularly useful because Hy4 preview tends to over‑think on simple tasks – a known limitation acknowledged in the README.

Model Specs Quick Reference

The MoE layout: layer 1 is a standard FFN; the remaining 77 layers are MoE with 256 routing experts and 1 shared expert per layer. Each token activates top‑8 routing experts plus the shared expert. That’s how 770B parameters become only 49B active.

The attention side uses Gated DSA with IndexCache for cross‑layer sparse index reuse; the residual side uses iHC (identity Hyper‑Connections). You don’t need to configure these – they’re already handled in vLLM/SGLang.

Fine‑Tuning and Quantization

The README includes separate sections for fine‑tuning and quantization, but they are brief and point to external resources:

• Fine‑tuning guide is at ./finetune/README_CN.md inside the repository.
• Quantization is powered by AngelSlim, Tencent’s compression toolkit.

Since an FP8 version is already provided, most users can deploy that directly. For custom lower‑bit quantization, refer to AngelSlim’s documentation.

Known Limitations

The README explicitly calls out several issues:

• Hy4 preview is an early iteration; both pre‑training and post‑training have room for improvement.
• It tends to over‑validate itself on complex tasks, leading to longer reasoning chains.
• The team plans to incorporate community feedback into the final Hy4 release.

This is not a dealbreaker – it simply means you should use reasoning_effort="no_think" for straightforward queries to avoid unnecessary thinking time.

Quick Start Checklist

Hardware – 8 GPUs. FP8 reduces VRAM significantly; prefer that over BF16.

vLLM path:

1. Create a Python 3.12 environment with uv.
2. Clone and install vLLM from source.
3. Start with vllm serve using --tensor-parallel-size 8, --speculative-config.method mtp, and --attention-backend FLASHMLA_SPARSE.

SGLang path:

1. Clone and install SGLang with transformers>=5.6.0.
2. Start with python3 -m sglang.launch_server using --tp-size 8 and --speculative-algorithm EAGLE.

API call:

• Use OpenAI‑compatible client at http://127.0.0.1:8000/v1.
• Set temperature=0.9, top_p=1.0.
• For simple Q&A, add extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}.

Model weights – Download from Hugging Face, ModelScope, GitCode, or CNB. Append -FP8 for the quantized version.

FAQ

Q: vLLM or SGLang – which should I use?

The README lists vLLM first and provides more detailed arguments (e.g., --attention-backend). Start with vLLM. If you encounter issues, switch to SGLang.

Q: What’s the difference between FP8 and BF16?

FP8 is quantized – lower VRAM usage and faster inference. The FP8 weights are officially provided and recommended for production.

Q: How do I control the reasoning mode?

Pass extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}} in your API request. The default is "high" for full CoT.

Q: MTP vs EAGLE – what’s the difference?

Both are speculative decoding algorithms. vLLM uses MTP (which leverages Hy4’s built‑in MTP layer), while SGLang uses EAGLE. They achieve the same goal but have different parameters.

Q: How can I test the 1M context length?

Send a request with a long content field (around 1M tokens) and check if the server responds without truncation. The README doesn’t provide a test command, but no context‑limit parameter is needed – 1M is supported by default.

Q: Where do I find the quantization tool?

Check the AngelSlim repository. For FP8, you don’t need to run quantization yourself – the pre‑quantized weights are ready.

Q: What hardware is required for fine‑tuning?

The README doesn’t specify, but with 770B parameters, even LoRA will require multiple A100/H100s. Refer to the fine‑tuning guide (./finetune/README_CN.md) for details before planning resources.