TLive-Omni: An Open-Source Omni-Modal Model for E-Commerce Live Streaming – 4B Outperforms 30B

Live streaming presents a unique challenge for AI understanding. Information is scattered across multiple channels and cross-references itself constantly. The host says “this shoe has a hidden 4cm height boost” while flipping the sole toward the camera, and chat messages flash prices simultaneously. Single-modality models inevitably miss connections.

Taobao & Tmall Group recently open-sourced TLive-Omni to tackle exactly this scenario. Two sizes are available – 4B and 9B – with model weights and a technical report now public. The benchmark results tell an interesting story: on core e-commerce live streaming metrics, the 4B model matches or surpasses closed-source heavyweights, and it doesn’t sacrifice general benchmark performance either.

Let’s walk through the architecture, training pipeline, and real numbers.

Getting Started: Installation and Basic Usage

The environment requirements are specific but straightforward: Python 3.10, Linux x86_64, CUDA 12.8, and PyTorch 2.10.0.

conda create -n tlive python=3.10 -y
conda activate tlive
pip install -r environments/requirements.txt

The requirements.txt includes custom wheels built for this specific environment and hardware. If your CUDA version, GPU architecture, or Python version doesn’t match, you’ll need to replace them with compatible builds.

Both model sizes (4B and 9B) are available on Hugging Face. Both have gone through the full SFT + Faithful-RFT pipeline. The 4B version is sufficient for most e-commerce tasks; the 9B version pushes metrics to state-of-the-art among open-source models across the board.

Inference supports five input modes: text-only, single image, standalone audio, video with audio track, and silent video.

# Text only
python examples/inference_five_modes.py \
  --model /path/to/model --mode text

# Image input
python examples/inference_five_modes.py \
  --model /path/to/model --mode image --image data/image.jpg

# Video with audio track
python examples/inference_five_modes.py \
  --model /path/to/model --mode vocal-video --video data/vocal_video.mp4

# Silent video (visual frames only)
python examples/inference_five_modes.py \
  --model /path/to/model --mode silence-video --video data/silence_video.mp4

For temporal localization outputs, the recommended interval format is MM:SS - MM:SS, e.g., 01:23 - 01:35.

If you need to control whether the processor uses the audio track from a video:

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    use_audio_in_video=True,  # True = with audio, False = visual only
)

vLLM Support

TLive-Omni also runs on vLLM, but you need a custom build. The project provides a pre-built wheel based on vLLM 0.19.0, compiled specifically for Python 3.10 + CUDA 12.8 + Linux x86_64 + NVIDIA H20 (sm_90). If you’re not on H20 or your driver/CUDA versions differ, you’ll need to build from source using the code in vllm/.

pip install https://github.com/TaoLiveAIGC/TLive-Omni/releases/download/v1.0.0-rc1/vllm-0.19.0+cu128-cp310-cp310-linux_x86_64.whl

The vLLM inference interface mirrors the standard version – just swap the script for inference_vllm_five_modes.py.

Architecture: How Audio and Video Are Integrated

TLive-Omni uses Qwen3.5 as its backbone and grafts the AuT audio encoder from Qwen3-Omni. The AuT encoder was pre-trained on 20 million hours of audio data and compresses speech to approximately 13 tokens per second – efficient enough to fit long audio streams into the context window.

On the vision side, it follows Qwen3.5’s native pipeline. A single image yields (h/32)*(w/32) tokens after spatial merging. For video, the token count follows the sampled frames: ceil(f/2)*(h/32)*(w/32). That ceil(f/2) matters – we’ll come back to why.

The architectural choice is straightforward: audio and vision each go through their own encoders, both projected into the same embedding space via lightweight MLP aligners, and the language model generates text output. Different training stages freeze or unfreeze different components.

Per-vGrid: The Devil Is in the Temporal Alignment

Getting audio and video into the same model isn’t hard. Keeping them temporally aligned is.

Here’s a concrete example from the paper. Suppose you have a 119-frame source video at 30 FPS – about 3.97 seconds. You request sampling at 2 FPS. Because frame indices must be integers, the actual sampled indices are [0, 20, 39, 59, 79, 98, 118] – 7 frames total. The realized sampling rate is 7/119*30 ≈ 1.76 FPS, not the 2 FPS you asked for.

At the requested 2 FPS, each temporal grid would be 1 second. But based on the actual sampled frames, each grid spans 2/1.76 ≈ 1.13 seconds. That’s a 13% difference.

In audio tokens, this deviation shows up as 1-2 tokens per grid (at 13 tokens/second). That doesn’t sound like much. But when your context window is 256K tokens and you’re processing 10+ minutes of live stream footage, the misalignment between audio and video accumulates to a non-trivial level.

Per-vGrid’s solution is simple and effective: compute timestamps from the actual sampled frames, not from the requested frame rate. Each grid gets an explicit textual timestamp prepended, rather than relying on positional encodings. The model directly “sees” the temporal position of each grid in the input sequence, and the audio span follows the actual frame indices.

There’s another detail: temporal patching groups two sampled frames into one grid, padding the last frame if it’s unpaired. Each group corresponds to one grid. So when you adjust the sampling frame count, the number of grids and the duration of each grid both change. Per-vGrid keeps the timestamps and audio spans consistent with these changes – no more “the visual index says second 3, but the audio tokens are actually covering 2.5 to 3.5 seconds.”

Three-Stage SFT: Why Split the Training?

Training is split into three progressive stages. Each stage freezes different components and uses different data mixtures. Hyperparameters are from the paper’s Appendix D.1.

Stage 1: Freeze the language model and audio encoder. Train only the audio aligner. 5 million ASR samples. Learning rate 1e-4, global batch size 1024, warmup ratio 0.01. The goal is simple: map acoustic features to text without interference from other modalities.

Stage 2: Expand the audio dataset to 26 million samples, adding audio captioning and audio QA. Now train both the audio encoder and its aligner. The language model stays frozen. Learning rate drops to 1e-5, batch size doubles to 2048. This stage teaches the audio pathway to “understand” semantics, not just transcribe words.

Stage 3: Mix all modalities – audio, image, video, and text – for a total of 14 million multimodal samples. Freeze the visual and audio encoders. Train the aligners and the language model. Learning rate is 4e-6, warmup ratio 0.05.

All three stages use the same optimizer configuration: AdamW with β1=0.9, β2=0.95, weight decay 0.1, gradient clipping at max norm 1, and DeepSpeed ZeRO-3. Each stage runs for one epoch.

The progressive approach breaks a complex problem into verifiable sub-problems. If you mix everything from the start, a training failure is nearly impossible to debug – you don’t know which modality, which component, or which data source caused the issue.

One more detail from the audio labeling pipeline: ASR labels come from an ensemble of ASR models with voting. Speaker diarization runs two parallel streams – an audio-only time-domain model predicts speaker IDs from acoustic features, while a multimodal LLM predicts from ASR transcripts plus visual cues. The two streams are cross-validated with temporal IoU. High-overlap segments are kept directly; mismatched segments are cross-verified using lip motion cues from video frames.

Synchronized Length-Grouped Sampling: Handling Variable-Length Data

Multimodal training has a practical problem: ASR samples might be a few dozen tokens, while a long video with audio can hit tens of thousands. Random sampling makes GPU utilization wildly uneven.

TLive-Omni’s solution groups samples by modality, sorts each group by token length, then slices them into global batches of fixed size. Samples within a batch have similar lengths, so padding is minimal.

In multi-GPU training, all workers share the same batch registry. At the start of each epoch, they initialize with seed + epoch and rebuild the same shuffled order. Each worker gets a local batch that’s a subset of the same global batch – workloads stay balanced.

The sampler also polls modalities in proportion to their remaining batch counts until all data is consumed. If a modality’s samples don’t divide evenly by the batch size, the remainder is discarded – a practical trade-off to keep batch size fixed.

Faithful-RFT: Reinforcement Fine-Tuning for Faithful Responses

After SFT, there’s an additional reinforcement learning stage using GRPO. Hyperparameters from the paper: G=8 candidate responses per prompt, reward normalization with 1e-4 smoothing, asymmetric clip bounds – lower 0.2, upper 0.28 – and KL coefficient β=0.1.

The asymmetric clipping favors conservative updates – it prevents a single bad step from pushing the policy too far. A β of 0.1 is moderate to conservative in GRPO practice.

Reward design routes by task type. A single batch may contain multiple-choice QA, visual grounding, and ASR transcription. Multiple-choice uses rule-based answer matching, visual grounding uses IoU, ASR uses CER. Each data point activates only the reward functions applicable to its task – inapplicable rewards return an invalid sentinel. Valid rewards are then weight-normalized to produce the final scalar reward.

Crucially, the reward mechanism discourages explicit reasoning traces. It doesn’t reward the model for writing out “what I’m thinking.” For real-time live streaming scenarios, that’s the right trade-off.

The rollout engine uses vLLM with a dynamic resampling mechanism: if the eight responses for a prompt show near-zero reward variance, the group provides no meaningful relative preference signal. It gets discarded and regenerated.

There’s also an In-Context ASR evaluation setting worth noting. The model receives a candidate keyword list (0 to 1000 terms) before transcription – product names, brand names, domain-specific terms. The zero-keyword setting serves as the no-context baseline. Larger lists test whether additional context improves keyword recognition or introduces interference. Results show keyword prompting improves both keyword recall and overall CER (character error rate), with TLive-Omni-9B achieving the best performance across all non-zero keyword list sizes.

Benchmark Results: What Can a 4B Model Actually Do?

Evaluation covers a wide range: ASR, speaker-attributed ASR (cpWER), audio description and QA, visual grounding, text understanding, temporal grounding, dense video captioning, video QA, and shot understanding.

Speech Recognition

TLive-Omni-9B achieves 6.46% CER, 4B at 6.66%. For comparison: Qwen3.5-Omni Flash at 6.81%, Qwen3-Omni (30B-A3B) at 6.75%, Gemini 2.5 Pro at 11.48%. For speaker-attributed ASR (cpWER), 9B hits 12.27% – the best among open-source models. Qwen3-Omni (30B-A3B) is at 27.84%, Gemini 3.5 Flash at 11.99%.

Visual Grounding

Average Precision at IoU=0.5 on live stream frames: 4B at 82.85%, 9B at 82.33%. Qwen3-Omni (30B-A3B) at 79.22%, Gemini 3.5 Flash at 84.15%. On product images: 4B at 91.45%, 9B at 89.96%, Qwen3-Omni at 68.88%, Gemini 3.5 Flash at 74.89%.

Text Understanding

Text localization F1: 9B at 87.59%, 4B at 86.99%. Recognition normalized edit distance (NED): 9B at 4.24%, 4B at 4.72%. Qwen3.5-Omni Flash is at 12.48%. Text classification accuracy: 9B at 79.85%.

Temporal Grounding

mIoU: 9B at 81.49%, 4B at 77.63%. Gemini 2.5 Flash at 76.50%, Qwen3-Omni at 39.22%.

Dense Video Captioning

9B: 74.63% accuracy, 8.76% hallucination rate. 4B: 69.23% accuracy, 9.57% hallucination rate. Gemini 2.5 Flash: 54.60% and 10.97%, respectively.

General Benchmarks

The model doesn’t sacrifice general capability either. On MMMU: 9B at 73.4%. MathVista: 81.9%. MMBench: 88.9% – best among open-source models. RealWorldQA: 76.6% – also best among open-source. On video benchmarks: Video-MME at 75.6% (open-source best), MLVU at 80.9% (open-source best), MMVU at 67.1% (open-source best). On TimeLens-Bench temporal grounding: 4B achieves 57.0% on Charades-TL, 58.2% on ActivityNet-TL, and 69.2% on QVHighlights-TL – all new open-source SOTA. On omni-modal benchmarks: AVUT at 80.0%, WorldSense at 56.0%, DailyOmni at 80.5% – all open-source best.

The overall pattern is consistent: the 4B model competes head-to-head with 7B-9B general-purpose models on most tasks and outperforms them on e-commerce-specific metrics. The 9B version pushes most indicators to the leading position among open-source models.

Quick Reference

  • Environment: Python 3.10 + Linux x86_64 + CUDA 12.8 + PyTorch 2.10.0
  • Model sizes: 4B and 9B, available on Hugging Face
  • Inference modes: text / image / standalone audio / video with audio / silent video
  • Temporal output format: MM:SS - MM:SS
  • vLLM: Custom wheel available, but only for H20 + sm_90 environments
  • Per-vGrid: Timestamps computed from actual sampled frames, audio spans follow
  • Three-stage training: Audio alignment → audio understanding → full multimodal SFT
  • Faithful-RFT: GRPO-based reinforcement learning with explicit reasoning traces suppressed

FAQ

Q: Which model size should I choose – 4B or 9B?

A: The 4B model is sufficient for most e-commerce live streaming tasks and offers better cost-performance. The 9B version performs better on general benchmarks and fine-grained tasks like temporal grounding and dense captioning. If you have the compute budget, go with 9B.

Q: Do I need to use the custom vLLM wheel?

A: If your GPU is an H20 and you’re on Python 3.10 + CUDA 12.8 + Linux x86_64, you can install the pre-built wheel. Otherwise, build from source using the code in vllm/.

Q: What’s the key takeaway for temporal alignment?

A: Don’t compute timestamps from the requested frame rate – compute them from the actual sampled frame indices. Otherwise, the audio span drifts relative to the visual grid, and the error accumulates over long videos.

Q: How long of a live stream segment can the model process?

A: The context window is 256K tokens. With audio at ~13 tokens/second plus visual tokens, actual duration depends on your sampling rate and video resolution. Roughly speaking, segments from a few minutes up to 10-15 minutes are within reach.

Q: How is Faithful-RFT different from standard RLHF?

A: Faithful-RFT scores the final response quality directly and doesn’t reward the reasoning process. The objective is fast, accurate, and concise answers – optimized for real-time live streaming scenarios.

Q: How does In-Context ASR work in practice?

A: You provide a candidate keyword list (product names, brand names, etc.) before transcription. The list can range from 0 to 1000 terms. Benchmark results show that keyword prompts significantly improve recall for domain-specific terms while also reducing overall CER.

Q: Can I use this model commercially?

A: Yes. It’s released under the Apache 2.0 License.