Preventing RLHF Training Crashes in Large Language Models
1 months ago 高效码农Why RL for Large Language Models Keeps Crashing — and the 7 Engineering Tweaks That Finally Made a 30B MoE Stable After 300k GPU Hours “ What makes policy-gradient RL for LLMs explode, and how do we stop it? Token-level objectives are only a first-order approximation of the true sequence reward. When the training-inference gap or policy staleness grows, the approximation breaks. Importance sampling, clipping and Routing Replay keep the two gaps small and training stable. 0. One-glance cheat-sheet Scenario Must-have knobs Typical failure signal Proven combo in paper Pure on-policy (N=1) Importance-Sampling (IS) KL(μ‖π) ↑ entropy ↓ MiniRL w/ …
How EGGROLL’s Hyperscale Evolution Strategies Revolutionize Gradient-Free AI Training
1 months ago 高效码农Evolution Strategies Go Hyperscale: How EGGROLL Trains Billion-Parameter Models Without Gradients A plain-language walkthrough of the paper “Evolution Strategies at the Hyperscale” Written for college-level readers who want facts, not fluff Word count: ≈ 3 200 1. Why should I care about “gradient-free” training? Because back-propagation is not always the best tool. Situation Why gradients struggle Model uses int8 weights only Tiny round-off errors explode during backward pass System contains non-differentiable code (hash table, cellular automaton, database call) Chain rule breaks Very long recurrent loops Vanishing/exploding signal You already own a huge inference cluster GPUs sit idle while you wait …
VibeThinker-1.5B: Compact AI Model Achieves High Performance At Scale
1 months ago 高效码农Exploring VibeThinker-1.5B: A Compact AI Model That Thinks Like the Big Ones Have you ever wondered if a small AI model could tackle tough math problems or write code as well as those massive ones that take up server farms? It sounds counterintuitive—after all, the tech world often pushes for bigger models with billions or trillions of parameters to get better results. But what if the key isn’t just size, but smarter training? That’s where VibeThinker-1.5B comes in. This 1.5 billion-parameter model, developed by a team at Sina Weibo, flips the script. It uses a fresh approach to post-training that …
DreamGym: Revolutionizing Synthetic RL for AI Agents with Synthesized Trajectories – Ultimate Guide
2 months ago 高效码农Scaling Agent Learning Through Experience Synthesis: An Introduction to DreamGym What Is DreamGym and Why Does It Matter for AI Agents? DreamGym is a groundbreaking framework that makes reinforcement learning (RL) for large language model (LLM) agents more practical by creating synthetic experiences instead of relying on expensive real-world interactions. At its core, it addresses the biggest hurdles in training AI agents—like high costs, limited task variety, unreliable feedback, and complex setups—by using a reasoning-based model to generate diverse, high-quality data. This approach allows agents to learn effectively in a controlled, scalable way, leading to better performance in real applications …
How to Train LLMs on Apple Silicon with MLX-LM-LoRA: A Step-by-Step Guide
7 months ago 高效码农Deep Dive into MLX-LM-LoRA: Training Large Language Models on Apple Silicon Introduction In the rapidly evolving landscape of artificial intelligence, training Large Language Models (LLMs) has become a focal point for both research and industry. However, the high computational costs and resource-intensive nature of LLM training often pose significant barriers. Enter MLX-LM-LoRA, a groundbreaking solution that enables local training of LLMs on Apple Silicon devices. This comprehensive guide explores the technical principles, real-world applications, and step-by-step implementation of MLX-LM-LoRA, tailored to meet the needs of developers, researchers, and enthusiasts alike. Understanding the Core Technology: MLX and LoRA 2.1 The Foundations …