Monet: Revolutionizing Visual Reasoning in AI’s Latent Space

Introduction: The Quest for Human-like Visual Intelligence

Imagine looking at a complex infographic and immediately understanding which data points matter most. Or glancing at a geometric diagram and intuitively seeing the solution. This human ability to “think with images” has long eluded artificial intelligence systems. While AI can now recognize objects in images with remarkable accuracy, true visual reasoning—the capacity to analyze, interpret, and draw conclusions from visual information—remains a significant challenge.

Recent advances in multimodal large language models have begun to bridge this gap. These systems can process both text and images, generating insights that draw from visual cues. However, most existing approaches rely on external tools: cropping image regions, calling specialized vision models, or generating code to manipulate images. While helpful, these methods lack the flexibility and efficiency of human visual thinking.

Enter Monet, a groundbreaking framework that enables AI to reason directly within the latent visual space—the abstract representation where meaning is encoded. Named after the impressionist artist Claude Monet, this approach allows models to form “visual thoughts” much like humans might mentally manipulate images without physically altering them.

In this comprehensive exploration, we’ll uncover how Monet represents a paradigm shift in visual reasoning, examine its innovative three-stage training process, and understand why it outperforms traditional methods across diverse tasks—from analyzing complex charts to solving abstract visual puzzles.

Understanding Visual Reasoning: Beyond Simple Image Recognition

What is Visual Reasoning?

Visual reasoning extends far beyond identifying objects in an image. It involves:

• Perception: Recognizing elements and their properties
• Analysis: Understanding relationships between elements
• Inference: Drawing logical conclusions based on visual evidence
• Application: Using visual insights to solve problems or answer questions

For example, when presented with a sales chart, visual reasoning enables you to not only read the numbers but identify trends, make comparisons, and predict future performance.

Limitations of Current Approaches

Most current multimodal AI systems face significant constraints:

Tool Dependency: Models trained to use specific visual tools (like bounding box prediction) struggle with tasks requiring different types of visual operations. A system optimized for cropping may fail at spatial reasoning problems that require mental rotation or perspective-taking.

Cognitive Overhead: Tool-dependent reasoning increases training complexity. Models often fail to generate valid tool calls without extensive supervision, limiting their practical application.

Deployment Challenges: Reliance on external tools or interpreters necessitates multi-turn, asynchronous inference, complicating real-world deployment and increasing response latency.

Rigidity: These systems lack the fluid, adaptive nature of human visual thinking, which seamlessly moves between different types of visual processing as needed.

The Monet Breakthrough: Thinking in Latent Space

Core Innovation: Latent Visual Reasoning

Monet’s fundamental insight is simple yet profound: instead of manipulating external images, why not enable models to reason directly within their internal representation space?

The framework trains multimodal large language models to generate continuous embeddings that function as intermediate visual thoughts. These “latent embeddings” serve as compact, meaningful representations of visual concepts that the model can manipulate during reasoning.

Think of it as the difference between physically rearranging paper cutouts versus mentally manipulating concepts. The latter is faster, more flexible, and doesn’t require external resources.

Technical Challenges Addressed

The Monet team identified and solved two critical challenges in latent visual reasoning:

Computational Efficiency: Traditional latent-visual alignment requires processing hundreds or thousands of image tokens, creating prohibitive computational costs. Monet introduces a more efficient alignment strategy that focuses on key observational tokens.

Supervision Gap: Next-token prediction provides weak supervision for latent embeddings, as models can simply memorize subsequent tokens rather than learning meaningful representations. Monet’s three-stage training pipeline ensures latent embeddings capture genuinely useful visual features.

Inside Monet’s Architecture: A Three-Stage Training Framework

Stage 1: Warm-up Training

The foundation of Monet begins with adapting the base model to interleaved reasoning patterns. Using the carefully curated Monet-SFT-125K dataset, the model learns to leverage intermediate-step images when predicting subsequent tokens.

This stage proves crucial—without it, models tend to ignore auxiliary images, and their observational token representations fail to capture sufficient visual information. The warm-up training essentially teaches the model when and how to use visual evidence in its reasoning process.

Research confirmed this behavior: unadapted base models showed almost no improvement in predicting observation tokens when using auxiliary images, while warm-up trained models demonstrated significantly better utilization of visual cues.

Stage 2: Obtaining High-Quality Target Latent Embeddings

This stage introduces a teacher-student framework where both are initialized from the warm-up model. The teacher processes chains of thought with ground-truth auxiliary images, while the student generates latent embeddings and reasons with them.

Key innovations in this stage include:

Alignment on Key Observation Tokens: Since latent embeddings should serve the same role as auxiliary images in predicting observation tokens, their hidden representations are aligned under both conditions using cosine similarity loss.

Structured Attention Flow: A modified attention mask ensures latent embeddings can directly attend to auxiliary image embeddings, creating a clear information pathway: auxiliary images → latent embeddings → observation tokens.

This dual approach ensures that latent embeddings capture meaningful visual features while maintaining focus on downstream reasoning.

Stage 3: Learning to Generate Latent Embeddings Without Assistance

The final supervised stage closes the gap between training and inference by removing access to ground-truth auxiliary images. The model learns to produce latent embeddings that align with the high-quality target embeddings obtained in Stage 2.

Unlike previous approaches that aligned only final-layer representations, Monet aligns all layers, providing stronger supervision and more robust learning.

VLPO: Reinforcing Latent Reasoning

Beyond Standard Reinforcement Learning

Traditional reinforcement learning methods like GRPO have a fundamental limitation for latent reasoning: they can only optimize text tokens, leaving latent embeddings largely untrained during reinforcement.

Monet introduces Visual-Latent Policy Optimization, a novel reinforcement learning algorithm specifically designed for latent reasoning. VLPO’s key innovation is estimating the output probability of continuous latent embeddings, enabling them to be optimized directly using reward signals.

How VLPO Works

The algorithm models latent embeddings as samples from a Gaussian distribution, allowing calculation of probability ratios for latent steps. When the advantage is positive, VLPO increases this ratio by pulling policy latent embeddings toward “good-action” embeddings that led to positive outcomes.

This approach enables direct optimization of latent embeddings using reward signals—a capability that GRPO fundamentally lacks.

Reward Design Philosophy

Monet employs a simple but effective reward scheme:

• Accuracy Reward: 1 for correct answers, 0 otherwise
• Format Reward: Encouraging proper answer formatting

Notably, the system doesn’t reward latent-reasoning behavior itself, preventing the model from invoking latent reasoning indiscriminately. This careful design ensures that latent reasoning emerges naturally when beneficial rather than becoming a default behavior.

The Training Data: Monet-SFT-125K

Quality Over Quantity

The Monet team identified three key limitations in existing image-text interleaved datasets:

1. Many samples could be solved using only the original image, reducing the incentive to learn meaningful intermediate features
2. Intermediate images were sometimes inaccurate, introducing noise into training
3. All text tokens were treated equally, overlooking those describing crucial visual information

Three-Stage Data Curation

To address these issues, the team developed a rigorous filtering pipeline:

Stage 1: Collect raw data from multiple sources (ReFocus, CogCoM, Zebra-CoT, Visual-CoT) and retain only samples that the base model couldn’t solve using just the question and original image.

Stage 2: Further filter to keep only samples that a more powerful model could solve using only the auxiliary images, ensuring these images contained accurate and sufficient information.

Stage 3: Use advanced LLMs to identify text tokens corresponding to crucial visual observations, providing fine-grained supervision for learning latent embeddings.

The resulting Monet-SFT-125K dataset contains 125,000 high-quality samples spanning real-world scenarios, documents, charts, and geometry problems, with visual operations ranging from simple cropping to creating entirely new visual representations.

Performance and Evaluation

Benchmark Dominance

Monet was evaluated across multiple standardized benchmarks covering real-world perception and reasoning tasks:

V*: Tests fine-grained visual understanding
HRBench (4K and 8K): Evaluates hierarchical reasoning capabilities
MME-RealWorld: Assesses performance on practical visual tasks
VisualPuzzles: Measures abstract visual reasoning on unseen problem types

Results Analysis

Monet demonstrated consistent improvements across all evaluated domains:

• Substantial gains over base models: 4.25% to 9.75% improvement over Qwen2.5-VL-7B
• Superior to standard training approaches: Outperformed both vanilla supervised fine-tuning and SFT+GRPO trained on the same data
• Advantage over specialized methods: Surpassed both Deepeyes (a cropping-based approach) and LVR (an earlier latent visual reasoning method) on most benchmarks

Most impressively, Monet exhibited strong out-of-distribution generalization, achieving top performance on VisualPuzzles despite these abstract visual reasoning problems being unseen during training.

Component Analysis: What Makes Monet Work?

Through careful ablation studies, the Monet team verified the necessity of each component:

Dual Supervision Signals

Removing either the observation token alignment or the auxiliary image visibility significantly degraded performance, confirming that both visual supervision on latent embeddings and representation alignment are essential.

VLPO’s Effectiveness

Unlike GRPO, which provided inconsistent improvements, VLPO consistently enhanced the SFT model’s performance, particularly on out-of-distribution tasks.

Latent-Only Backpropagation

Allowing alignment losses to update non-latent representations caused dramatic performance drops, indicating that models would exploit shortcut paths without this constraint.

Practical Applications and Case Studies

3D Spatial Reasoning

When presented with chairs arranged at specific angles and asked to identify matching angles from options, Monet generated latent embeddings to analyze the spatial relationships directly, rather than attempting to describe angles verbally. This approach mirrors human intuitive visual thinking.

2D Transformation Tasks

For problems requiring understanding reflection patterns (like determining how a number would appear after multiple reflections), Monet used latent reasoning to identify transformation rules that weren’t explicitly stated.

Complex Diagram Understanding

Faced with sales data across multiple countries, Monet demonstrated hierarchical reasoning: first focusing on the relevant chart section through latent embeddings, then extracting and comparing specific values.

Fine-Grained OCR

In tasks requiring information extraction from dense text, Monet accurately located and interpreted target information, even when positioned in visually complex areas of the image.

Adaptive Reasoning Strategy

Notably, Monet doesn’t always invoke latent thinking. For pure-text mathematical problems, it relied exclusively on textual reasoning, demonstrating an ability to match reasoning strategies to problem types.

The Science of Latent Embeddings

Optimal Embedding Count

Research revealed interesting patterns about how the number of latent embeddings affects performance:

• For in-distribution tasks, performance typically peaked at test-time latent sizes larger than training sizes
• Only VLPO-enhanced models consistently benefited from latent reasoning on out-of-distribution tasks
• VLPO improved robustness to the choice of test-time latent size
• GRPO primarily strengthened non-latent reasoning, providing limited benefits for latent reasoning

These findings suggest that SFT alone cannot induce strong out-of-distribution generalization and that explicit optimization of latent embeddings (as in VLPO) is necessary for robust performance.

Scaling Properties

The relationship between training and test-time latent sizes indicates that Monet-SFT enables test-time scaling of latent embeddings, while VLPO extends this scaling behavior to challenging out-of-distribution scenarios.

Limitations and Future Directions

Current Constraints

Despite its impressive capabilities, Monet has certain limitations:

Training Complexity: The multi-stage SFT pipeline increases overall training complexity and computational requirements. Future work might explore more streamlined approaches that maintain performance while reducing overhead.

Reward Design Exploration: The research hasn’t yet explored how different reward structures might influence latent visual reasoning, leaving room for potential enhancements.

Promising Research Directions

Monet opens several exciting avenues for future work:

Unified Reasoning Frameworks: Developing systems that seamlessly integrate latent visual reasoning with other reasoning modalities.

Efficient Training Methods: Creating more computationally efficient approaches to latent reasoning training.

Broader Application Domains: Applying latent reasoning principles to other domains requiring complex visual understanding, such as scientific analysis or creative tasks.

Conclusion: The Future of Visual Reasoning

Monet represents a significant leap toward more human-like visual intelligence in AI systems. By enabling reasoning in the latent space, it overcomes the limitations of tool-dependent approaches while demonstrating stronger generalization and flexibility.

The framework’s carefully designed training pipeline addresses fundamental challenges in latent visual reasoning, while VLPO ensures that latent embeddings receive appropriate optimization during reinforcement learning. The result is a system that not only excels at standard visual reasoning tasks but also demonstrates remarkable adaptability to novel problems.

As AI continues to evolve, approaches like Monet point toward a future where systems can understand and reason about visual information with human-like flexibility and insight. Rather than treating visual reasoning as a specialized subproblem, Monet integrates it naturally into the reasoning process—much as humans seamlessly blend visual and conceptual thinking.

The true promise of Monet lies not just in its current capabilities but in its paradigm-shifting approach to visual intelligence. By moving beyond external manipulations to internal representations, it opens new possibilities for more efficient, flexible, and general visual reasoning systems.

For researchers and practitioners, Monet offers both practical advances and conceptual inspiration. Its code, model, and data are openly available, inviting the broader community to build upon this foundation and explore new frontiers in visual reasoning.

As we stand at this intersection of visual perception and reasoning, Monet lights a path toward AI systems that don’t just see the world but understand it—transforming pixels into insights and images into intelligence.

---

Monet’s code, models, and dataset are available at https://github.com/NOVAglow646/Monet for research and development purposes.

Frequently Asked Questions

How does Monet differ from other visual reasoning approaches?

Monet moves beyond tool-dependent methods by enabling reasoning directly in the latent space. Instead of manipulating external images through cropping or annotation, it generates compact latent embeddings that serve as internal visual thoughts, allowing more flexible and efficient reasoning.

What types of visual tasks can Monet handle?

Monet has demonstrated strong performance across diverse domains including real-world perception, chart understanding, document analysis, geometric reasoning, spatial reasoning, and abstract visual puzzles. Its flexible architecture allows it to adapt to various visual reasoning challenges.

Does Monet always use latent reasoning?

No, Monet adapts its reasoning strategy to the problem. For non-visual problems (like pure-text math questions), it relies on standard textual reasoning. The system automatically determines when to invoke latent reasoning based on the problem context.

How was Monet trained?

Monet underwent a three-stage training process: warm-up adaptation to interleaved reasoning, supervised learning to generate high-quality latent embeddings, and training to produce these embeddings without access to ground-truth auxiliary images. This was followed by specialized reinforcement learning using VLPO.

What are the practical applications of Monet?

Potential applications include automated data analysis from charts, visual question answering, educational tools for visual subjects, document understanding systems, and AI assistants capable of reasoning about visual information in real-world scenarios.