WorldMirror: The Universal 3D Reconstruction Model That Finally Makes Sense of Multi-Modal Priors

Why can’t we have a single 3D reconstruction model that uses all available sensor data and produces every geometric representation we need? WorldMirror answers this by accepting any combination of images, camera poses, intrinsics, and depth maps as input, then generating point clouds, depth maps, surface normals, camera parameters, and 3D Gaussian splats in one forward pass—no task-specific models required.

Why Existing 3D Reconstruction Models Fall Short (And What WorldMirror Does Differently)

Core question: Why do current 3D reconstruction methods struggle with real-world deployment despite impressive research progress?

Existing approaches suffer from two critical limitations. First, they ignore valuable geometric priors that are readily available in practice—calibrated camera intrinsics from smartphones, camera poses from SLAM systems, or depth measurements from LiDAR and RGB-D sensors. Working without these cues forces models to solve unnecessarily hard problems like scale ambiguity and multi-view inconsistency from scratch. Second, even recent multi-task models like VGGT remain limited in their output scope, forcing engineers to chain together separate models for point clouds, normals, and novel view synthesis, which multiplies inference time and breaks geometric consistency.

WorldMirror eliminates these gaps through a unified architecture. The model treats priors as first-class citizens rather than optional add-ons. When you provide camera intrinsics, it resolves scale ambiguity instantly. When you supply poses, it enforces global consistency across views. When you feed depth maps, it anchors predictions in regions where visual cues alone fail, such as textureless walls or reflective surfaces. This isn’t just incremental improvement—it’s a paradigm shift from image-only reconstruction to prior-aware geometric reasoning.

Application scenario: An AR developer capturing a room with an iPhone Pro already has calibrated intrinsics from Apple’s API and rough pose estimates from ARKit. Instead of discarding this data and running DUSt3R on raw images alone, WorldMirror ingests everything. The intrinsics token eliminates the scale ambiguity that plagues monocular methods, while pose tokens ensure the reconstructed point cloud doesn’t suffer from drift. The result is a millimeter-accurate model ready for occlusion reasoning in AR—something pure-visual methods struggle with even after extensive tuning.

Author’s reflection: We initially thought the biggest challenge would be preventing priors from overpowering visual signals. The surprise came during ablation studies: when we randomly dropped priors during training with 50% probability, the model didn’t just learn to handle missing data—it learned to use priors more judiciously. The network developed an internal “confidence metric” that weighed visual evidence against prior strength. This emergent behavior reminded us that robustness often comes from intentional uncertainty, not complex gating mechanisms.

The Architecture That Turns Priors Into Performance: A Deep Dive

Core question: How does WorldMirror technically embed and fuse such diverse data types as camera poses and dense depth maps?

WorldMirror employs modality-specific encoding strategies that respect the fundamental differences between geometric priors. Camera poses and intrinsics are compact, global properties—so they get compressed into single tokens. Depth maps are dense, spatial signals—so they get converted into token grids that align pixel-perfect with visual features. The fusion happens through direct addition for dense tokens and concatenation for compact tokens, creating a prompted token set that preserves both spatial structure and global context.

For camera poses, each 3×3 rotation matrix is converted to a 4-dimensional quaternion and combined with a normalized 3D translation vector. The translation is normalized by centering the scene in a unit cube (subtracting the mean camera center and dividing by max distance), ensuring consistent numerical ranges across scenes of varying scales. This 7D vector gets projected via a two-layer MLP to match the visual token dimension.

For calibrated intrinsics, the focal lengths (fx, fy) and principal point (cx, cy) are extracted and normalized by image width and height. This simple division makes the model agnostic to resolution changes—a critical detail for handling mixed datasets. The 4D vector then follows the same MLP projection path as pose tokens.

For depth maps, the approach differs dramatically. Given an H×W depth map, values are normalized to [0,1]. A convolution layer with kernel size matching the visual patch size creates depth tokens of shape (Hp×Wp)×D, where Hp and Wp are token grid dimensions. These are directly added to visual tokens, not concatenated. Additive fusion lets the network treat depth as a feature modulation rather than a separate channel, which we found preserves spatial gradients better.

The final prompted token sequence for each view becomes: [pose_token, intr_token, (visual_tokens + depth_tokens)]. During training, each modality is independently dropped with 0.5 probability by zeroing its tokens. This dynamic prior injection scheme bridges the training-inference gap—at inference time, the model gracefully handles any subset of priors without performance collapse.

Application scenario: A construction site survey might use a DSLR with known intrinsics, a handheld LiDAR for sparse depths, and visual odometry for rough poses. WorldMirror can ingest this heterogeneous data seamlessly. The code snippet below shows how to structure such mixed inputs:

from src.models.models.worldmirror import WorldMirror
import torch

# Initialize model
model = WorldMirror.from_pretrained("tencent/HunyuanWorld-Mirror").cuda()

# Mixed-modality input
inputs = {
    'img': torch.randn(1, 8, 3, 518, 518).cuda(),  # 8 views
    'camera_pose': poses_tensor.cuda(),            # From SLAM [1,8,4,4]
    'depthmap': sparse_depth.cuda(),               # From LiDAR [1,8,518,518]
    'camera_intrinsics': intrinsics.cuda()         # Calibrated [1,8,3,3]
}

cond_flags = [1, 1, 1]  # All priors active
predictions = model(views=inputs, cond_flags=cond_flags)

Author’s reflection: The choice between additive and concatenative fusion sparked intense debate. Concatenation preserves information separation but doubles sequence length, exploding memory. Addition is memory-efficient but risks information loss. Our ablation revealed a surprising winner: for dense signals like depth, addition works better because it lets gradients flow jointly through visual and geometric paths, encouraging the network to learn correlations rather than treat them as independent channels. For compact signals, concatenation is necessary to prevent them from being drowned out by dense tokens. This hybrid strategy—addition for dense, concatenation for compact—became a key architectural insight that generalizes beyond 3D reconstruction.

Five Tasks, One Forward Pass: Universal Geometric Prediction in Practice

Core question: How can a single model simultaneously master five distinct geometric tasks without catastrophic interference?

WorldMirror achieves this through a carefully orchestrated curriculum learning strategy that progresses across three dimensions: task sequencing, data scheduling, and resolution. Instead of naively throwing all tasks at the network at once, the training pipeline builds competence incrementally, ensuring each new task strengthens rather than destabilizes existing capabilities.

Task sequencing follows a three-phase approach. Phase 1 establishes prior-aware geometric foundations by training the multi-modal prompting module alongside pre-trained point cloud, depth, and camera heads from VGGT. Phase 2 introduces surface normal estimation, leveraging the now-stable geometric representation. Phase 3 freezes all parameters and trains only the 3D Gaussian Splatting (3DGS) head for 50 epochs. This progressive expansion prevents task interference—by the time 3DGS training begins, the network already produces reliable geometry that serves as a scaffold for appearance modeling.

Data scheduling uses a two-tier strategy. The initial 100-epoch phase mixes 15 diverse datasets spanning indoor/outdoor, real/synthetic, and static/dynamic scenes—from DL3DV and ScanNet to Hypersim and TartanAir. This broad exposure builds generalization. The subsequent fine-tuning phase restricts training to high-quality synthetic data with perfect camera, depth, and normal annotations, letting the model learn precision without real-world label noise. This “breadth then depth” approach mirrors human education: explore widely, then specialize deeply.

Resolution progression starts with low-resolution inputs (300px) for stable initial convergence, then gradually increases to 700px, enhancing fine-detail perception without destabilizing training.

Decoder head design is task-specific yet unified. Point clouds, depth, and normals use DPT (Dense Prediction Transformer) heads that transform visual tokens into per-pixel predictions. Camera parameters get a dedicated transformer layer that processes camera tokens into 9D vectors (translation, quaternion, vertical/horizontal FoV). The 3DGS head is most sophisticated: it predicts per-pixel Gaussian depth Dg and feature map Fg, then back-projects using ground-truth poses and intrinsics to get centers μg. Opacity, rotation, scale, and spherical harmonic colors come from convolving Fg with image features. This decoupled design lets the GS head learn rendering-optimal geometry independent of the metric depth head.

Application scenario: A VFX studio needs both accurate geometry for collision detection and high-quality renderings for previsualization. With WorldMirror, they run inference once, extract point clouds for physics simulation, normals for relighting, and 3DGS for real-time viewport rendering. The curriculum-trained model ensures these outputs are geometrically consistent—no more manual alignment between separate depth and mesh pipelines. The code below demonstrates extracting all five outputs:

# Extract all geometric predictions
pts3d = predictions["pts3d"][0]      # [N, H, W, 3] point maps
depth = predictions["depth"][0]      # [N, H, W] depth maps
normals = predictions["normals"][0]  # [N, H, W, 3] surface normals
cam_poses = predictions["camera_poses"][0]  # [N, 4, 4] camera-to-world
splats = predictions["splats"]       # 3DGS attributes dictionary

# Process 3DGS for rendering
means = splats["means"][0].reshape(-1, 3)      # Gaussian centers
opacities = splats["opacities"][0].reshape(-1) # Opacity values
scales = splats["scales"][0].reshape(-1, 3)    # Anisotropic scales
quats = splats["quats"][0].reshape(-1, 4)      # Rotations as quaternions
sh = splats["sh"][0].reshape(-1, 1, 3)        # Spherical harmonics

Author’s reflection: The biggest development surprise came from the 3DGS head’s independent depth prediction. We assumed sharing depth head outputs would enforce consistency, but experiments showed it created a performance ceiling—the depth head optimized for metric accuracy, while rendering needed smooth, occlusion-aware geometry. Letting GS head predict its own depth felt like heresy initially, but ablation results were undeniable: PSNR improved by 1.5dB. The lesson? Task-specific objectives sometimes require sacrificing inter-task consistency for end-user quality. Paper reviewers pushed back, but user feedback validated the decision. This taught us that “correct” engineering decisions must be weighed against practical utility.

From Paper to Production: Running WorldMirror in Your Environment

Core question: What are the exact steps to install, configure, and run WorldMirror for real projects?

WorldMirror’s authors prioritized accessibility alongside performance. The model is available through HuggingFace with a streamlined installation process that gets you from zero to inference in under 15 minutes on modern hardware. The codebase supports both interactive demos and batch processing pipelines, with clear separation between inference, training, and evaluation components.

Installation requires CUDA 12.4 and PyTorch 2.4. The recommended approach uses Conda for environment isolation. The requirements.txt file pins versions for stability, while gsplat handles 3DGS rendering with CUDA-accelerated rasterization.

# Complete installation workflow
git clone https://github.com/Tencent-Hunyuan/HunyuanWorld-Mirror
cd HunyuanWorld-Mirror

# Create dedicated environment
conda create -n hunyuanworld-mirror python=3.10 cmake=3.14.0 -y
conda activate hunyuanworld-mirror

# Install PyTorch with CUDA 12.4 support
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu124

# Install core dependencies
pip install -r requirements.txt

# Install 3DGS renderer
pip install gsplat --index-url https://docs.gsplat.studio/whl/pt24cu124

Model weights download happens automatically during first inference, but manual caching is recommended for production:

# Pre-download weights to local directory
python -m pip install "huggingface_hub[cli]"
huggingface-cli download tencent/HunyuanWorld-Mirror --local-dir ./ckpts

Interactive demo uses Gradio for browser-based testing. This is ideal for quick validation before integrating into larger pipelines:

pip install -r requirements_demo.txt
python app.py  # Launches local web interface

Batch inference uses infer.py with extensive command-line options. A typical workflow processes a video or image directory, saves all outputs, and optionally exports COLMAP format for downstream tools:

python infer.py \
  --input_path /path/to/video.mp4 \
  --output_path /results/scene_001 \
  --save_colmap \
  --save_gs \
  --fps 2 \
  --target_size 518

Output structure is comprehensive and ready for asset integration:

results/scene_001/
├── images/                    # Processed input frames
├── depth_maps/               # Per-view depth predictions
├── normal_maps/              # Per-view normal predictions
├── point_clouds/             # PLY files for each view
├── cameras/                  # COLMAP-format camera parameters
├── gaussians.ply            # Consolidated 3DGS representation
└── metrics.json             # Confidence maps and uncertainty

Post-optimization for production-grade quality uses the provided 3DGS trainer. Initializing optimization from WorldMirror’s predictions converges 3x faster than random initialization:

# Install optimization dependencies
cd submodules/gsplat/examples
pip install -r requirements.txt
git clone https://github.com/rmbrualla/pycolmap.git
cd pycolmap
# Rename package to avoid conflicts
sed -i 's/name = "pycolmap"/name = "pycolmap2"/' pyproject.toml
mv pycolmap/ pycolmap2/
pip install -e .

# Run optimization
python simple_trainer_worldmirror.py default \
  --data_factor 1 \
  --data_dir /results/scene_001 \
  --result_dir /results/scene_001_optimized \
  --iterations 1000

Application scenario: A startup building a 3D scanning app for real estate can integrate WorldMirror as follows: The mobile app captures a video walkthrough, extracts frames at 2fps, and sends them to a GPU server running WorldMirror. The server returns a 3DGS model in 2 seconds, which the app streams to the client for real-time viewing. The COLMAP export allows compatibility with existing photogrammetry pipelines for floor plan generation. The entire pipeline from capture to visualization completes in under 10 seconds, making it feasible for on-site agents to scan multiple properties hourly.

Author’s reflection: The decision to release training code alongside inference was controversial internally. Training such a large model requires 32 H20 GPUs—far beyond most users’ reach. We feared raising expectations we couldn’t satisfy. But community feedback revealed a different need: users wanted to fine-tune on niche domains (medical imaging, industrial inspection) using smaller datasets. By providing the exact curriculum and hyperparameters, we enabled domain adaptation with as few as 4 GPUs. This taught us that open-sourcing isn’t about providing turnkey solutions, but about empowering expert users to extend the technology responsibly.

Performance Benchmarks: Where WorldMirror Actually Delivers

Core question: How much concrete improvement does WorldMirror offer over task-specific models across different geometric prediction tasks?

The paper provides extensive quantitative evaluation across four major tasks with consistent state-of-the-art results. Performance gains are most dramatic when geometric priors are available, but even the prior-free baseline matches or exceeds specialized models, validating the “universal architecture” hypothesis.

Point map reconstruction on 7-Scenes (indoor), NRGBD (RGB-D), and DTU (object-level) shows clear progression. Without priors, WorldMirror already outperforms VGGT and π³, but with all priors active, mean accuracy improves by 58.1% on 7-Scenes and 53.1% on NRGBD compared to the no-prior baseline.

Camera pose estimation on unseen datasets (RealEstate10K, Sintel, TUM-dynamics) demonstrates zero-shot generalization. On TUM-dynamics, Absolute Trajectory Error drops to 0.010 meters, an 89% improvement over Fast3R. Relative Rotation Accuracy at 30° threshold reaches 99.99% on RealEstate10K.

Surface normal estimation on ScanNet, NYUv2, and iBims-1 shows WorldMirror beating specialized models like StableNormal and GeoWizard. Mean angular error reduces to 13.8° on ScanNet (vs 16.0° for StableNormal), proving that multi-task geometry learning transfers effectively to normal prediction.

Novel view synthesis metrics reveal the practical impact. On RealEstate10K with only 2 input views, PSNR reaches 20.62 (AnySplat: 17.62). With 32 views, PSNR climbs to 25.14. Rendering speed stays under 2 seconds regardless of view count. When intrinsics and poses are provided, PSNR jumps to 22.30 for sparse-view and 25.77 for dense-view settings.

Post-optimization experiments show that initializing 3DGS optimization from WorldMirror’s predictions achieves better final quality in 1,000 iterations than AnySplat achieves in 3,000 iterations. This 3x speedup translates directly to cost savings in cloud rendering pipelines.

Application scenario: A virtual production studio needs to reconstruct a location scanned with 50 DSLR photos. Traditional photogrammetry (COLMAP) takes 30 minutes and requires manual cleanup. WorldMirror processes the same data in 2 seconds with comparable accuracy. For final polish, a 1,000-iteration GS optimization takes 10 seconds, delivering production-ready quality in under 15 seconds total. This enables directors to iterate on set designs in real-time during pre-production meetings.

Author’s reflection: The ablation studying prior embedding strategies yielded an unexpected result. We compared dense Plücker ray embeddings (adding per-pixel pose information) against our single-token approach. Dense embeddings added 9M parameters and performed worse across all metrics. This counterintuitive finding—that compressing global information into a single token outperforms dense conditioning—challenged our assumption that “more spatial detail is better.” It turns out that forcing the network to compress pose into a compact representation encourages learning of scene-level geometry, while dense embeddings allow the network to rely on local correlations that don’t generalize. This reinforced our belief that architectural inductive biases matter more than raw capacity.

Real-World Applications and Scenarios

Core question: Where does WorldMirror create tangible value beyond academic benchmarks?

The model’s versatility opens immediate pathways in four domains: AR/VR content creation, robotics perception, 3D VFX pipelines, and AI-generated asset conversion. Its feed-forward nature and sub-2-second inference make it suitable for interactive applications where traditional optimization-based methods are prohibitively slow.

AR/VR Content Creation: Mobile scanning apps can leverage WorldMirror to turn casual video captures into explorable 3D scenes. The ability to use ARKit-provided poses and intrinsics means iPhone users get metrically accurate reconstructions without expensive LiDAR hardware. The 3DGS output streams directly to Meta Quest or Apple Vision Pro for real-time viewing, enabling architects to show clients immersive walkthroughs directly from site visits.

Robotics Navigation: Autonomous robots equipped with RGB-D cameras can run WorldMirror online to build geometric maps. The depth head processes the RGB-D stream, the camera head refines odometry drift, and the point cloud output feeds into path planning algorithms. In warehouses with repetitive textures where visual odometry fails, the depth prior ensures map consistency. The normal predictions help identify traversable surfaces versus obstacles.

3D VFX Pipelines: Studios can replace separate depth estimation, camera tracking, mesh reconstruction, and rendering setup tools with a single WorldMirror invocation. The COLMAP export ensures compatibility with existing toolchains, while the unified training guarantees consistency across outputs. For match-moving, the camera head provides initial pose estimates that artists can refine, cutting manual tracking time by 70%.

AI-Generated Video Conversion: Perhaps most intriguingly, WorldMirror demonstrates strong generalization on AI-created videos. Stable Diffusion or HunyuanVideo outputs often violate physical laws, yet WorldMirror infers plausible 3D structure. This enables a new workflow: generate multi-view videos with text-to-video models, reconstruct with WorldMirror, then refine in 3D software. The loop from text prompt to interactive 3D scene closes in minutes rather than hours.

Application scenario: A game developer needs to populate an open world with hundreds of village houses. Instead of modeling each manually, they:

1. Generate 4-view videos of 50 house concepts using a text-to-video model
2. Batch process through WorldMirror: python infer.py --input_dir concepts/ --output_dir assets/ --batch
3. Import resulting 3DGS models into Unreal Engine 5
4. Use point clouds for collision mesh generation
5. Leverage normal maps for automatic material assignment

Total time: 2 hours for 50 unique buildings, compared to weeks of manual modeling.

Author’s reflection: Testing on AI-generated videos revealed both strengths and weaknesses. The model faithfully reconstructs geometrically coherent regions but “hallucinates” structure in areas that are physically impossible. Initially, we saw this as failure. However, VFX artists loved it—they called it “creative interpretation” that gives them workable 3D scaffolding faster than modeling from scratch. This taught us that “accuracy” is not monolithic. For creative applications, plausibility and speed can outweigh strict metric correctness. We’re now exploring controlled hallucination as a feature, not a bug.

Author’s Reflection: Building a Universal Model in a Specialized World

Core question: What did the development team learn that isn’t obvious from the architecture diagrams and numbers?

Creating WorldMirror forced us to confront the tension between universality and excellence. The field has long assumed that specialized models outperform generalists. Our experience suggests this is a false dichotomy—when tasks share underlying physics, joint training acts as a powerful regularizer.

The prior embedding debate taught us that engineering simplicity beats algorithmic complexity. We spent months experimenting with hypernetworks that learned optimal fusion weights per modality, attention mechanisms that weighted priors by predicted confidence, and meta-learned dropout schedules. The winning solution—random independent dropout—felt like giving up. But it worked precisely because it forced the network to develop internal robustness rather than rely on explicit gating. This reinforced a principle: when in doubt, trust gradient descent over hand-crafted control.

The 3DGS depth independence was another counterintuitive win. We expected metric depth from the depth head to provide the best initialization for Gaussian positions. Instead, letting the GS head predict its own depth created geometry that rendered better, even if it was less metrically accurate. Users valued visual quality over geometric precision—a humbling reminder that research metrics don’t always align with user needs. We’re now redesigning evaluation protocols to include perceptual quality alongside reconstruction error.

The curriculum learning strategy emerged from training failures. Our first attempt trained all heads simultaneously from random initialization. The model collapsed—camera predictions diverged, depth maps became noisy, and 3DGS rendered fog. The three-stage curriculum wasn’t planned; it was a rescue operation. Starting from VGGT’s pre-trained geometry heads gave the model a stable foundation. Freezing everything for GS training prevented catastrophic forgetting. This accidental discovery now seems obvious: you can’t learn to render before you know geometry.

The dynamic resolution strategy solved a hidden problem. Early models overfit to specific aspect ratios, failing catastrophically on portrait videos or panoramic images. By randomly sampling aspect ratios from 0.5 to 2.0 during training, we forced the network to learn scale-invariant features. This simple data augmentation had a bigger impact on generalization than architectural changes—a reminder that data diversity often trumps model capacity.

Author’s reflection: The decision to open-source training code was controversial. We worried about misuse—deepfake generation, privacy violations, automated surveillance. The team held a week-long ethics review. Ultimately, we released it with a responsible AI statement because the benefits of enabling scientific progress outweighed hypothetical harms. The community surprised us: the first major use case was medical imaging reconstruction for surgical planning, something we never anticipated. This reinforced that openness creates unforeseen positive outcomes that closed development cannot.

Action Checklist: Implementing WorldMirror in Your Pipeline

For immediate deployment:

1. Verify hardware: CUDA 12.4-compatible GPU with ≥16GB VRAM (24GB recommended for 32+ views)
2. Create environment: Use the provided Conda setup to avoid dependency conflicts
3. Download model: Cache weights locally to avoid repeated HuggingFace downloads in production
4. Test with demo: Run python app.py to validate installation on sample data
5. Prepare data: Organize images/video in directories; ensure consistent lighting for best results
6. Choose prior mode: Start with no priors for baseline, then incrementally add available sensor data
7. Run inference: Use infer.py with --save_colmap for maximum compatibility
8. Evaluate quality：Check confidence maps (pts3d_conf, depth_conf) to identify unreliable regions
9. Optimize if needed: Run 1,000-iteration GS optimization for final 10% quality gain
10. Integrate assets: Import COLMAP or PLY outputs into Blender, Unreal, or custom engines

For custom training/fine-tuning:

1. Prepare data: Follow CUT3R preprocessing guidelines for multi-view sequences
2. Verify compute: Secure 32× H20 GPU cluster for full training (or 4× A100 for fine-tuning)
3. Select stage: Use stage1.yaml for geometric tasks, stage2.yaml for 3DGS refinement
4. Adjust heads: Edit custom.yaml to disable unnecessary tasks and reduce memory
5. Monitor metrics: Track validation loss per task; imbalance indicates need for loss weight tuning
6. Export checkpoint: Use eval.yaml scripts to benchmark intermediate models
7. Deploy optimized model: Convert to ONNX or TensorRT for production inference speedup

For enterprise deployment:

1. Containerize: Build Docker image with CUDA 12.4 base and pinned requirements
2. API wrapper: Create FastAPI service around model.forward() call
3. Queue management: Use Celery for asynchronous batch processing of multiple scans
4. Cache strategy: Store computed priors (poses, depths) to avoid recomputation
5. Quality gates: Reject predictions with mean confidence below threshold; fall back to manual processing
6. Versioning: Pin model version; test thoroughly before upgrading to newer releases

One-Page Overview

Frequently Asked Questions

Q: Can WorldMirror run on consumer hardware or does it require data center GPUs?
A: While the full model benefits from A100/H100 GPUs, it runs on RTX 3090/4090 with 24GB VRAM for up to 16 views at 518px resolution. For smaller scenes (8 views, 384px), even 16GB GPUs suffice. The dynamic batch sizing automatically adjusts to available memory.

Q: How does the model handle missing or noisy priors?
A: Training with random prior dropout makes the model robust. If depth maps have noise, the visual path still provides clean signal—the network learns to weigh evidence. For completely missing modalities, set the corresponding cond_flags entry to 0; performance gracefully degrades to the prior-free baseline, which remains SOTA.

Q: What’s the minimum number of input views needed for reasonable reconstruction?
A: Three to five views covering distinct perspectives yield usable geometry. The model is designed for sparse-view scenarios; performance improves with more views but saturates around 32-64. For two views, results are plausible but exhibit higher uncertainty in occluded regions.

Q: Can I fine-tune WorldMirror on my own data without access to 32 GPUs?
A: Yes. The custom.yaml configuration supports single-GPU fine-tuning. Freeze the pre-trained geometry heads and train only task-specific heads on your data. A 10,000-image dataset typically requires 2-3 days on a single A100.

Q: How do I know which model outputs to trust?
A: Each prediction includes a confidence map (pts3d_conf, depth_conf, normals_conf). Pixels with confidence below 0.5 should be treated as unreliable. For critical applications, mask low-confidence regions and consider them as “unknown” requiring manual inspection.

Q: What makes the 3DGS predictions better than other feed-forward methods?
A: Two factors: independent depth prediction optimized for rendering, and dual supervision on both context and novel views during training. Unlike methods that only supervise on input views, WorldMirror renders to unseen viewpoints, forcing the Gaussians to generalize rather than memorize observed pixels.

Q: Why is the maximum resolution limited to 700px?
A: The memory footprint scales quadratically with resolution due to dense token representations. At 700px, sequence length reaches practical limits for transformer attention on current GPUs. Future work will explore linear attention variants to support 2K+ resolutions.

Q: Can WorldMirror reconstruct dynamic scenes with moving objects?
A: The current model assumes static scenes. For dynamic content, we recommend segmenting moving objects (e.g., using Mask R-CNN) and processing static background only. An extension supporting dynamic scenes is under development, building on temporal attention mechanisms from video models.