WorldClaw: Agentic 3D Open-World Generation at Scale — A Deep Dive
Generating a freely explorable 3D world from a single sentence is deceptively complex. It’s not just about making a pretty picture; the system must simultaneously maintain global spatial coherence, ensure rich local content, and produce explicit assets that you can actually edit, reuse, or drop into a game engine.
WorldClaw tackles this with a “coarse-to-fine, global-to-regional” strategy. Instead of generating everything at once, it establishes the global scene structure first (semantics, spatial layout, terrain foundation) and then progressively fills in detailed content where needed. The entire pipeline is driven by specialized agents that translate high-level intent into executable plans, orchestrate procedural and generative tools, and iteratively refine the output.
We’ll walk through the three core stages—Intent Analysis & Planning, Global Terrain Generation, and Regional Object Generation & Placement—highlighting the actual implementation details, the engineering decisions, and the trade-offs involved.
1. Intent Analysis and Planning: Turning Prompts into Executable Specs
A user prompt like “a tropical pirate island” or “a river canyon with tribal settlements” is far too vague for downstream generators. You need specific details: regional composition, terrain types, object categories, spatial relationships, and visual style.
WorldClaw doesn’t just feed the raw prompt to a generator. It passes it through a two-agent planning module first.
The Intent Analysis Agent does exactly what it says: it extracts and normalizes the explicit constraints from the user prompt. It identifies the scene type, theme, visual style, key regions, objects, and spatial relationships. Crucially, it only summarizes what is already in the prompt. It does not introduce new content or complete missing attributes. This separation ensures the original user intent is preserved and acts as a reliable constraint for subsequent steps.
The Scene Planning Agent takes over to resolve ambiguities and complete the missing information required by the downstream modules. It follows a predefined scene specification schema to produce a structured plan P = (R, C_terrain, C_object). R describes the major regions and their spatial relationships. C_terrain specifies terrain types, landform characteristics, and surface appearance. C_object defines required object categories, approximate densities, and regional spatial relationships. Global properties (theme, atmosphere, material preferences) are retained as shared attributes to keep terrain and object generation consistent.
This stage produces a shared semantic interface. It doesn’t generate geometry yet; it creates a reliable blueprint that guides every subsequent module.
2. Global Terrain Generation: Building the Skeleton
WorldClaw treats terrain not as a flat parking lot for objects, but as the geometric and spatial backbone of the world. It determines the overall shape, scale, navigability, and regional organization.
This is broken down into three sub-stages: Terrain Planning, Terrain Asset Generation, and Terrain Generation & Refinement.
2.1 Terrain Planning
The high-level scene specification P doesn’t provide hard numbers for terrain construction. The Terrain Planning Agent bridges this gap by generating a structured terrain specification P_terrain = (p_layout, p_asset, p_material, θ_terrain).
-
p_layout: Region categories, relative positions, adjacency, and approximate coverage. -
p_asset: Asset categories, regional affinities, and target densities (e.g., rocks in the canyon, vegetation on the plains). -
p_material: Surface types, visual styles, and texture requirements. -
θ_terrain: Numerical parameters like world scale, region-specific base elevations, noise frequencies/amplitudes, geomorphic operators (peaks, dunes, terraces), and boundary blending widths.
If the prompt involves concepts requiring external knowledge, the agent invokes a search tool. If text is insufficient for spatial or stylistic expression, it generates a scene concept image I_concept to condition the subsequent generation.
2.2 Terrain Asset Generation
Now we convert the textual spec into explicit visual conditions. The Terrain Asset Generation Agent produces A_terrain = (I_layout, I_asset, O_asset, M_terrain).
-
Scene Layout Map I_layout: Unlike indoor scenes that use floor plans, outdoor natural terrain has curved boundaries and irregular shapes. WorldClaw generates a 2D semantic layout map where distinct colors encode predefined terrain categories. This map is used for region mask extraction, height-field generation, material assignment, and asset scattering. -
Terrain Asset Prototypes: For repeatedly instantiated elements (rocks, vegetation clusters), the agent generates representative images I_assetand then uses Hunyuan3D’s image-to-3D capability to convert them into reusable 3D asset prototypesO_asset. -
Materials M_terrain: Surface appearance definitions.
2.3 Terrain Generation & Refinement
Initial Height-Field Generation: The agent parses I_layout, extracts masks for each region r, and applies boundary smoothing to get soft weights m_r. The global height field is built as:
[
H(\mathbf{x}) = \sum_{r}\tilde{m}{r}(\mathbf{x})\left[h{r} + \sum_{k}w_{r,k}N_{r,k}(\mathbf{x}) + \sum_{j}\alpha_{r,j}G_{r,j}(\mathbf{x})\right]
]
Where x is a 2D position, h_r is the base elevation, N is noise, and G is a geomorphic operator (peak, dune, terrace). Region-specific composition allows for continuous composite terrain with irregular boundaries. The same weights are used to assign and blend materials.
Global Terrain Asset Scattering: The system instantiates the reusable prototypes O_asset across the terrain. It samples candidate locations based on p_asset and filters/adapts them using local elevation, slope, and surface normals to reduce floating, penetration, and unnatural distributions.
Terrain Refinement: The initial terrain often has abrupt transitions, inconsistent scales, or texture issues. Powered by BlenderMCP, the refinement agent re-renders the scene from predefined viewpoints, inspects the geometry, materials, and scattering, and applies localized corrections. It tweaks region-specific parameters, boundary blends, texture scales, and asset transforms until no major issues are detected or the iteration budget is reached.
3. Regional Object Generation and Placement: Populating the World
With the global terrain established, we move to populating the selected regions. This module handles fine-grained objects with explicit functions and spatial relationships, using a three-step skill pipeline: Region Composition, Object Generation, and Object Placement.
3.1 Regional Planning
The scene specification P doesn’t decide which regions need object refinement or provide executable configurations. The Regional Planning Agent jointly examines P and the generated terrain T, prioritizes regions with uninstantiated object requirements, selects a subset R+ for development, and refines constraints into:
[
\mathcal{P}_{regional} = \left{\left(r,\phi_r,\mathcal{C}_r^{object},\mathbf{p}_r^{spatial},\mathbf{p}_r^{appearance}\right)\mid r\in \mathcal{R}^+\right}
]
Each tuple defines the functional role, object categories/counts, object-object and object-terrain relationships, and regional appearance.
3.2 Object Generation and Placement
Region Composition: For each selected region r, the agent renders the existing terrain with its camera parameters, producing I_r_terrain. This preserves topography, materials, and spatial context. It then generates a composition image:
[
\mathbf{I}r^{comp} = \mathcal{G}{image}\left(\mathbf{I}_r^{terrain},\mathcal{P}r,\mathbf{I}{concept}\right)
]
This 2D composition acts as a layout prior, constraining both appearance and spatial organization.
Object Generation: Reconstructing the entire I_r_comp as a single 3D scene leads to poor instance boundaries and quality. WorldClaw uses text-guided SAM3 to extract individual 2D instances from I_r_comp. Overlapping sliding windows improve recall across scales.
For each extracted instance i, the system builds an object-centric image I_i and mask S_i, applying an affine transform A_i from the region coordinates to object coordinates. SAM3D then predicts the object mesh M_i, appearance U_i, local-to-camera transform T_l2c_i, and reconstruction intrinsics K_i_o.
Single-view reconstruction often has scale deviations, so an image-space scale calibration iteratively adjusts a scale factor λ_i around the mesh center until the projected bounding box area matches the reference area b_i_ref within asymmetric tolerances (over-projection is more heavily penalized).
3D Placement: We recover the 3D placement using a pair of corresponding rays. Cast a ray through the object-center pixel from the reconstruction camera to get the object reference point P_o and depth Z_o. Map the focused image center back to the region composition using A_i^{-1} and cast a ray from the terrain camera to get the terrain anchor P_t and depth Z_t.
The initial 3D scale preserving apparent object size is:
[
s_{i} = \frac{Z_{t}}{Z_{o}}\frac{f_{i}^{o}}{\hat{f}_{i}}
]
Where f_i_o is the reconstruction focal length and hat_f_i is from the equivalent intrinsics. The complete placement transform is:
[
T_{place}^{i} = \left[ \begin{array}{cc}s_{i}R_{i} & P_{t} – s_{i}R_{i}P_{o}\ \mathbf{0}^{\mathrm{T}} & 1 \end{array} \right]T_{l2c}^{i}
]
Finally, to mitigate floating from discretization errors, the system jointly searches anchor depth and isotropic scale along the terrain ray, terminating when the contact ratio between bottom object voxels and terrain reaches a threshold.
3.3 Scene Refinement
Even after placement, you’ll find inconsistent scales, poor geometry, or floating objects. The Scene Refinement Agent uses an executable interface (MCP) to connect to Blender.
Object Refinement: The agent evaluates the pose, mesh quality, and scale of pending objects. For poor SAM3D reconstructions, it conditions Hunyuan3D on both the coarse mesh (structural constraint) and the object-centric image (appearance cues) to improve geometry and textures.
Terrain Refinement: The agent checks for floating, excessive penetration, or unstable support. When detected, it performs object-terrain co-deformation—repositioning the object or locally displacing/flattening the terrain within the support region to conform to the object footprint. All edits are verified through re-rendering and diagnostics.
4. Key Observations from the Experiments
WorldClaw uses Claude Opus 4.8 as the underlying agent model, leveraging GPT-Image-2, SAM3, SAM3D, and Hunyuan3D. Experiments run on 4 NVIDIA H20 GPUs, with Blender 5.1.1 handling rendering and construction.
The system handles diverse prompts: tropical pirate islands, river canyons with tribal settlements, desert battlefields, and snow-covered mountain valleys. The global orbit views show significant elevation variation and clear regional semantics. Local walk views reveal dense, context-appropriate content—settlements, defensive structures, vehicles, and vegetation—distributed according to terrain function.
In qualitative comparisons with SynCity, Marble, MajutsuCity, WorldGen, and GPT-5.6 Sol:
-
Terrain & Organization: Alternatives often produce flat ground, weaker long-range organization, or coarse regional transitions. WorldClaw uses semantic layout maps to construct continuous terrain with pronounced elevation variation and clear spatial boundaries. -
Content Richness: Competing methods often repeat structures or stick to limited categories. WorldClaw selectively introduces settlements, vegetation, animals, and environmental objects across regions, preserving open areas where needed. -
Editability: SynCity and Marble output scene-level representations without independent object assets. WorldClaw outputs independent, controllable textured meshes, suitable for game-engine workflows.
5. Limitations and Future Directions
Heavy reliance on foundation models. The decoupled pipeline pushes planning, image generation, and 3D reconstruction to their limits. Current open-source language models often struggle with procedural terrain generation; open-source image models frequently fail to preserve object appearance during extraction. Validating the pipeline currently requires capable commercial models like Claude Opus 4.8 and Hunyuan3D.
Stability risks in code generation. Several stages rely on LLM-generated programs (terrain construction, material graphs, asset placement). Errors in scale estimation or node connectivity directly manifest as inconsistent landforms or inaccurate materials. Blender’s complex node-based workflows remain challenging for current language models, often leading to simplified material approximations.
Efficiency overhead. Generating and reconstructing each object individually, plus multiple refinement rounds, incurs substantial latency and computational cost. While this provides fine-grained control for complex scenes, it’s overkill for simpler environments that holistic methods can handle in fewer steps.
Looking ahead: Code-native 3D modeling offers a promising complement. WorldClaw already explores this for terrain materials via Blender node graphs and shader scripts. These are explicitly parameterized, readily editable, and achieve decent results. Future work could extend this to object geometry, replacing fixed meshes with executable programs that encode composition, logic, and motion constraints. Integrating with game engines like Unreal Engine could also bridge the gap to runtime systems for navigation, physics, and interaction.
Quick Operations Checklist
-
Input: User provides a single open-ended text prompt. -
Intent Analysis: Intent agent extracts explicit constraints. -
Scene Planning: Planning agent completes missing info → outputs structured scene spec P. -
Terrain Planning: Terrain agent generates P_terrain(layout, assets, materials, parameters). -
Terrain Asset Generation: Generates layout map I_layout, asset prototypesO_asset, and materials. -
Terrain Construction: Builds height-field H(x)→ scatters terrain assets → refinement loop via BlenderMCP. -
Regional Planning: Selects regions R+for object populating → generatesP_regional. -
Region Composition: Renders terrain I_r_terrain→ generates composition imageI_r_comp. -
Object Generation: SAM3 segments instances → SAM3D reconstructs + scale calibration. -
Object Placement: Ray-casting to recover 3D position + scale + depth search to fix floating. -
Scene Refinement: Object quality check (re-run Hunyuan3D if needed) → Terrain contact check (co-deformation). -
Output: Independent textured meshes + terrain, ready for free-view rendering and engine import.
FAQ
Q: What format is the final output? Can I export a single building?
A: The output consists of explicit terrain meshes and independent instance-level textured meshes. Each object retains its geometry, materials, and placement transform. You can export, edit, or reuse them individually—they aren’t merged into a single unbreakable asset.
Q: How is this different from standard procedural generation in Blender?
A: WorldClaw uses agents to make decisions—they plan the layout, choose material parameters, decide where to place objects, and detect/fix floating. Traditional procedural generation requires manual rule-writing and parameter tuning. Internally, WorldClaw still uses procedural methods for heightfields and materials, but the logic driving them is agentic.
Q: How accurate is SAM3D single-image reconstruction?
A: It has inherent scale ambiguity. WorldClaw compensates with image-space bounding-box scale calibration and ray-based terrain anchoring. If quality is still insufficient, the refinement stage re-generates the geometry with Hunyuan3D using both the coarse mesh and original image as conditions.
Q: How does the system handle floating objects?
A: After initial placement, it runs a joint depth and scale search along the terrain ray to maximize object-terrain contact. If issues persist, the terrain refinement agent performs localized co-deformation to adjust the surface below the object.
Q: How long does it take to generate a scene?
A: It depends heavily on object count and refinement iterations. Since each object is reconstructed individually and multiple refinement loops run, complex scenes incur significant overhead. The paper explicitly lists “efficiency overhead” as a major limitation.
Q: Do I absolutely need Claude Opus 4.8? Can I use open-source models?
A: The paper states that current open-source language models frequently fail at procedural terrain and material generation. Open-source image models also struggle with layout maps and object appearance consistency. For full quality, commercial models are currently necessary. Using open-source alternatives would likely reduce quality and executability.
Q: Can the generated scenes be used in Unreal Engine?
A: The output is standard explicit meshes and textures, so they can be imported. However, the current pipeline is built around Blender’s scripting interface. Integration with UE’s runtime procedural generation, physics, and navigation systems is listed as a future direction.
