WorldGen Pipeline

WorldGen PipelineπŸ”—

The WorldGen pipeline transforms LocalGPT Gen from β€œthe agent places things one by one” to β€œthe agent designs worlds through a structured interface.” Instead of the agent’s model generating geometry directly, it generates parameters β€” layout specs, density hints, style references β€” and procedural systems handle the actual placement.

This mirrors the architecture described in Meta’s WorldGen paper: language models should generate parameters, not geometry.

Pipeline StagesπŸ”—

1. Plan          gen_plan_layout        Text β†’ BlockoutSpec (JSON)
   ↓
2. Blockout      gen_apply_blockout     BlockoutSpec β†’ terrain + regions + paths
   ↓
3. Navmesh       gen_build_navmesh      Terrain β†’ walkability grid
   ↓
4. Populate      gen_populate_region    Regions β†’ entities (hero/medium/decorative)
   ↓
5. Evaluate      gen_evaluate_scene     Screenshot + highlighting β†’ the agent's feedback
   ↓
6. Refine        gen_auto_refine        Automated evaluate β†’ adjust loop
   ↓
7. Edit          gen_modify_blockout    Non-destructive region changes
   ↓
8. Regenerate    gen_regenerate         Incremental update after edits

Tools (15)πŸ”—

Layout & BlockoutπŸ”—

ToolDescription
gen_plan_layoutGenerate a structured world layout plan from a text description. Returns a BlockoutSpec JSON with terrain type, regions, paths, and density hints.
gen_apply_blockoutApply a blockout spec to create terrain, regions with hero slots, medium density zones, and connecting paths.
gen_modify_blockoutAdd, remove, resize, or move blockout regions with incremental regeneration. Supports non-destructive editing.
gen_regenerateRegenerate regions after blockout changes, preserving manually placed entities.

PlacementπŸ”—

ToolDescription
gen_populate_regionPopulate a region with entities using three-tier placement: hero (landmarks), medium (structures), decorative (props/vegetation). Respects collision clearance and ground snap.
gen_set_tierSet an entity’s placement tier (hero, medium, decorative). Affects clearance and placement priority.
gen_set_roleSet an entity’s semantic role (ground, structure, prop, vegetation, water, sky, light, effect). Enables bulk operations by role.
gen_bulk_modifyModify multiple entities by role or tier. Example: recolor all vegetation, rescale all decorative props.
ToolDescription
gen_build_navmeshBuild a grid-based walkability analysis for the current terrain. Detects steep slopes, obstacles, and blocked areas.
gen_validate_navigabilityCheck that key points (spawn, landmarks, exits) are reachable via A* pathfinding on the navmesh.
gen_edit_navmeshManually override navmesh cells: block areas, allow previously blocked cells, add connections between disconnected regions.

EvaluationπŸ”—

ToolDescription
gen_evaluate_sceneTake a screenshot with optional entity highlighting (red emissive overlay) and camera presets (current, top-down, isometric, front, entity-focus). Annotate with entity names and bounding boxes. For the agent’s own evaluation.
gen_auto_refineAutomated generate β†’ screenshot β†’ evaluate β†’ adjust loop. The agent evaluates its own output and makes corrections.

DepthπŸ”—

ToolDescription
gen_render_depthRender a depth map of the scene from a specified camera angle (top-down, isometric, front, custom). Outputs a grayscale PNG an agent can hand to its own image tools.

Three-Tier PlacementπŸ”—

The population system uses a hierarchy that matches how real environments are built:

TierRoleClearanceExamples
HeroLandmarks, focal pointsLarge (8m+)Town hall, ancient tree, monument
MediumSupporting structuresMedium (3-5m)Houses, wells, market stalls
DecorativeDetail and atmosphereNone/smallFlowers, barrels, rocks, grass

Hero slots are placed first at specific positions defined in the blockout. Medium entities fill density targets. Decorative entities scatter to reach the region’s decorative density.

BlockoutSpec FormatπŸ”—

The gen_plan_layout tool returns a JSON blockout specification:

{
  "terrain": {
    "type": "hills",
    "verticality": 0.6,
    "roughness": 0.4,
    "base_height": 0.0
  },
  "layout": {
    "style": "organic",
    "density": 0.5,
    "regularity": 0.3
  },
  "regions": [
    {
      "id": "village_center",
      "bounds": { "center": [0, 0], "size": [20, 20] },
      "type": "structured",
      "density": 0.7,
      "walkable": true,
      "hero_slots": [
        { "position": [0, 0, 0], "size": [8, 6, 8], "role": "landmark", "hint": "town hall" }
      ],
      "medium_density": 0.5,
      "decorative_density": 0.3
    }
  ],
  "paths": [
    {
      "from": "village_center",
      "to": "forest_edge",
      "style": "cobblestone",
      "width": 2.0
    }
  ]
}

Semantic RolesπŸ”—

Every entity can be tagged with a semantic role for bulk operations:

RoleDescriptionExample Bulk Operation
groundTerrain, floorsRetexture all ground
structureBuildings, wallsScale all structures 1.2x
propFurniture, objectsHide all props
vegetationTrees, bushes, grassRecolor all vegetation to autumn
waterRivers, lakes, oceanAdjust all water transparency
skySky dome, cloudsChange time of day
lightLight sourcesDim all lights by 50%
effectParticles, fogDisable all effects

Scene DecompositionπŸ”—

The pipeline analyzes imported glTF scenes to auto-assign tiers and roles:

  • Mesh segmentation β€” splits a single glTF into connected components
  • Role inference β€” flat/large β†’ ground, tall/narrow β†’ structure, small β†’ decorative
  • Name matching β€” entity names like β€œtree_01” β†’ vegetation, β€œlamp_post” β†’ light

This enables semantic operations on imported scenes, not just procedurally generated ones.