🏞️ Layout Generation — Interactive 3D Scenes

Layout Generation enables the generation of diverse, physically realistic, and scalable interactive 3D scenes directly from natural language task descriptions, while also modeling the robot's pose and relationships with manipulable objects. Target objects are randomly placed within the robot's reachable range, making the scenes readily usable for downstream simulation and reinforcement learning tasks in any mainstream simulator.

Model Requirement

The text-to-image model is based on SD3.5 Medium. Usage requires agreement to the model license.

---

Prerequisites — Prepare Background 3D Scenes

Before running layout-cli, you need to prepare background 3D scenes. You can either generate your own using the scene3d-cli, or download pre-generated backgrounds for convenience.

Each scene takes approximately 30 minutes to generate. For efficiency, we recommend pre-generating and listing them in outputs/bg_scenes/scene_list.txt.

# Option 1: Download pre-generated backgrounds (~4 GB)
hf download xinjjj/scene3d-bg --repo-type dataset --local-dir outputs

# Option 2: Download a larger background set (~14 GB)
hf download xinjjj..RLv2-BG --repo-type dataset --local-dir outputs

Generate Interactive Layout Scenes

Use the layout-cli to create interactive 3D scenes based on task descriptions. Each layout generation takes approximately 30 minutes.

layout-cli \
  --task_descs "Place the pen in the mug on the desk" \
               "Put the fruit on the table on the plate" \
  --bg_list "outputs/bg_scenes/scene_list.txt" \
  --output_root "outputs/layouts_gen" \
  --insert_robot

You will get the following results:

Batch Generation

You can also run multiple tasks via a task list file in the backend.

CUDA_VISIBLE_DEVICES=0 nohup layout-cli \
  --task_descs "apps/assets/example_layout/task_list.txt" \
  --bg_list "outputs/bg_scenes/scene_list.txt" \
  --output_root "outputs/layouts_gens" \
  --insert_robot > layouts_gens.log &

💡 Remove --insert_robot if you don’t need robot pose consideration in layout generation.

Layout Randomization

Using compose_layout.py, you can recompose the layout of the generated interactive 3D scenes.

python embodied_gen/scripts/compose_layout.py \
--layout_path "outputs/layouts_gens/task_0000/layout.json" \
--output_dir "outputs/layouts_gens/task_0000/recompose" \
--insert_robot

Load Interactive 3D Scenes in Simulators

We provide sim-cli, that allows users to easily load generated layouts into an interactive 3D simulation using the SAPIEN engine.

sim-cli --layout_path "outputs/layouts_gen/task_0000/layout.json" \
--output_dir "outputs/layouts_gen/task_0000/sapien_render" --insert_robot

Recommended Workflow

1. Generate or download background scenes using scene3d-cli.
2. Create interactive layouts from task descriptions using layout-cli.
3. Optionally recompose them using compose_layout.py.
4. Load the final layouts into simulators with sim-cli.