OmniAI Group of ZJU ACES Lab
Scene-Designer.Learning Design Intelligence知之愈明，则行之愈笃；行之愈笃，则知之益明
* Equal contribution
Repository link coming soon.
01 / OVERVIEW
We define design intelligence as the ability to both design a scene and critique its quality.
Scene design concerns constructing physically valid, functionally correct, and layout-coherent scenes, while efficiently correcting errors without unnecessary trial and error when problems arise.
Scene critique focuses on assessing scene quality along the same dimensions and distinguishing better designs from poorer ones.
These two capabilities are closely connected, since effective design requires reliable critique to guide construction, while meaningful critique requires an understanding of what constitutes good scene design.
02 / MOTIVATION
Two diagnostic studies reveal limitations in both scene design and scene critique.
Across 100 scenes generated by GPT-5.2, even straightforward construction errors often trigger multiple corrective attempts. Recognizing an error is not enough: effective scene design requires the experience to correct it efficiently.
Reliable scene critique must assess physical validity, functional correctness, and layout coherence. General-purpose models can overlook functional relationships and coherent organization when comparing scenes.
03 / THE METHOD
Two kinds of experience build a shared foundation. Scene-GRPO then uses two complementary local rewards to select better design choices and reduce unnecessary trial and error.
Observation · Thought · Action
1,000 scenes · 35,000 interaction stepsObservation · Thought · Critic
≈15,000 scene comparison examplesEffective construction decisions and reliable scene preferences.
Construct physically valid scenes, imitate efficient corrections, and evaluate scene quality.
Rank sibling scenes and assess improvement over the parent.
Compares sibling scenes generated from the same parent, encouraging the policy to favor better design choices.
Compares each child scene with its parent to determine whether the edit actually improves scene quality, discouraging ineffective modifications.
Error-aware candidate identification flags potentially ineffective operations. Correction-effect evaluation preserves effective decisions and necessary intermediate states, removing only ineffective operations without downstream contributions. Replaying the trajectory verifies the same final scene.
Scene-OTC uses mild layout deviations, functional degradation, and severe physical violations. Accumulated perturbations and different strengths on the same object allow automatic ranking. Applying the same perturbation to different objects requires manual annotation.
The branch-ranking reward compares sibling scenes; the improvement reward compares each child scene with its parent. Format, global length, and object-completeness rewards support efficient, complete scene construction. The Critic remains frozen during Scene-GRPO.
DATASET CONSTRUCTION
Scene-OTA-35K captures effective construction decisions, while Scene-OTC-15K provides scene preference comparisons. Together, they teach the model how to design a scene and how to assess its quality.
Scene-OTA construction. Causal Trajectory Consolidation removes ineffective operations without downstream contributions while retaining effective decisions and necessary intermediate states. Replaying the consolidated trajectory verifies that it reaches the same final scene.
Scene-OTC construction. Scene perturbations create preference comparisons through mild layout deviations, functional degradation, and severe physical violations, providing supervision across complementary dimensions of scene quality.
TRAINING STRATEGY
Joint supervised fine-tuning establishes a shared foundation for scene design and critique. Scene-GRPO then expands multiple candidate edits at each step and uses a frozen Critic to guide the policy toward better design decisions.
Choose among alternatives. The branch-ranking reward compares sibling scenes generated from the same parent, encouraging the policy to select higher-quality actions.
Verify actual progress. The improvement reward compares each child scene with its parent, discouraging ineffective modifications. Global length and object-completeness rewards encourage efficient, complete scene construction.
04 / EVALUATION
Qualitative comparisons of generated scenes, and quantitative evaluation of scene preferences.
SCENE DESIGN · QUALITATIVE EVALUATION
Compare generated scenes under the same instructions across seven methods. These examples illustrate differences in physical validity, functional correctness, and layout coherence, from existing approaches to Scene-Builder, Scene-Designer, and Scene-Designer with test-time Critic guidance.
SCENE CRITIQUE · QUANTITATIVE EVALUATION
On the Scene-OTC test sets, evaluators identify the better scene in pairwise comparisons and predict the complete preference order of four scenes. Joint training on design and critique experience achieves the highest accuracy in both tasks.
| Model | Training data | Pairwise comparisonn = 2 · 94 samples | Four-scene rankingn = 4 · 55 samples |
|---|---|---|---|
| Qwen3-VL-8B-InstructBase model | Base | 72% | 2% |
| GPT-5.2General-purpose model | — | 72% | 15% |
| SFT · OTC onlyCritique experience | Scene-OTC | 86% | 51% |
| Scene-Critic OursJoint SFT · Design + critique experience | Scene-OTA + OTC | 89% | 76% |
Results from the paper’s overall preference accuracy comparison. Four-scene accuracy requires the entire ranking to be correct. Bold marks the best result; percentages are rounded as reported in the paper.
Learning design and critique together improves four-scene ranking from 51% to 76% over critique-only training.
05 / FROM PROMPT TO SCENE
Explore ten generated spaces, one prompt at a time.
Each film is paired with its scene prompt.
THE PROMPT
Design a serene master bedroom centered on a queen bed with matching nightstands and reading lamps, a dresser against the opposite wall, and a soft rug underfoot for a calm.
THE PROMPT
Create a family kitchen built for cooking together - plenty of cabinet storage, a range and fridge grouped for an efficient work triangle, and a dining table with chairs for everyday meals.
THE PROMPT
Create a spa-like bathroom centered on a freestanding soaking tub, with a separate shower, a sleek vanity with a large mirror, and calm, minimal finishes.
THE PROMPT
Create a warm family dining room with a sturdy wooden table and six chairs, a buffet for dishes and linens, and soft overhead lighting for everyday dinners.
THE PROMPT
Design a compact contemporary kitchen with a range, refrigerator, and sink arranged efficiently, generous base cabinets, a narrow preparation island, and two stools for casual breakfasts.
THE PROMPT
Create a relaxed conversation-focused living room with a sofa and two armchairs around a coffee table, a media console against one wall, and a floor lamp beside the seating area.
THE PROMPT
Design a focused home office with a broad desk facing the window, an ergonomic chair, closed storage for supplies, and a reading chair with a floor lamp in one quiet corner.
THE PROMPT
Design a compact contemporary kitchen with a range, refrigerator, and sink arranged efficiently, generous base cabinets, a narrow preparation island, and two stools for casual breakfasts.
THE PROMPT
Create a relaxed conversation-focused living room with a sofa and two armchairs around a coffee table, a media console against one wall, and a floor lamp beside the seating area.
THE PROMPT
Design a focused home office with a broad desk facing the window, an ergonomic chair, closed storage for supplies, and a reading chair with a floor lamp in one quiet corner.
06 / CITE THIS WORK
If you find this work useful,
please consider citing our paper.
@misc{liu2026scenedesigner,
title = {Scene-Designer: Learning Design Intelligence
for Agentic 3D Scene Generation},
author = {Xuanyu Liu and Hongyan Feng and Miao Pan
and Sunlai Chen and Lizhou Liang and Bing Sun
and Wenqi Zhang and Xuhong Zhang},
year = {2026},
note = {Research manuscript}
}