TelePhysica: Agentic Reconstruction of Physically Interactive Scenes

Turning Any Visual Content into an Interactive Physical World

Ceramic cup in flightVACE A14B · RGB guidance

From image to interaction

Choose a scene to inspect the photograph, the applied action, and the resulting motion.

Input image
Input image of a yellow cup, calculator, pencil, and eraser
Action on the scene
Rendered response 0.00 / 5.00 s

Tumble & land

A force sends the cup upward, into a tumble, and back to the tabletop.

Abstract

Direct image-to-video diffusion models entangle pixel synthesis with implicit latent motion, precluding persistent 3D mechanical equilibrium, mid-rollout intervention, and counterfactual force control. Conversely, single-pass programmatic 3D reconstruction from an uncalibrated photograph leaves monocular depth and support clearance underconstrained, causing reconstructed scenes to interpenetrate or collapse under gravity. We present TelePhysica, a closed-loop framework that reconstructs an executable, multi-regime physical world from a single photograph to drive stateful 3D interaction and trajectory-grounded video synthesis. A vision-language coding agent synthesizes separable visual meshes and declarative physical models within a two-stage verification loop that pairs pixel-aligned silhouette critique with a zero-force gravitational equilibrium preflight. The verified world compiles into a unified solver coupling rigid bodies, articulated chains, co-dimensional cloth shells, and 3D tetrahedral elastodynamics, whose simulated rollouts condition pretrained video diffusion models through pixel-aligned structural streams. On a 100-scene single-image benchmark, closed-loop execution raises equilibrium-verified world recovery from 0% in single-pass synthesis to 81%. Controlled constitutive ablations show over an order-of-magnitude larger localized corner displacement and out-of-plane pleating in finite-element cloth than in matched rigid-plate proxies. The completed 60-configuration, 480-video scale-up compares eight generation arms across Wan2.1-VACE-1.3B, Wan2.2-VACE-Fun-A14B and MiniMax-H3. On 53 matched core scenes, trajectory conditioning improves motion-mask agreement over the corresponding direct image-to-video baselines. The full W49 benchmark contains 49 rigid scenes; two counterfactuals, two photo-aligned constitutive variants and three legacy cases are evaluated separately.

Reconstructing a scene that holds together

A vision-language coding agent synthesizes visual geometry and declarative physical models inside a two-stage loop combining pixel silhouette critique with a zero-force gravitational equilibrium gate.

Two-stage verification

Of 100 tabletop reconstructions, 85 passed visual alignment, 81 passed equilibrium verification, and 49 completed a five-second force rollout. On the first submission, two passed visual alignment and none passed equilibrium verification.

Unified multi-regime physics

Evolve 6-DoF rigid bodies, 164-link cables, co-dimensional cloth shells, and 3D tetrahedral elastic solids from checkpointable states.

Trajectory-grounded video

Drive Wan2.1/2.2 VACE (1.3B & A14B) and MiniMax-H3 with simulated edge or RGB rollouts for counterfactual-consistent video synthesis.

TelePhysica closed-loop architecture and multi-regime rigid, articulated, thin-shell cloth, and 3D solid interactions
Framework & multi-regime overview. Closed-loop single-image world recovery, four-regime stateful mechanics, and phase-aligned trajectory-grounded video synthesis. Open full-size figure ↗
Closed-loop diagnostic traces on hybrid scenes and 100-scene open-loop vs closed-loop audit
Closed-loop vs. open-loop audit. Stage-1 silhouette refinement and Stage-2 equilibrium preflight catch occluded support gaps (68.3 mm nodal drop) and lift verified recovery from 0/100 to 81/100 scenes. Open full-size figure ↗
Input and simulated states for rigid objects, a lifted earphone cable, two textiles, a seat cushion, and sponges pushing a bottle
From photograph to physical response. Rigid objects slide and collide, a prescribed earbud lift raises a 164-link cable, and localized loads fold a towel, peel a glove, indent a cushion, and push a bottle. Open full-size figure ↗

How materials respond to force

Explicit cloth shells, layered textiles, 164-link cables, and 3D tetrahedral foams capture localized buckling, elastic rebound, and soft-to-rigid momentum transfer.

Constitutive proxy ablation on microfiber towel, layered glove peel, cable lift, bowling state continuation, 3D cushion rebound, and sponge-to-bottle sliding
How material choice changes the response. Under the same corner load, the towel corner travels 771.9 mm in the cloth model and 72.1 mm in the matched rigid plate. The cloth develops an 80.4 mm pleat. The seat cushion rebounds to within 0.84 mm of its initial height, and the compressed sponges move the bottle 76.7 mm. Open full-size figure ↗

Scene explorer

Yellow cup

Original input image
1 · Input image
Recorded force on the reconstructed scene
2 · Applied force
3 · Physics rollout
4 · MiniMax-H3 final video
0.00 / 5.00 s

Shared five-second playback · H3 audio is opt-in.