DimensionX

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion

DimensionX is a novel framework that enables the creation of 3D and 4D scenes from a single input image. It leverages controllable video diffusion techniques to generate dynamic scenes, offering control over both spatial and temporal aspects. This technology is particularly useful for generating novel view videos and fusing spatial-temporal controls.

What is DimensionX?

DimensionX is a framework designed to produce 3D and 4D scenes from a single image. It stands out due to its ability to create controllable video diffusion, allowing users to manipulate the spatial and temporal elements within the generated scene.

How does DimensionX work?

The DimensionX pipeline is divided into three main parts:

1. ST-Director for Controllable Video Generation: This component decomposes spatial and temporal parameters in video diffusion models. It learns dimension-aware LoRA (Low-Rank Adaptation) on dimension-variant datasets to achieve controllable video generation.
2. 3D Scene Generation with S-Director: Given a single view, a high-quality 3D scene is recovered from the video frames generated by S-Director.
3. 4D Scene Generation with ST-Director: Starting with a single image, a temporal-variant video is produced by T-Director. A key frame is selected from this video to generate a spatial-variant reference video. Guided by the reference video, per-frame spatial-variant videos are generated by S-Director, which are then combined into multi-view videos. The multi-loop refinement of T-Director ensures consistent multi-view videos, which are then used to optimize the 4D scene.

Key Features and Components:

• ST-Director: Decomposes spatial and temporal parameters using dimension-aware LoRA.
• S-Director: Generates high-quality 3D scenes from video frames.
• T-Director: Produces temporal-variant videos from a single image.

Example Use Cases:

• Any Camera Control Video Generation: Demonstrates the ability to control the camera in the generated video, including static, orbit right, orbit left, and zoom in motions.
• Spatial-Temporal Fused Controllable Video Generation: Shows the framework's capability to fuse spatial and temporal controls for video generation.
• Single View 3D Generation: Generates 3D scenes from a single input view, allowing for 360-degree orbits.
• Sparse View 3D Scene Generation: Creates 3D scenes from two input views.
• 4D Scene Generation: Generates dynamic 4D scenes with novel view videos.

Why choose DimensionX?

DimensionX offers a unique approach to 3D and 4D scene generation by providing:

• Controllability: Users have precise control over the spatial and temporal aspects of the generated scenes.
• High Quality: The framework generates high-quality 3D and 4D scenes from a single image.
• Versatility: It supports various applications, including camera control, spatial-temporal fusion, and novel view generation.

Who is DimensionX for?

DimensionX is suitable for:

• Researchers in computer vision and graphics.
• Content creators looking to generate dynamic 3D and 4D scenes.
• Developers working on applications that require controllable video generation.

DimensionX builds upon the Clarity Template, further enhancing its capabilities. The DimensionX project also introduces the "X Family," which includes ReconX for reconstructing scenes from sparse views, with more additions planned for the future.

Citation

@article{sun2024dimensionx,
    title={DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion},
    author={Sun, Wenqiang and Chen, Shuo and Liu, Fangfu and Chen, Zilong and Duan, Yueqi and Zhang, Jun and Wang, Yikai},
    journal={arXiv preprint arXiv:2411.04928},
    year={2024}
}

DimensionX empowers users to create stunning 3D and 4D scenes from a single image, making it a valuable tool for various applications in research and content creation. It uses innovative techniques and provides fine-grained control over the generated content, allowing for highly customized and visually appealing results.