Hongfei Zhang 1*, Kanghao Chen 1,5*, Zixin Zhang 1,5, Harold H. Chen 1,5, Yuanhuiyi Lyu 1, Yuqi Zhang 3, Shuai Yang 1, Kun Zhou 4, Ying-Cong Chen 1,2,✉
1 HKUST(GZ) 2 HKUST 3 Fudan University 4 Shenzhen University 5 Knowin
* Equal Contribution. ✉Corresponding author.
open-world.mp4
✅ 2025.12 — Released training code (Paper setting) 🚀
✅ 2025.11 — Adding Gradio Demo ✨
✅ 2025.11 — Released inference pipeline & demo dataset ✔️
✅ 2025.11 — Uploaded official DualCamCtrl checkpoints to HuggingFace 🔑
⬜ Release the training code (For lora fine-tuning) 🚀
This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the SemantIc Guided Mutual Alignment (SIGMA) mechanism, which performs RGB–depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation with over 40% reduction on camera motion errors compared with prior methods.
Comparison between our method and other state-of-the-art approaches. Given the same camera pose and input image as generation conditions, our method achieves the best alignment between camera motion and scene dynamics, producing the most visually accurate video. The ’+’ signs marked in the figure serve as anchors for visual comparison.
Quantitative comparisons on I2V setting. ↑ / ↓ denotes higher/lower is better. Best and second best results highlighted.
Quantitative comparisons on T2V setting across REALESTATE10K and DL3DV.
git clone https://github.com/soyouthinkyoucantell/DualCamCtrl.git
conda create -n dualcamctrl python=3.11 -y
conda activate dualcamctrl
cd DualCamCtrl
pip install -e .
mkdir dependency
cd dependency
git clone https://github.com/rmbrualla/pycolmap.git
cd pycolmap
pip install -e .
pip install numpy==1.26.4 peft accelerate==1.9.0 decord==0.6.0 deepspeed diffusers omegaconf
We utilize Video Depth Anything (VDA) to predict both image depth and video depth. For inference, only image depth is needed, so you could run VDA with a single image input.
Get the checkpoints from the HuggingFace repo: DualCamCtrl Checkpoints and put it the checkpoints dir (You could just simply download the HF repo which contains a dir named 'checkpoints'.)
Your project structure should be like:
DualCamCtrl/
├── checkpoints/ # ← Put downloaded weight file here
│ └── dualcamctrl_diffusion_transformer.pt
├── demo_dataset/ # Small demo dataset strcture
├── demo_pic/ # Demo images for quick inference
├── diffsynth/
├── examples/
├── ....
├── requirements.txt
├── README.md
└── setup.py
cd .. # make sure you are at the root dir
export PYTHONPATH=.
python -m test_script.test_demo
Install gradio dependency (needs large memory GPU)
pip install gradio
Run app
export PYTHONPATH=.
python gradio/app.py # For Large Memory GPU
Please refer to this document for training.
If you find our work useful, please consider cite our work:
@article{zhang2025dualcamctrl,
title={DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation},
author={Zhang, Hongfei and Chen, Kanghao and Zhang, Zixin and Chen, Harold Haodong and Lyu, Yuanhuiyi and Zhang, Yuqi and Yang, Shuai and Zhou, Kun and Chen, Yingcong},
journal={arXiv preprint arXiv:2511.23127},
year={2025}
}