ControlDreamer: Stylized 3D Generation with Multi-View ControlNet

Venue: BMVC 2024

Authors: Yeongtak Oh*, Jooyoung Choi*, Yongsung Kim, Minjun Park, Chaehun Shin, Sungroh Yoon (* denotes equal contribution)

Project Page | Paper | Demo

Installation

This part is the same as original MVDream-threestudio. Skip it if you already have installed the environment.

Install ControlDreamer

ControlDreamer using multi-view ControlNet is provided in a different codebase. Install it by:

export PYTHONPATH=$PYTHONPATH:./extern/MVDream
export PYTHONPATH=$PYTHONPATH:./extern/ImageDream
pip install -e extern/MVDream 

Further, to provide depth-conditioned MV-ControlNet, download from url or please put midas ckpt file on: ControlDreamer/extern/MVDream/mvdream/annotator/ckpts

Quickstart

Please download the model from MV-ControlNet under ./extern/MVDream/mvdream/ckpt

In the paper, we use the configuration with soft-shading for source generation. An A40 GPU is required, and we recommend setting num_samples_per_ray to 256 (originally 512) to prevent out-of-memory issues in most cases. Additionally, we provide an example source NeRF representation of Hulk, generated from MVDream. If you want to use this, put this file into outputs/source.

To get the source representation:

python launch.py --config configs/mvdream-sd21-shading.yaml \
    --train --gpu 0 \
    system.prompt_processor.prompt="A high-resolution rendering of a Hulk, 3d asset"

After generation, refine the source representation using MV-ControlNet by transforming it into DMTet:

CFG_PATH=configs/controldreamer-sd21-shading.yaml
LOADPATH=outputs/source/Hulk/ckpts/last.ckpt

python launch.py --config ${CFG_PATH} \
    --train --gpu 0 \
    system.prompt_processor.prompt="A high-resolution rendering of an Iron Man, 3d asset" \
    system.geometry_convert_from=${LOADPATH} \
    system.geometry_convert_override.isosurface_threshold=10.

Note: Please refer to our 3D edit prompt bench for creative 3D generation.

Credits

• This code is developed using threestudio, MVDream, and ImageDream.

Citing

If you find ControlDreamer helpful, please consider citing it:

@article{oh2023controldreamer,
  title={ControlDreamer: Stylized 3D Generation with Multi-View ControlNet},
  author={Oh, Yeongtak and Choi, Jooyoung and Kim, Yongsung and Park, Minjun and Shin, Chaehun and Yoon, Sungroh},
  journal={arXiv preprint arXiv:2312.01129},
  year={2023}
}