Skip to content

Commit 090dc1c

Browse files
jianzongwujianzongwu
committed
12-4
1 parent e0b922f commit 090dc1c

17 files changed

Lines changed: 3143 additions & 2 deletions

‎.gitignore‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# Initially taken from GitHub's Python gitignore file
2+
3+
# Byte-compiled / optimized / DLL files
4+
__pycache__/
5+
*.py[cod]
6+
*$py.class
7+
8+
# C extensions
9+
*.so
10+
11+
# tests and logs
12+
tests/fixtures/cached_*_text.txt
13+
logs/
14+
logs
15+
lightning_logs/
16+
lang_code_data/
17+
18+
# Distribution / packaging
19+
.Python
20+
build/
21+
develop-eggs/
22+
dist/
23+
downloads/
24+
eggs/
25+
.eggs/
26+
lib/
27+
lib64/
28+
parts/
29+
sdist/
30+
var/
31+
wheels/
32+
*.egg-info/
33+
.installed.cfg
34+
*.egg
35+
MANIFEST
36+
37+
# PyInstaller
38+
# Usually these files are written by a Python script from a template
39+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
40+
*.manifest
41+
*.spec
42+
43+
# Installer logs
44+
pip-log.txt
45+
pip-delete-this-directory.txt
46+
47+
# Unit test / coverage reports
48+
htmlcov/
49+
.tox/
50+
.nox/
51+
.coverage
52+
.coverage.*
53+
.cache
54+
nosetests.xml
55+
coverage.xml
56+
*.cover
57+
.hypothesis/
58+
.pytest_cache/
59+
60+
# Translations
61+
*.mo
62+
*.pot
63+
64+
# Django stuff:
65+
*.log
66+
local_settings.py
67+
db.sqlite3
68+
69+
# Flask stuff:
70+
instance/
71+
.webassets-cache
72+
73+
# Scrapy stuff:
74+
.scrapy
75+
76+
# Sphinx documentation
77+
docs/_build/
78+
79+
# PyBuilder
80+
target/
81+
82+
# Jupyter Notebook
83+
.ipynb_checkpoints
84+
85+
# IPython
86+
profile_default/
87+
ipython_config.py
88+
89+
# pyenv
90+
.python-version
91+
92+
# celery beat schedule file
93+
celerybeat-schedule
94+
95+
# SageMath parsed files
96+
*.sage.py
97+
98+
# Environments
99+
.env
100+
.venv
101+
env/
102+
venv/
103+
ENV/
104+
env.bak/
105+
venv.bak/
106+
107+
# Spyder project settings
108+
.spyderproject
109+
.spyproject
110+
111+
# Rope project settings
112+
.ropeproject
113+
114+
# mkdocs documentation
115+
/site
116+
117+
# mypy
118+
.mypy_cache/
119+
.dmypy.json
120+
dmypy.json
121+
122+
# Pyre type checker
123+
.pyre/
124+
125+
# vscode
126+
.vs
127+
.vscode
128+
129+
# Pycharm
130+
.idea
131+
132+
# TF code
133+
tensorflow_code
134+
135+
# Models
136+
proc_data
137+
138+
# examples
139+
runs
140+
/runs_old
141+
/wandb
142+
/examples/runs
143+
/examples/**/*.args
144+
/examples/rag/sweep
145+
146+
# data
147+
serialization_dir
148+
149+
# emacs
150+
*.*~
151+
debug.env
152+
153+
# vim
154+
.*.swp
155+
156+
# ctags
157+
tags
158+
159+
# pre-commit
160+
.pre-commit*
161+
162+
# .lock
163+
*.lock
164+
165+
# DS_Store (MacOS)
166+
.DS_Store
167+
168+
# RL pipelines may produce mp4 outputs
169+
*.mp4
170+
171+
# dependencies
172+
/transformers
173+
174+
# ruff
175+
.ruff_cache
176+
177+
# wandb
178+
wandb
179+
180+
181+
# ours
182+
nohup*
183+
checkpoints
184+
results*
185+
outputs*
186+
data

‎README.md‎

Lines changed: 165 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,165 @@
1-
# DiffSensei
2-
Implementation of "DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation"
1+
<br />
2+
<p align="center">
3+
<h1 align="center">MotionBooth: Motion-Aware Customized <br> Text-to-Video Generation</h1>
4+
<p align="center">
5+
<br />
6+
<a href="https://jianzongwu.github.io/"><strong>Jianzong Wu</strong></a>
7+
·
8+
<a href="https://lxtgh.github.io/"><strong>Xiangtai Li</strong></a>
9+
·
10+
<a href="https://zengyh1900.github.io/"><strong>Yanhong Zeng</strong></a>
11+
·
12+
<a href="https://zhangzjn.github.io/"><strong>Jiangning Zhang</strong></a>
13+
.
14+
<a href="https://qianyuzqy.github.io/"><strong>Qianyu Zhou</strong></a>
15+
.
16+
<a href="https://github.com/ly015"><strong>Yining Li</strong></a>
17+
·
18+
<a href="https://scholar.google.com/citations?user=T4gqdPkAAAAJ"><strong>Yunhai Tong</strong></a>
19+
.
20+
<a href="https://chenkai.site/"><strong>Kai Chen</strong></a>
21+
</p>
22+
23+
<p align="center">
24+
<a href='https://arxiv.org/abs/2406.17758'>
25+
<img src='https://img.shields.io/badge/Paper-PDF-green?style=flat&logo=arXiv&logoColor=green' alt='arXiv PDF'>
26+
</a>
27+
<a href='https://github.com/jianzongwu/MotionBooth'>
28+
<img src='https://img.shields.io/badge/Github-Code-blue?style=flat&logo=Github' alt='Code'>
29+
</a>
30+
<a href='https://jianzongwu.github.io/projects/motionbooth'>
31+
<img src='https://img.shields.io/badge/Project-Page-blue?style=flat&logo=webpack' alt='Project Page'>
32+
</a>
33+
</p>
34+
<br />
35+
36+
## Examples
37+
38+
**Customization and subject motion control**
39+
40+
<img src="assets/demo1.gif" width="600">
41+
42+
**Hybrid control on customization, subject and camera motion**
43+
44+
<img src="assets/demo2.gif" width="600">
45+
46+
47+
## 🎉 News
48+
49+
- [2024-6-28] Inference code, training code, and checkpoints are released!
50+
51+
## 📖 TL;DR
52+
53+
DiffSensei can generate vivid black-and-white manga panels with precise character and dialog layout control!
54+
55+
## 🛠️ Quick Start
56+
57+
### Installation
58+
59+
``` bash
60+
# Create a new environment with Conda
61+
conda create -n diffsensei python=3.11
62+
conda activate diffsensei
63+
# Install Pytorch and Diffusers related packages
64+
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
65+
conda install -c conda-forge diffusers transformers accelerate
66+
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
67+
# Install other dependencies
68+
pip install -r requirements.txt
69+
# Third-party repo for running the gradio demo
70+
pip install gradio-image-prompter
71+
```
72+
73+
74+
### Pre-trained Model Preparation
75+
76+
We use [Zeroscope](https://huggingface.co/cerspense/zeroscope_v2_576w) and [LaVie-base](https://huggingface.co/Vchitect/LaVie) for the base T2V models. Please download Zeroscope from the [official huggingface page](https://huggingface.co/cerspense/zeroscope_v2_576w). For LaVie, we provide a script to convert their original checkpoint into the format that is suitable for Diffusers. Please download the [LaVie-base](https://huggingface.co/Vchitect/LaVie) model and the [Stable-Diffusion-v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4) checkpoint.
77+
78+
Then, organize the pre-trained models in the `checkpoints` folder.
79+
80+
```
81+
checkpoints
82+
|- zeroscope_v2_576w
83+
|- stable-diffusion-v1-4
84+
|- lavie_base.pt
85+
```
86+
87+
Then, run the following command to convert the checkpoint
88+
89+
``` bash
90+
python -m scripts.convert_ckpts.convert_lavie_to_diffusers
91+
```
92+
93+
Then, rename the `stable-diffusion-v1-4` folder to `lavie`. Additionally, you should replace the config file to LaVie's configs, following [checkpoint guide](docs/checkpoints.md).
94+
95+
The final checkpoint folder looks like this:
96+
97+
```
98+
checkpoints
99+
|- zeroscope_v2_576w
100+
|- lavie
101+
|- lavie_base.pt (Not used anymore)
102+
```
103+
104+
We use the converted lavie model for all the experiments.
105+
106+
107+
### Inference with Gradio
108+
109+
For quick inference and re-producing the examples in paper, please download our trained customized checkpoints for the target subjects in [huggingface](https://huggingface.co/jianzongwu/MotionBooth). The names of the checkpoints correspond to the subject names in the MotionBooth dataset.
110+
111+
Please place the checkpoints in che `checkpoints` folder like this:
112+
113+
```
114+
checkpoints
115+
|- customized
116+
|- zeroscope
117+
|- ...
118+
|- lavie
119+
|- ...
120+
|- zeroscope_v2_576w
121+
|- lavie
122+
|- lavie_base.pt (Not used anymore)
123+
```
124+
125+
We use simple script files to indicate the subject and camera motion. We provide several examples in `data/scripts`. In these script files, the "bbox" controls the bounding box sequence for the subjects' motion, while the "camera speed" controls the corresponding camera motion speed.
126+
127+
We provide the inference script in `scripts/inference.py` for all types of MotionBooth applications. It uses [Accelerate PartialState](https://huggingface.co/docs/accelerate/index) to support multi GPU inference.
128+
129+
### MangaZero Data Download
130+
131+
We collect 26 objects from [DreamBooth](https://dreambooth.github.io/) and [CustomDiffusion](https://github.com/adobe-research/custom-diffusion) to perform the experiments in paper. These objects include pets, plushies, toys, cartoons, and vehicles. We also annotate masks for each image. We name it the MotionBooth dataset. Please download our dataset from [huggingface](https://huggingface.co/datasets/jianzongwu/MotionBooth).
132+
133+
Note that a few images from the original datasets are deleted because the low quality of the obtained masks. Additionally, a few images are resized and cropped to square shapes.
134+
135+
After downloading, please unzip and place the dataset under the `data` folder. It should look like this:
136+
137+
```
138+
data
139+
|- MotionBooth
140+
|- images
141+
|- cat2
142+
|- ...
143+
|- masks
144+
|- cat2
145+
|- ...
146+
|- scripts
147+
```
148+
149+
150+
151+
152+
## 📢 Disclaimer
153+
154+
Please try more samples under different random seeds to have better results. 🤗
155+
156+
## Citation
157+
158+
```
159+
article{wu2024motionbooth,
160+
title={MotionBooth: Motion-Aware Customized Text-to-Video Generation},
161+
author={Jianzong Wu and Xiangtai Li and Yanhong Zeng and Jiangning Zhang and Qianyu Zhou and Yining Li and Yunhai Tong and Kai Chen},
162+
journal={NeurIPS},
163+
year={2024},
164+
}
165+
```

‎configs/inference/diffsensei.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
num_inference_steps: 30
2+
guidance_scale: 7.5
3+
negative_prompt: "think lines, pure black background, colored, lowres, bad anatomy, worst quality, low quality"
4+
num_samples: 1
5+
ip_scale: 0.6
6+
mllm_scale: 0.4

‎configs/model/diffsensei.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
image_generator:
2+
max_num_ips: 4
3+
num_vision_tokens: 16
4+
num_dummy_tokens: 16
5+
max_num_dialogs: 8
6+
7+
resampler:
8+
dim: 1280
9+
depth: 4
10+
dim_head: 64
11+
heads: 20
12+
ff_mult: 4
13+
14+
agent:
15+
input_resampler:
16+
grid_size: 8
17+
embed_dim: 5120
18+
num_heads: 32
19+
kv_dim: 2048
20+
output_resampler:
21+
grid_size: 8
22+
embed_dim: 2048
23+
num_heads: 32
24+
kv_dim: 5120
25+
mse: True
26+
lm_loss_scale: 1.0
27+
rec_loss_scale: 6.0

‎jobs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../../../../../data00/wjz/checkpoints/diffusion/jobs

0 commit comments

Comments
 (0)