DiffSynth-Studio
DiffSynth-Studio is an open-source Diffusion model engine from the ModelScope Community team. It integrates image, video, audio, and image-quality metric models, with VRAM management and quantization for low-VRAM inference and training.
Visit modelscope/DiffSynth-StudioOverview
DiffSynth-Studio is an open-source Diffusion model engine developed and maintained by the ModelScope Community team. It integrates mainstream open-source Diffusion models covering image generation, video generation, audio generation, and image quality metrics models. The framework is built around model aggregation, VRAM management, parameter quantization, and training support, including split training.
Key Features
- Integrates mainstream open-source Diffusion models across image generation, video generation, audio generation, and image quality metrics models.
- VRAM management dynamically schedules model parameters across disk, memory, and VRAM so consumer-grade GPUs with low VRAM can run large-model inference.
- Parameter quantization converts model parameters to precisions such as NF4 and INT8, reducing VRAM requirements for inference and LoRA training.
- Almost every model that supports inference also supports training, including base models, LoRAs, and Adapter models with additional inputs.
- Split Training uses a computational graph inference engine to track every variable in the Pipeline and splits training into two stages for efficient training.
- Companion projects include DiffSynth-ComfyUI for workflow-based inference and DiffSynth-WebUI for lightweight LoRA training on consumer-grade GPUs.
Use Cases
- Running large Diffusion image, video, or audio models for inference on consumer-grade GPUs with limited VRAM.
- Fine-tuning base models, LoRAs, or Adapter models across supported Diffusion models.
- Building ComfyUI workflows for model inference with DiffSynth-ComfyUI.
- Training LoRAs on consumer-grade GPUs through DiffSynth-WebUI or quantized training workflows.
Getting Started
- Install the package from PyPI with pip install DiffSynth, then consult the developer documentation, starting with the VRAM management, parameter quantization, model training, and split training guides.
Before You Adopt
- License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
- Major version updates have left some old features unmaintained; the source advises using the last historical version before the major update if those features are needed.
- The project states that development capacity is limited, so new feature progress and issue response may be slower.
- CPU Offload Training is currently limited to single-GPU training.
- Training MiniMax-H3-Image is recommended at resolutions larger than 2048×2048.