ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models

web

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: GitHub

ControlNet is relevant to AI safety discussions around misuse of image generation capabilities, including deepfakes and disinformation, though the repository itself is a technical tool rather than a safety or policy resource.

Metadata

Importance: 42/100tool pagetool

Summary

ControlNet is an open-source neural network architecture that adds fine-grained spatial control to large pretrained text-to-image diffusion models like Stable Diffusion. It enables users to condition image generation on inputs such as edge maps, depth maps, pose skeletons, and segmentation maps. The repository provides the official implementation and has become a widely used tool in AI image generation research and applications.

Key Points

•Introduces a trainable copy of diffusion model encoder blocks that accepts additional conditioning inputs while preserving the original model weights.
•Supports diverse spatial conditioning signals including Canny edges, depth maps, human pose (OpenPose), segmentation masks, and more.
•Enables precise compositional control over generated images without retraining the base diffusion model from scratch.
•Widely adopted in creative AI workflows and downstream research, demonstrating significant capability advancement in controllable generation.
•Open-source repository with pretrained models, training code, and inference scripts, lowering barrier to reproducing and extending results.

Cited by 1 page

Page	Type	Quality
AI Disinformation	Risk	54.0

Cached Content Preview

HTTP 200Fetched Apr 9, 202616 KB

GitHub - lllyasviel/ControlNet: Let us control diffusion models! · GitHub 

 
 
 
 

 
 

 

 

 
 

 
 

 

 

 

 

 

 

 

 

 

 

 
 
 

 
 
 

 

 

 
 
 
 

 

 

 

 

 

 

 
 

 

 

 
 

 
 
 

 
 

 

 

 
 
 
 

 
 Skip to content 

 
 
 
 
 
 

 
 
 
 
 

 

 

 

 
 
 
 
 
 You signed in with another tab or window. Reload to refresh your session. 
 You signed out in another tab or window. Reload to refresh your session. 
 You switched accounts on another tab or window. Reload to refresh your session. 

 
 
 
 Dismiss alert 

 
 
 

 

 

 
 
 
 
 
 
 
 
 
 
 
 {{ message }} 

 
 
 
 
 

 

 
 
 
 
 

 

 

 

 
 
 
 
 
 
 
 
 
 lllyasviel
 
 / 
 
 ControlNet 
 

 Public 
 

 

 
 
 
 

 
 
 
 Notifications
 You must be signed in to change notification settings 

 

 
 
 
 Fork
 3k 
 
 

 
 
 
 
 
 Star
 33.8k 
 
 

 

 
 

 
 

 

 
 

 
 
 

 
 
 

 
 
 
   main Branches Tags Go to file Code Open more actions menu Folders and files

 Name Name Last commit message Last commit date Latest commit

   History

 188 Commits 188 Commits annotator annotator     cldm cldm     docs docs     font font     github_page github_page     ldm ldm     models models     test_imgs test_imgs     .gitignore .gitignore     LICENSE LICENSE     README.md README.md     config.py config.py     environment.yaml environment.yaml     gradio_annotator.py gradio_annotator.py     gradio_canny2image.py gradio_canny2image.py     gradio_depth2image.py gradio_depth2image.py     gradio_fake_scribble2image.py gradio_fake_scribble2image.py     gradio_hed2image.py gradio_hed2image.py     gradio_hough2image.py gradio_hough2image.py     gradio_normal2image.py gradio_normal2image.py     gradio_pose2image.py gradio_pose2image.py     gradio_scribble2image.py gradio_scribble2image.py     gradio_scribble2image_interactive.py gradio_scribble2image_interactive.py     gradio_seg2image.py gradio_seg2image.py     share.py share.py     tool_add_control.py tool_add_control.py     tool_add_control_sd21.py tool_add_control_sd21.py     tool_transfer_control.py tool_transfer_control.py     tutorial_dataset.py tutorial_dataset.py     tutorial_dataset_test.py tutorial_dataset_test.py     tutorial_train.py tutorial_train.py     tutorial_train_sd21.py tutorial_train_sd21.py     View all files Repository files navigation

 News: A nightly version of ControlNet 1.1 is released!

 
 ControlNet 1.1 is released. Those new models will be merged to this repo after we make sure that everything is good.

 Below is ControlNet 1.0

 
 Official implementation of Adding Conditional Control to Text-to-Image Diffusion Models .

 ControlNet is a neural network structure to control diffusion models by adding extra conditions.

 

 It copys the weights of neural network blocks into a "locked" copy and a "trainable" copy.

 The "trainable" one learns your condition. The "locked" one preserves your model.

 Thanks to this, training with small dataset of image pairs will not destroy the production-ready diffusi

... (truncated, 16 KB total)

Resource ID: 8e6dfe3346e322e8 | Stable ID: sid_eJXH3sUw7Y