INTRODUCTION
With Flux being out for nearly two weeks, it was only a matter of time before we saw the tools and workflows we’ve become accustomed to with Stable Diffusion. Now, we have our first Flux ControlNet model! Today, we’ll explore the Canny ControlNet models and how we can use them with Flux to exercise greater control over the images we generate.
Installing the ControlNet Models
Before we dive into the details, we need to install the new ControlNet models. These can be found on Hugging Face, with two versions available: XLabs-AI and InstantX. While I’ve primarily used the XLabs-AI model, as the InstantX has given me some errors, the installation process for both is identical.
Once downloaded, place the models in the ControlNet folder within your ComfyUI directory. To stay organized, I recommend renaming the files to something recognizable, such as xlabs_ai_controlnet.safe_tensor or instantx_controlnet.safe_tensor.
Setting Up the Workflow
Now that the models are in place, let’s set up the ControlNet workflow in ComfyUI. I’ve prepared a simple workflow with all the necessary components. If you’d like to follow along, you can download the Based Flux ComfyUI Workflow—link in the description.
The main nodes we’ll be working with today are:
- Load ControlNet Model
- Apply ControlNet
- Canny
- Flux Guidance
Note: These nodes are available in the Based ComfyUI setup, so if you’ve installed that, you should already have them.
Let’s go through the process of connecting them.
- Apply ControlNet: Connect this to Flux Guidance so that whatever ControlNet model we load can influence the images we generate.
- Load ControlNet Model: Plug this into the ControlNet node.
- Canny: Feed the image output from Canny into the Apply ControlNet node.
- Source Image: Connect the source image to Canny so we can create our outline.
- Preview Image: Optionally, connect the Canny image to a preview node to visualize the outline before applying the model.
- Guidance Strength: Make sure to set the Flux Guidance strength to 4—I’ve found that anything lower results in strange image artifacts.
Once these are in place, you can queue up your prompt and start generating images with greater control.
The Importance of Crafting the Right Prompt
The quality of an image is often defined by the prompt behind it. In the process of working with Flux and ControlNet, utilizing a more refined prompt can significantly improve the final result. A valuable resource for this is PromptCrafters.co, which offers a comprehensive database designed to elevate image generation projects.
Initially created for MidJourney, PromptCrafters has expanded to support Flux, providing categorized prompts that cater to various creative needs. For instance, while working on an image featuring a woman in a wedding dress standing in a desert at golden hour, the photography section of the database provided a suitable starting point. After some adjustments to fit the specific requirements of Canny and ControlNet models, the result was a refined image that retained the structure of the original but with enhanced details and style.
Final Thoughts
Using the Flux ControlNet model with well-crafted prompts allows for significantly more control and creativity in the image generation process. This approach opens up new possibilities for customizing and refining visuals based on precise requirements. For those looking to dive deeper into this topic, all the links and resources mentioned here can be found below.
Important Links and Resources
Workflows: / endangeredai
Prompt DB: https://promptcrafters.co
Learn to use Flux on ComfyUI: This new Open Source Model is better …
Patreon: / endangeredai – Access exclusive content and early releases.
Buymeacoffee: https://www.buymeacoffee.com/endangered – Support our content creation.
Newsletter: Stay updated with the latest AI tutorials and news by signing up at https://endangeredai.com/sign-up/
🔈 Join my Discord server: / discord
Read More:
Flux 1.0: Complete Guide to the New Model! It Looks Like a Better Model Than Midjourney and SD3