Get an AI Consultation

Stable Diffusion 3 is Finally Here!

Posted by:

Trizia Fortich

|

On:

|

Introduction

Stable Diffusion 3 is the latest image generation model from Stability AI, offering significant advancements over previous versions. It promises better text generation, prompt adherence, and overall image quality. This guide will walk you through the features, setup, and usage of Stable Diffusion 3.

Key Features of Stable Diffusion 3

1. Improved Text Generation: Generates more accurate and coherent text within images.

2. Enhanced Prompt Adherence: Better understanding and execution of detailed instructions.

3. Superior Image Quality: Higher fidelity and more detailed images compared to previous models.

4. Advanced Character Placement: Accurate positioning and trait assignment for characters.

5. Reduced Anomalies: Fewer errors like multiple arms or misplaced objects.

Setting Expectations

While Stable Diffusion 3 offers impressive out-of-the-box capabilities, it is essential to manage expectations. The model serves as a robust base for the community to fine-tune and improve further. Initial results may not be perfect, but community efforts will enhance its performance over time.

Setup Guide

The optimal way to use Stable Diffusion 3 currently is through ComfyUI or SwarmUI, both from Stability AI. For Stability Matrix users, simply go to the packages section, add a package, and install Stable Swarm UI. If you’re unfamiliar with Stability Matrix, stay tuned for my upcoming video on my youtube channel: https://www.youtube.com/@EndangeredAI

After installing Stable Swarm, visit the Hugging Face and find the Stable Diffusion 3 medium model. You’ll need a Hugging Face account to download it, so log in if you already have one or sign up if you don’t.

They offer a creator plan priced at approximately $20 per month if your usage falls below a million dollars in revenue or 6,000 generations, or similar thresholds. If you’re not monetizing the model, there’s no cost involved.

The model is intended for non-commercial use unless you obtain a separate license from Stability AI. Agree to the terms to gain access to the model. Navigate to the file section where you’ll find three versions: SD3 medium, SD3 medium with Clips, and SD3 medium with Clips and T5. The SD3 medium variant is the lightest, lacking additional text encoders. The Clips and T5 versions include various text encoders, with the Clips plus T5 being the most comprehensive at 10 GB, demanding higher resources. Alternatively, you can download the text encoders separately and integrate them into ComfyUI workflows.

For my setup with a 3090 GPU, I’ll opt for the complete version and run it first on Swarm, then ComfyUI. Let’s proceed with the download. Once finished, locate the file in your downloads folder or models folder. If you’re using Stability Matrix Stable Diffusion, place the model in your checkpoints folder.

Using Stable Diffusion 3

Stable Swarm UI

  1. Navigate to the models section and select the Stable Diffusion 3 medium model. Input your prompts. Note that the first run may take longer as it downloads necessary components.

Testing Prompts

  • Single Subject Prompts: Start with simple prompts and gradually increase complexity.

Example #1

Let’s explore another prompt. I tested this one with MidJourney, and interestingly, I didn’t need to include any negative prompts. The model’s performance has notably improved since downloading the clips, demonstrating remarkable inference speed. Even on my 3090, it only took a few seconds to generate.

Example #2

Text within Images:

Include text in your prompts to test the model’s text generation. Example: “A woman holding a sign that says ‘Like and Subscribe’”.

Multiple Subjects:

Test the model’s handling of multiple subjects and interactions.

Prompt: The two ship adrift in the sea, in the middle of a violent storm. The ship on the left is red, and the one on the right is blue.

Adjustments and Fine-Tuning

  • If results are not as expected, use negative prompts to refine the output.
  • Experiment with different styles (e.g., adding “anime” to your prompt).
  • For complex scenarios, consider using ComfyUI for more control over the generation process.

Using Stable Swarm with Comfy UI for Stable Diffusion 3

Stable Swarm also integrates with Comfy UI, providing a seamless experience for using Stable Diffusion 3. Let’s walk through the basic workflow of using Stable Swarm, highlighting the important nodes and steps needed to generate images.

  1. Loading Your Model:

Begin with the standard procedure of loading your checkpoint to bring your model into the environment.

Triple Clip Loader (Optional):

Typically used to load three separate clips into the workflow. If your clips are already embedded or not required, skip this step and proceed to selecting the clip directly from the interface.

SD3 Latent Image Loader:

This dedicated loader indicates potential changes in latent image generation methods. For testing purposes, retain the default text prompt unless specific adjustments are needed.

Understanding Node Configurations:

Notice the series of nodes inserted between prompts and the Ksampler. These include conditioning steps such as ‘zero out from the negative prompt’ and ‘set time step range’. These configurations refine how the model processes inputs. For now, maintain these settings to establish a baseline.

Running the Workflow:

  1. Execution:

Run the Nodes: Execute the workflow by running the nodes in sequence. The image generated from the provided prompt will be displayed.

  1. Review the Image:

Quality Check: Evaluate the generated image for detail and quality.

The image generated is detailed and well-crafted. The hand prompt by the Stability AI team produced a high-quality result. The image is lengthy, with an impressive level of detail, and it turned out very well.

Conclusion

Stable Diffusion 3 from Stability AI represents a major breakthrough in image generation technology. With enhanced text generation, improved prompt adherence, and superior image quality, this model offers substantial improvements. This guide has covered its features, setup, and usage, ensuring you can fully utilize Stable Diffusion 3.

For the best results, leverage tools like ComfyUI and SwarmUI, and collaborate with the community to refine your workflows. Whether you’re a creator or developer, Stable Diffusion 3 provides a robust foundation for innovative and high-quality image generation.

Imagine the possibilities—what stunning visuals will you create with the power of Stable Diffusion 3?

Join the Community