Get an AI Consultation

How to Train LoRas Locally with Flux: An Easy Guide

Posted by:

Trizia Fortich

|

On:

|

Have you ever wanted to Train Loras locally with flux, but felt overwhelmed by the process? You’re not alone! In this tutorial, I’ll walk you through each step, making it as simple and straightforward as possible. Whether you’re new to AI or just need a local offline guide, this is for you.

What You’ll Need Before You Start

Make sure your computer meets the following requirements:

  1. A Computer with a Powerful Graphics Card (GPU): You need a GPU with at least 24 GB of VRAM, like the Nvidia 3090 or 4090. If your computer doesn’t have this, you might struggle with the training process.
  2. Operating System: This guide is for Windows and Linux users. Unfortunately, Mac users will need a different setup.
  3. Tools: You’ll be using AI-Toolkit by Ostris and ComfyUI.

Setting Up Your Environment

First, we need to get your computer ready to train a LoRa. This involves installing some software and setting up a special workspace.

1.1 Install AI-Toolkit

AI-Toolkit is the main tool you’ll use to train your LoRa. Here’s how to set it up:

Go to the AI-Toolkit GitHub page. Find the code block that’s for Windows, copy it, and paste it into the Command Prompt. This will download the AI-Toolkit to your computer.

Choose Where to Install AI-Toolkit:

Pick a place on your computer to install AI-Toolkit, like your desktop, so it’s easy to find later.

Open Command Prompt:

In the folder where you want to install AI-Toolkit, click the address bar at the top, type CMD, and press Enter. This will open the Command Prompt.

Download AI-Toolkit:

Go to the AI-Toolkit GitHub page. Find the code block that’s for Windows, copy it, and paste it into the Command Prompt. This will download the AI-Toolkit to your computer.

1.2 Create a Virtual Environment

A virtual environment is like a special workspace where all your AI-Toolkit files will be stored safely. Here’s how to set it up:

Let AI-Toolkit Create the Environment:

  • When you downloaded AI-Toolkit, it should have automatically started creating this environment for you. It’s important because it keeps everything organized and separate from the rest of your computer.

Activate the Virtual Environment:

  • Each time you restart your computer or close the Command Prompt, you’ll need to reactivate this environment. To do this, open the Command Prompt in your AI-Toolkit folder and type in the command to activate it (this might be .\venv\Scripts\activate for Windows).

1.3 Install Additional Tools

Sometimes, the Command Prompt might tell you that certain tools like Git or Python are missing. Here’s how to fix that:

  1. Install Git:
    • If the Command Prompt says Git isn’t installed, search for “Git download” on Google, download it, and install it.
  2. Install Python:
    • Similarly, if Python is missing, search for “Python download”, download the latest version, and install it.

Download and Configure Your Model

Now that the toolkit is set up, you need to download the model you’ll be training and make some configurations.

2.1 Choose the Right Model

AI-Toolkit uses different models, depending on what you want to do with your LoRa.

  1. Schnell Model: Use this if you plan to use your LoRa commercially (for profit).
  2. Dev Model: Use this for personal projects or experiments.

2.2 Get an Access Token for the Dev Model

If you chose the Dev model, you’ll need a special code called an access token to download it:

  1. Visit Hugging Face: Go to the Hugging Face website, create an account if you don’t have one, and navigate to your profile settings.
  2. Create an Access Token: In your settings, find the option for Access Tokens, and create a new one. Name it something like “ai-toolkit”. Copy this token.
  3. Save the Token in a File: Go back to your AI-Toolkit folder. Create a new text file, name it .env, and paste the token inside. This will let AI-Toolkit download the Dev model.

Preparing Your Training Data

For your LoRa to learn, it needs data—specifically, images and descriptions (captions) of those images.

3.1 Collect Your Images

Gather a set of images that you want your LoRa to learn from. These could be photos, drawings, or any visuals related to your project.

3.2 Add Captions to Your Images

Each image needs a description to help the AI understand what it’s looking at. You can do this manually or automatically.

Manual Captioning (Time-Consuming):

  • For each image, create a .txt file with the same name as the image and write a short description of what’s in the picture. For example, if your image is called dog.jpg, your text file should be called dog.txt.

Automatic Captioning (Recommended):

  • To save time, use a tool called ComfyUI to automatically generate captions.
  • Install ComfyUI: Download and install ComfyUI on your computer.
  • Run Captioning: Open ComfyUI, load your images, and let it create the captions for you. The captions will be saved in .txt files with the same names as your images.

Training Your LoRa

Now comes the exciting part—training your LoRa! This is where your AI learns from the images and captions you provided.

4.1 Set Up the Training Configuration

You’ll need to tweak some settings in the AI-Toolkit before you start training.

  1. Open the Configuration File:
    • In your AI-Toolkit folder, find a file called ‘train_lora_flux_24gb.yml.’ This file contains all the settings for your training process.