LoRA from train_dreambooth_lora_sdxl.py is not working in A1111 anymore

[patryk-bartkowiak-nitid]

Describe the bug

I have been using train_dreambooth_lora_sdxl.py and convert_diffusers_sdxl_lora_to_webui.py to train LoRA for specific character, It was working till like a week ago. I am using the same baseline model and the same data.

I realized that previous size of all the LoRA files had 29967176 bytes, now it has 29889672 and less keys in dict after I load it as pure .safetensors file.

I realized that it works fine with inference guide in README:

import torch
from diffusers import DiffusionPipeline

pretrained_model = "./pretrained_models/dreamshaper-xl"
lora_weights = "./outputs/dreamshaper-xl_claire/checkpoint-2000/"

prompt = "photo of wff woman, sitting in train"
negative_prompt = "text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated"

pipe = DiffusionPipeline.from_pretrained(pretrained_model, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
pipe.load_lora_weights(lora_weights)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    seed=420,
).images[0]

image.save("lora_inference.png")

But after I convert and load to A1111 (it loads correctly) it doesnt work anymore, looks like its adding some noise to the output only.

I already tried checkpointing to previous commits on diffusers, torch and torchvision, but nothing really helps. I am still not able to use LoRA in A1111.

Reproduction

Code to train LoRA:

export MODEL_NAME="pretrained_models/dreamshaper-xl"
export INSTANCE_DIR="data/claire"
export MAX_TRAIN_STEPS=5000
export CHECKPOINTING_STEPS=500


export OUTPUT_DIR="outputs/$(basename ${MODEL_NAME})_$(basename ${INSTANCE_DIR})_tmp"
export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1

printf "\n\nTraining Claire model with $MODEL_NAME on $INSTANCE_DIR, saving to $OUTPUT_DIR\n\n"

accelerate launch diffusers/examples/dreambooth/train_dreambooth_lora_sdxl.py \
	--instance_prompt="photo of wff woman, isolated on white background" \
	--pretrained_model_name_or_path=$MODEL_NAME \
	--instance_data_dir=$INSTANCE_DIR \
	--output_dir=$OUTPUT_DIR \
	--resolution=1024 \
	--train_batch_size=2 \
	--gradient_accumulation_steps=4 \
	--learning_rate=1e-4 \
	--lr_scheduler="constant" \
	--lr_warmup_steps=0 \
	--max_train_steps=$MAX_TRAIN_STEPS \
	--seed="0" \
	--train_text_encoder \
	--enable_xformers_memory_efficient_attention \
	--gradient_checkpointing \
	--use_8bit_adam \
	--checkpointing_steps=$CHECKPOINTING_STEPS

Code to convert to A1111 format

python /project/diffusers/scripts/convert_diffusers_sdxl_lora_to_webui.py {input_path} {output_path}

Logs

Can't really post any errors, looks like typical image generation, no errors or warning during training and conversion

System Info

- `diffusers` version: 0.26.0.dev0
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.27
- Python version: 3.10.9
- PyTorch version (GPU?): 2.0.0 (True)
- Huggingface_hub version: 0.20.3
- Transformers version: 4.37.2
- Accelerate version: 0.26.1
- xFormers version: 0.0.19
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Who can help?

@yiyixuxu @sayakpaul @DN6 @patrickvonplaten

[sayakpaul]

Thanks for the detailed thread. Can you pin me a version that was working as expected for you?

I am asking because none of those scripts went through significant logical changes in the past 7 days.

[patryk-bartkowiak-nitid]

Yeah that's the thing, I am unable to restore the environment perfectly and I'm blocked right now, not sure where the issue is :/

[sayakpaul]

Ah then it's a bit of a pity. In any case, please do ping me here if you're able to give me a pinpointed version. I am happy to look further from there :-)

[patryk-bartkowiak-nitid]

Anyway going through README guide it's not working properly, I am happy to meet or whatever to solve this issue :)

[sayakpaul]

README guide? Do you mean the commands from https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md don't work? Can you provide a fully reproducible snippet for me?

I am happy to meet or whatever to solve this issue :)

Sorry, we cannot do that. As maintainers, we need to be cognizant of our time and keep the discussions as open as possible,

[patryk-bartkowiak-nitid]

I mean command from
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md
combined with
https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_sdxl_lora_to_webui.py

Not sure on what part of the pipeline there is an issue, like I said I am able to use LoRA using code for inference that you provided in README, but can't correctly convert it. Might be both the conversion itself or LoRA has some different properties that conversion script can't handle.

Let me send you full pipeline for you to reproduce the issue, I will try to include as many details as possible:

1. Create VM with this docker image: pytorch/pytorch:2.0.0-cuda11.7-cudnn8-devel
2. Install dependencies:

apt update
apt install vim git tmux ffmpeg libsm6 libxext6 wget python3 python3-venv libgl1 libglib2.0-0 google-perftools -y

git clone https://github.com/huggingface/diffusers.git
cd diffusers
pip install -e .
cd examples/dreambooth
pip install -r requirements.txt
accelerate config default
pip install bitsandbytes xformers==0.0.19

3. Download baseline SDXL model:

wget https://civitai.com/api/download/models/333449 -O DreamShaperXL.safetensors

4. Convert .safetensors to suitable format using python:

import diffusers
pipe = diffusers.StableDiffusionXLPipeline.from_single_file("DreamShaperXL.safetensors")
pipe.save_pretrained("DreamShaperXL")

5. Train LoRA (6 images with the same woman on white background):

export MODEL_NAME="DreamShaperXL"
export INSTANCE_DIR="data/claire"
export MAX_TRAIN_STEPS=5000
export CHECKPOINTING_STEPS=500


export OUTPUT_DIR="outputs/$(basename ${MODEL_NAME})_$(basename ${INSTANCE_DIR})"
export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1

printf "\n\nTraining Claire model with $MODEL_NAME on $INSTANCE_DIR, saving to $OUTPUT_DIR\n\n"

accelerate launch diffusers/examples/dreambooth/train_dreambooth_lora_sdxl.py \
	--instance_prompt="photo of wff woman, isolated on white background" \
	--pretrained_model_name_or_path=$MODEL_NAME \
	--instance_data_dir=$INSTANCE_DIR \
	--output_dir=$OUTPUT_DIR \
	--resolution=1024 \
	--train_batch_size=2 \
	--gradient_accumulation_steps=4 \
	--learning_rate=1e-4 \
	--lr_scheduler="constant" \
	--lr_warmup_steps=0 \
	--max_train_steps=$MAX_TRAIN_STEPS \
	--seed="0" \
	--train_text_encoder \
	--enable_xformers_memory_efficient_attention \
	--gradient_checkpointing \
	--use_8bit_adam \
	--checkpointing_steps=$CHECKPOINTING_STEPS

6. Convert to Kohya format:

python /diffusers/scripts/convert_diffusers_sdxl_lora_to_webui.py outputs/DreamShaperXL_claire/pytorch_lora_weights.safetensors test.safetensors

7. Move to A1111:

mv test.safetensors stable-diffusion-webui/models/Lora/

[sayakpaul]

As mentioned I need to know a version that was working as expected for you.

CC: @linoytsaban @apolinario here.

[patryk-bartkowiak-nitid]

Well because I can't really provide it - can we just focus on the current version that is probably not working properly?

I was also considering A1111 to not work, but I am able to work with my previous LoRA's so I think it has to be something in this pipeline

[sayakpaul]

That makes it thousand times more difficult for us to make progress here actually, hence I am a bit adamant on it. To be able to pinpoint the issue -- can we say the trained LoRA provides expected results when the inference is done from diffusers?

Your initial issue description suggests so. So, I quite suspect that it's the conversion script that's the culprit here.

[patryk-bartkowiak-nitid]

Yes, LoRA provides expected results when the inference is done from diffusers.

When it's done in A1111 it actually changes the output image (same seed), but not in a way that it should, looks like its just adding some noise at the beginning of the generation process. I will send an example in 3 minutes

[sayakpaul]

Then it's quite likely that the conversion script is the problem as mentioned. So, I will let @apolinario and @linoytsaban comment further (as they are the developers of that script).

[patryk-bartkowiak-nitid]

A1111 Config:

photo of wff woman, rides gondola in Venice,
Negative prompt: text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated, BadDream, UnrealisticDream
Steps: 7, Sampler: DPM++ SDE Karras, CFG scale: 2, Seed: 420, Size: 1024x1024, Model hash: 676f0d60c8, Model: DreamShaperXL, Version: v1.7.0

Image without any LoRA:

Image with previously trained LoRA that works - trained for 8000 iterations with batch_size=1:

Image with new LoRA - trained for 4000 iterations with batch_size=2:

[patryk-bartkowiak-nitid]

Also adding an image generated locally with new LoRA that doesn't work in A1111 - trained for 4000 iterations with batch_size=2

Code to generate:

import torch
from diffusers import DiffusionPipeline

pretrained_model = "DreamShaperXL"
lora_weights = "./outputs/DreamShaperXL_claire/checkpoint-4000/"

prompt = "photo of wff woman, rides gondola in Venice,"
negative_prompt = "text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated"

pipe = DiffusionPipeline.from_pretrained(pretrained_model, torch_dtype=torch.float32)
pipe = pipe.to("cuda")
pipe.load_lora_weights(lora_weights)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    seed=420,
).images[0]

image.save("lora_inference.png")

Image:

Note

As you can see it's much closer, of course quality is not good enough because in AUTOMATIC1111 there are some additional things that make it look better like negative embeddings etc.