-
Notifications
You must be signed in to change notification settings - Fork 6.3k
Description
Describe the bug
When checkpointing at defined steps, such as every 50 steps I get a 'CLIPTextModel' object has no attribute 'peft_config' while trying to save the model. This only happens when checkpointing_steps < max_train_steps, as the code wouldn't run if it was higher than the train steps
Reproduction
python train_dreambooth_lora_sdxl_advanced.py --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" --dataset_name="./trainimages/" --caption_column=prompt --instance_prompt="a TOK person" --validation_prompt="a TOK person dressed as Yoda" --output_dir="huggy_v11" --caption_column="prompt" --mixed_precision="bf16" --resolution=1024 --train_batch_size=1 --repeats=1 --gradient_accumulation_steps=1 --gradient_checkpointing --train_text_encoder_ti --learning_rate=1 --text_encoder_lr=1 --lr_scheduler="constant" --snr_gamma=5.0 --lr_warmup_steps=0 --rank=32 --max_train_steps=1000 --checkpointing_steps=50 --seed="0" --optimizer="prodigy" --prodigy_safeguard_warmup=True --prodigy_use_bias_correction=True --adam_beta1=0.9 --adam_beta2=0.99 --adam_weight_decay=0.01
Logs
Traceback (most recent call last):
File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 2100, in <module>
main(args)
File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 1895, in main
accelerator.save_state(save_path)
File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\accelerate\accelerator.py", line 2793, in save_state
hook(self._models, weights, output_dir)
File "C:\Users\\Documents\GitHub\\train_dreambooth_lora_sdxl_advanced.py", line 1320, in save_model_hook
get_peft_model_state_dict(model)
File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\peft\utils\save_and_load.py", line 64, in get_peft_model_state_dict
config = model.peft_config[adapter_name]
File "C:\Users\\Documents\GitHub\\venv\lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'CLIPTextModel' object has no attribute 'peft_config'
System Info
diffusers
version: 0.25.0.dev0- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.16
- PyTorch version (GPU?): 2.3.0.dev20240102+cu121 (True)
- Huggingface_hub version: 0.20.1
- Transformers version: 4.37.0.dev0
- Accelerate version: 0.23.0
- xFormers version: 0.0.21
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
Who can help?
No response