Are Flux1-dev Lora's trained with kohya sd-scripts usable with stable-diffusion.cpp?

[ddpasa]

The reason I ask is because I'm getting very different results compared to SwarmUI (with a Comfy backend).

As a followup, this lora also trains the t5 and clip encoders, is that still ok?

[stduhpf]

I think they should work. Are there any strange messages in the logs?

[ddpasa]

I think they should work. Are there any strange messages in the logs?

I don't get any errors but the image is practically the same as before.

[stduhpf]

Also have you tried playing with the --lora-apply-mode parameter to see how it affects the output?

[ddpasa]

Also have you tried playing with the --lora-apply-mode parameter to see how it affects the output?

I tried auto and at_runtime and got identical results. Any other settings I should try?

Is there a check against a key mismatch? For example any keys in the Lora that are not used because they don't match the model?

[stduhpf]

Is there a check against a key mismatch? For example any keys in the Lora that are not used because they don't match the model?

There is, and it should be displayed in the logs if it happens to be the case.

[ddpasa]

Is there a check against a key mismatch? For example any keys in the Lora that are not used because they don't match the model?

There is, and it should be displayed in the logs if it happens to be the case.

Ok, scanning the logs I don't see any lora specific errors. Here is the relevant part of the log:

[INFO ] model.cpp:1580 - loading tensors completed, taking 1.41s (process: 0.00s, read: 0.39s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.29s)
[DEBUG] stable-diffusion.cpp:720  - finished loaded file
[INFO ] stable-diffusion.cpp:792  - total params memory size = 21661.05MB (VRAM 21661.05MB, RAM 0.00MB): text_encoders 9318.83MB(VRAM), diffusion_model 12247.64MB(VRAM), vae 94.57MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:865  - running in Flux FLOW mode
[DEBUG] stable-diffusion.cpp:3163 - generate_image 1024x1024
[DEBUG] stable-diffusion.cpp:1143 - lora /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors:1.00
[INFO ] stable-diffusion.cpp:1009 - apply lora at runtime
[INFO ] model.cpp:373  - load /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors using safetensors format
[DEBUG] model.cpp:503  - init from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors', prefix = 'lora.'
[INFO ] lora.hpp:40   - loading LoRA from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors'
[DEBUG] model.cpp:1351 - using 16 threads for model loading
[DEBUG] model.cpp:1373 - loading tensors from /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors

  |==========================>                       | 849/1632 - 0.00s/it�[K
  |==================================================| 1632/1632 - 8119.40it/s�[K
[INFO ] model.cpp:1580 - loading tensors completed, taking 0.20s (process: 0.00s, read: 0.00s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] model.cpp:373  - load /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors using safetensors format
[DEBUG] model.cpp:503  - init from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors', prefix = 'lora.'
[INFO ] lora.hpp:40   - loading LoRA from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors'
[DEBUG] model.cpp:1351 - using 16 threads for model loading
[DEBUG] model.cpp:1373 - loading tensors from /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors

  |==================================================| 1632/1632 - 1632000.00it/s�[K
[INFO ] model.cpp:1580 - loading tensors completed, taking 0.00s (process: 0.00s, read: 0.00s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] model.cpp:373  - load /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors using safetensors format
[DEBUG] model.cpp:503  - init from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors', prefix = 'lora.'
[INFO ] lora.hpp:40   - loading LoRA from '/StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors'
[DEBUG] model.cpp:1351 - using 16 threads for model loading
[DEBUG] model.cpp:1373 - loading tensors from /StableDiffusion/models/flux/lora/flux1_lora_v3.safetensors

  |==================================================| 1632/1632 - 0.00s/it�[K
[INFO ] model.cpp:1580 - loading tensors completed, taking 0.00s (process: 0.00s, read: 0.00s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:1153 - apply_loras completed, taking 0.25s
[INFO ] stable-diffusion.cpp:3197 - sampling using Euler method
[INFO ] denoiser.hpp:364  - get_sigmas with discrete scheduler
[INFO ] stable-diffusion.cpp:3320 - TXT2IMG

[stduhpf]

What if you set the apply mode to immediately?

[ddpasa]

What if you set the apply mode to immediately?

same result.

this is the command I'm running:

sd-cli --diffusion-model /StableDiffusion/models/flux/flux1-dev-q8_0.gguf --vae /StableDiffusion/models/flux/vae/diffusion_pytorch_model.safetensors --clip_l /StableDiffusion/models/flux/clip_l.safetensors --t5xxl /StableDiffusion/models/flux/t5xxl_fp16.safetensors --cfg-scale 1.0 --sampling-method euler -v --lora-model-dir /StableDiffusion/models/flux/lora' --width 1024 --height 1024 --steps 50 -s 11 -p "prompt goes here <lora:lorafilename:1>" --preview-interval 10 --preview vae --lora-apply-mode immediately

[stduhpf]

It's supposed to display stats about the unused LoRA weights in immediate mode.

[ddpasa]

ah you are right, now I see the problem in the immediate mode.

error.txt

None of the lora layers are actually applied.

[leejet]

Can you provide the CivitAI or Hugging Face link for the LoRA you’re using?

[ddpasa]

Can you provide the CivitAI or Hugging Face link for the LoRA you’re using?

Hello @leejet , this is the default output format of kohya sd-scripts. I trained this myself. It works out of the box with sd-forge and comfyui.

If it is helpful, I can create a huggingface account tomorrow and put it there, and share the link.

[leejet]

Perhaps PR #1103 can solve this problem.

[ddpasa]

#1103

@leejet , this helps! It doesn't solve all of the problem, but when previously no lora tensors were applied, now 912 / 1128 are applied. Log is attached. It looks like the transformer components are applied, but the text encoders (clip and t5) are not.

lora.txt