[New Pipeline]: DeepCache: Accelerating Diffusion Models for Free

[kadirnar]

Repo: https://github.com/horseee/DeepCache
Project Page: https://horseee.github.io/Diffusion_DeepCache/

@patrickvonplaten @sayakpaul

I tested it locally and it is a very fast pipeline. But I need help adding it to the diffusers library. Should I add this to the community pipeline section?

I just need to update the unet class.

https://github.com/horseee/DeepCache/tree/master/DeepCache/sdxl

[sayakpaul]

It's indeed quite fast.

I am happy to support it via a community pipeline first.

[sayakpaul]

Actually, on second thought, I think it should be a part of the core offerings from diffusers.

I quickly skimmed through the codebase, and the changes seem fairly simple enough to be incorporated. Given the speedup it achieves, I feel relatively strongly about making it a core feature.

Some thoughts about the design are below.

In the original codebase, the authors rely on the call and forward arguments of the pipeline and the UNet, respectively, to implement DeepCache. I wonder if it makes sense to follow that or rather do it in a way similar to how we did FreeU i.e., via enable and disable methods.

I think following the original implementation strategy from the authors is a better choice here as the DeepCache arguments are better experimented like that. Users would experiment with values like cache_interval, cache_layer_id, etc., and then settle with the values as per their needs. With an enable/disable design, we might have to call the corresponding disable method first to be able to properly set the DeepCache arguments.

Wonder what others think here. Cc: @yiyixuxu @patrickvonplaten @DN6.

[horseee]

Hi @sayakpaul @kadirnar ,

We have a new implementation that doesn't require any modifications to the forward function and its arguments. Now DeepCache can be toggled on or off using an external helper. Here is an example:

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16).to("cuda:0")

from DeepCache import DeepCacheSDHelper
helper = DeepCacheSDHelper(pipe=pipe)
helper.set_params(
    cache_interval=3,
    cache_branch_id=0,
)
helper.enable()

deepcache_image = pipe(
        prompt,
        output_type='pt'
).images[0]
helper.disable()

We test upon three pipelines: StableDiffusionPipeline, StableDiffusionXLPipeline, StableVideoDiffusionPipeline. We've successfully tested the new implementation across three different pipelines, and I'm happy to report that it performs consistently well. The results are in line with those from our previous version of implementation.

Here is a quick review of the results:

• StableDiffusionPipeline (SD1.5): Left: Original Pipeline (2.57s) / Right: + DeepCache (1.11s)

• StableDiffusionXLPipeline: Left: Original Pipeline (12.93s) / Right: + DeepCache (5.26s)

• StableVideoDiffusion: Left: Original Pipeline (97.16s) / Right: + DeepCache (58.35s)

I'm wondering about the feasibility of integrating this implementation of DeepCache into Diffusers. Looking forward to any insights or suggestions on this!

[jorgemcgomes]

@horseee I was testing this new implementation of deepcache (very easy to use, just copy paste a small file and import the helper 👍 ), and I'm getting different results. The speedups are indeed very impressive (2-3x faster), but the quality nowhere close to the original.
All results below using a fixed seed. The model is a fine-tune of SD 2.1 (can't share). Tried with both Euler-A and UniPC schedulers -- similar issues. 25 steps.

No deepcache:

With deepcache (cache_interval=3):

With cache_interval=2 things look significantly better, but still far from lossless:

I still think the results are pretty decent for the amount of speedup it provides, but calling it "nearly lossless" is a stretch...

[sayakpaul]

All results below using a fixed seed. The model is a fine-tune of SD 2.1 (can't share). Tried with both Euler-A and UniPC schedulers -- similar issues. 25 steps.

Does this observation still stand with the default SD 2.1? That question essentially helps us answer if the behaviour of DeepCache is sensitive to fine-tunes (although it shouldn't be).

[sayakpaul]

@horseee thank you so much for your efforts here!

I think to truly honor your efforts, we'd like to refer the users to your great repository. We do the same for token merging too: https://huggingface.co/docs/diffusers/main/en/optimization/tome. Would this be okay for you?

[horseee]

All results below using a fixed seed. The model is a fine-tune of SD 2.1 (can't share). Tried with both Euler-A and UniPC schedulers -- similar issues. 25 steps.

Does this observation still stand with the default SD 2.1? That question essentially helps us answer if the behaviour of DeepCache is sensitive to fine-tunes (although it shouldn't be).

Hi all, here are our results on the default pipeline of SD2.1:

Left: Original Pipeline (2.3s) / Right: + DeepCache (1.0s)

[horseee]

@horseee thank you so much for your efforts here!

I think to truly honor your efforts, we'd like to refer the users to your great repository. We do the same for token merging too: https://huggingface.co/docs/diffusers/main/en/optimization/tome. Would this be okay for you?

Yes! It is good for me😄

Would it be okay if I prepare the documentation and then submit a Pull Request for review?

[sayakpaul]

Yes, let's do that :)

[jorgemcgomes]

@horseee This caught my curiosity so I tested this in an isolated environment, everything as default as possible.
https://colab.research.google.com/drive/1f1LbV2Vc4H4uXd1lCYv0z1P4L-TNp64j?usp=sharing
Still couldn't replicate your good results.

With the default pipeline, there are some clear differences in some image pairs (see for instance last image where the astronaut is not riding the horse with DeepCache), but no always.

With a different scheduler and lower inference steps, the differences seem larger and definitely worse.

I'm going out on a limb here and hypothesise that this approach only works well with high step counts.
Which would not be very interesting for a lot of use cases, since we already have schedulers that can deliver excellent results with very low step counts.

default pipeline (DDIM 50 steps)

Deepcache disabled

Deepcache enabled

20 steps and UniPC

Deepcache disabled

Deepcache enabled

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

We have an entire doc page dedicated to DeepCache: https://huggingface.co/docs/diffusers/main/optimization/deepcache