[Doc] Add DeepCache in section `optimization/General optimizations` #6390

horseee · 2023-12-29T18:16:36Z

What does this PR do?

This PR is for the documentation of DeepCache, which is a new training-free method for accelerating diffusion models. DeepCache can be easily used on StableDIffusionPipeline to accelerate the inference speed. The doc contains the installation and usage of DeepCache, and also a quick overview of the quality and speed after applying DeepCache on the Stable Diffusion Pipeline.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
[New Pipeline]: DeepCache: Accelerating Diffusion Models for Free #6193
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@sayakpaul

sayakpaul · 2023-12-30T03:55:06Z

Thanks for the very nice doc!

@stevhliu, would it make sense to have a dedicated page for the optimization techniques that lie outside of diffusers? We have one already: https://huggingface.co/docs/diffusers/main/en/optimization/tome. Now, we will have another on DeepCache.

sayakpaul · 2023-12-30T03:56:27Z

docs/source/en/optimization/deepcache.md

+-->
+
+# DeepCache
+[DeepCache](https://huggingface.co/papers/2312.00858) accelerates [`StableDiffusionPipeline`] by strategically caching and reusing high-level features, while efficiently updating low-level features, which leverages the unique properties of the U-Net architecture. 


If it supports StableDiffusionXLPipeline too, I'd mention it here.

Additionally, out of curiosity: does it support the img2img, and inpainting pipelines from SD and SDXL yet? If not, it might be extremely beneficial for the community to have them supported because they are so widely used.

Yes, it supports StableDiffusionXLPipeline!

For other tasks, like inpainting, img2img, and video, I listed some of the results here:

Task Model Original Pipeline + DeepCache Speedup

inpainting diffusers/stable-diffusion-xl-1.0-inpainting-0.1 1.8x

inpainting runwayml/stable-diffusion-inpainting 1.7x

img2img runwayml/stable-diffusion-v1-5 1.6x

video stabilityai/stable-video-diffusion-img2vid-xt 1.7x

In these cases, under the premise of ensuring quality and consistency, the acceleration ratio obtained is not particularly high.

I would later add scripts of these cases in the DeepCache repo😃

sayakpaul · 2023-12-30T03:57:18Z

docs/source/en/optimization/deepcache.md

+The `set_params` method accepts two arguments: `cache_interval` and `cache_branch_id`. `cache_interval` means the frequency of feature caching, specified as the number of steps between each cache operation. `cache_branch_id` identifies which branch of the network (ordered from the shallowest to the deepest layer) is responsible for executing the caching processes. 
+Opting for a lower `cache_branch_id` or a larger `cache_interval` can lead to faster inference speed; however, this may come at the cost of reduced image quality. Once those arguments are set, use the `enable` or `disable` methods to activate or deactivate the DeepCacheSDHelper respectively.


Very clear! If there's any ablation already done on these hyperparameters in the original DeepCache paper, I'd mention it here too.

The ablation experiments for cache_interval are detailed in Tables 13 and 14 of the appendix, using SD1.5 as the base model. Regarding cache_branch_id, its corresponding ablation experiments are also presented, found in Table 12; however, these experiments were carried out using the DDPM model.

cache_interval: (N here represents the cache interval)

cache_branch_id:

Would this be okay?

Sure. Feel free to include this in the doc.

docs/source/en/optimization/deepcache.md

sayakpaul

Sweet!

Co-authored-by: Sayak Paul <[email protected]>

horseee · 2024-01-02T07:07:08Z

Thanks @sayakpaul for the wonderful suggestions😃 Following your advice, I have updated the doc, and also the DeepCache package to support inpaint/img2img tasks for SD.

Warm welcome for any further advice or feedback

sayakpaul · 2024-01-02T08:00:40Z

Thank you! I have requested for another review from @stevhliu. Let's see what Steven has to say :-)

stevhliu

LGTM, thanks for adding the docs! 😄

docs/source/en/optimization/deepcache.md

sayakpaul · 2024-01-04T04:00:54Z

@stevhliu WDYT #6390 (comment) ?

HuggingFaceDocBuilderDev · 2024-01-04T04:08:54Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

stevhliu · 2024-01-04T18:56:17Z

would it make sense to have a dedicated page for the optimization techniques that lie outside of diffusers?

Oops sorry I missed that Sayak! I assume you mean creating a section for optimization techniques outside of Diffusers and grouping ToMe and DeepCache under it since we're already adding a dedicated page for DeepCache here.

I don't think it's necessary at this point since we only have the two (ToMe + DeepCache), and it'll probably make the table of contents more cluttered to add another subsection. I also don't think users really care whether a technique is native or external to Diffusers as long as it works and is clearly documented, so maybe we don't need to organize it so granularly right now 🙂

sayakpaul · 2024-01-05T01:33:04Z

@horseee let's apply the changes suggested by Steven and ship this 🚀

Co-authored-by: Steven Liu <[email protected]>

horseee · 2024-01-05T04:24:52Z

@horseee let's apply the changes suggested by Steven and ship this 🚀

Thanks @sayakpaul and @stevhliu! I have applied the suggested modifications.

sayakpaul · 2024-01-05T04:27:31Z

Thank you so much for your contributions and awesome work on DeepCache! Let's continue to make diffusion models more accessible!

Also, if you want, feel free to include a link to the doc page (when it appears on the Diffusers site) from your DeepCache GitHub repo.

horseee · 2024-01-05T07:14:54Z

Thank you so much for your contributions and awesome work on DeepCache! Let's continue to make diffusion models more accessible!

Also, if you want, feel free to include a link to the doc page (when it appears on the Diffusers site) from your DeepCache GitHub repo.

Many thanks for your support!

I've updated the GitHub repo to include the link, and I would continue to support DeepCache with more pipelines/models in Diffusers. Thanks again for your support!

sayakpaul · 2024-01-05T07:17:26Z

Just for folks visiting this PR, the doc is live here: https://huggingface.co/docs/diffusers/main/en/optimization/deepcache.

…uggingface#6390) * add documentation for DeepCache * fix typo * add wandb url for DeepCache * fix some typos * add item in _toctree.yml * update formats for arguments * Update deepcache.md * Update docs/source/en/optimization/deepcache.md Co-authored-by: Sayak Paul <[email protected]> * add StableDiffusionXLPipeline in doc * Separate SDPipeline and SDXLPipeline * Add the paper link of ablation experiments for hyper-parameters * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Steven Liu <[email protected]>

horseee and others added 8 commits December 30, 2023 00:21

add documentation for DeepCache

292359a

fix typo

ad0ed7e

add wandb url for DeepCache

bde6f9f

fix some typos

befdce5

add item in _toctree.yml

161db38

update formats for arguments

4b1c9f7

Update deepcache.md

ca32b48

Merge branch 'main' into main

8079403

sayakpaul reviewed Dec 30, 2023

View reviewed changes

docs/source/en/optimization/deepcache.md Outdated Show resolved Hide resolved

sayakpaul approved these changes Dec 30, 2023

View reviewed changes

horseee and others added 4 commits December 30, 2023 17:02

Update docs/source/en/optimization/deepcache.md

0aa17b8

Co-authored-by: Sayak Paul <[email protected]>

add StableDiffusionXLPipeline in doc

d6a77b7

Separate SDPipeline and SDXLPipeline

c10e7ed

Add the paper link of ablation experiments for hyper-parameters

5381ec4

sayakpaul requested a review from stevhliu January 2, 2024 08:00

Merge branch 'main' into main

91beda0

stevhliu approved these changes Jan 3, 2024

View reviewed changes

Merge branch 'main' into main

ca98018

sayakpaul and others added 2 commits January 5, 2024 07:03

Merge branch 'main' into main

214c8ae

Apply suggestions from code review

f8b4bd8

Co-authored-by: Steven Liu <[email protected]>

sayakpaul merged commit 811fd06 into huggingface:main Jan 5, 2024

Task	Model	Speedup
inpainting	diffusers/stable-diffusion-xl-1.0-inpainting-0.1	1.8x
inpainting	runwayml/stable-diffusion-inpainting	1.7x
img2img	runwayml/stable-diffusion-v1-5	1.6x
video	stabilityai/stable-video-diffusion-img2vid-xt	1.7x

		The `set_params` method accepts two arguments: `cache_interval` and `cache_branch_id`. `cache_interval` means the frequency of feature caching, specified as the number of steps between each cache operation. `cache_branch_id` identifies which branch of the network (ordered from the shallowest to the deepest layer) is responsible for executing the caching processes.
		Opting for a lower `cache_branch_id` or a larger `cache_interval` can lead to faster inference speed; however, this may come at the cost of reduced image quality. Once those arguments are set, use the `enable` or `disable` methods to activate or deactivate the DeepCacheSDHelper respectively.

[Doc] Add DeepCache in section optimization/General optimizations #6390

[Doc] Add DeepCache in section optimization/General optimizations #6390

Uh oh!

Conversation

horseee commented Dec 29, 2023

What does this PR do?

Before submitting

Who can review?

Uh oh!

sayakpaul commented Dec 30, 2023

Uh oh!

sayakpaul Dec 30, 2023

Choose a reason for hiding this comment

Uh oh!

horseee Dec 30, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul Dec 30, 2023

Choose a reason for hiding this comment

Uh oh!

horseee Dec 30, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul Dec 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

horseee commented Jan 2, 2024

Uh oh!

sayakpaul commented Jan 2, 2024

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul commented Jan 4, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Jan 4, 2024

Uh oh!

stevhliu commented Jan 4, 2024

Uh oh!

sayakpaul commented Jan 5, 2024

Uh oh!

horseee commented Jan 5, 2024

Uh oh!

sayakpaul commented Jan 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

horseee commented Jan 5, 2024

Uh oh!

sayakpaul commented Jan 5, 2024

Uh oh!

Uh oh!

[Doc] Add DeepCache in section `optimization/General optimizations` #6390

[Doc] Add DeepCache in section `optimization/General optimizations` #6390

sayakpaul commented Jan 5, 2024 •

edited

Loading