support for dadapaption V3

[sdbds]

Dadapaption Change Log V3
1、Major improvements to DAdaptAdam, improving the performance particularly on Transformer models. This variant may behave differently in practice. The old version is availiable under experimental/d_adapt_adam_preprint if you wish to continue to use it.The IP variant is now the main variant of the method.
2、Added Lion. This is highly experimental. Feedback on it's performance is welcome.

What i change:
1、keep old DadaptAdam for use DAdaptation or DAdaptAdamPreprint
2、use new DadaptAdam variant.
3、add DAdaptLion
4、add DAdaptAdan IP variant for use DAdaptAdanIP

update way:

.\venv\Scripts\activate
pip install dadaptation -U

[kohya-ss]

Thank you for this! I will review and merge as soon as I have time.

[FurkanGozukara]

what learning rate and weight decay suggested?

[sdbds]

what learning rate and weight decay suggested?

see here