FEAT Convert non-LoRA PEFT adapters to LoRA (#2939)

This adds the possibility to convert a non-LoRA adapter into a LoRA
adapter. Not all LoRA adapters will support this, but many will.

Conversion is not precise, there will be a loss of performance. The
higher the rank, the lower the loss, but also the less efficient the
adapter. Also, for now, this only supports linear layers. Still, this
has some advantages:

- In PEFT, LoRA supports more features than most other methods, e.g.
mixed adapter batches. Thus the converted adapter can be used with those
features.

- Some downstream packages support LoRA adapters, but not other PEFT
methods, e.g. Diffusers. The conversion allows to use a non-LoRA adapter
with those packages.

Users can pass a fixed rank for the LoRA adapter or a float that will
use a dynamic rank based on the threshold of the contribution of the
singular values.

Unrelated changes

I noticed that the VB-LoRA layer had no __repr__, so it was added.

The return type annotation of set_peft_model_state_dict was incorrect.

Author: BenjaminBossan

docs/source/_toctree.yml

@@ -155,5 +155,7 @@
       title: Hotswapping adapters
     - local: package_reference/functional
       title: Functions for PEFT integration
+    - local: package_reference/lora_conversion
+      title: Converting non-LoRA adapters to LoRA
     title: Utilities
   title: API reference

docs/source/package_reference/hotswap.md

@@ -69,6 +69,8 @@ Hotswapping works with transformers models and diffusers models. However, there
 - It only works for the same PEFT method, so no swapping LoRA and LoHa, for example.
 - The adapter that is being swapped in must target the same layers as the previous adapter or a subset of those layers. It cannot target new layers. Therefore, if possible, start with the adapter that targets most layers.
 
+## API
+
 [[autodoc]] utils.hotswap.hotswap_adapter
     - all
 

docs/source/package_reference/lora_conversion.md

@@ -0,0 +1,146 @@
+<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+-->
+
+# LoRA conversion
+
+Functions that allow to convert non-LoRA PEFT models to LoRA models.
+
+## Description
+
+PEFT supports dozens of different parameter effficient fine-tuning techniques. The most popular one by far is LoRA. This means that many other packages support LoRA too. For example, [Diffusers](https://huggingface.co/docs/diffusers/main/en/api/loaders/lora) allows to load LoRA adapters to change the capabilities of diffusion models. [vLLM](https://docs.vllm.ai/en/stable/features/lora/) allows serving models with LoRA adapters. This is nice but unfortunately, all the other, non-LoRA PEFT methods are rarely supported. Therefore, even if another PEFT method would work better for your specific use case, you may be prevented from using it because downstream packages offer no support.
+
+Here we present a potential solution. PEFT offers two functions, [`save_as_lora`] and [`convert_to_lora`], which allow to convert a PEFT adapter into a LoRA adapter. Not all PEFT methods support this for now, but if they do, it means you can start with the PEFT method that works best for you and then later use it as if it were a LoRA adapter.
+
+## Example
+
+The LoRA rank for the converted adapter can either be set to a fixed rank by passing an int > 0 to the `rank` argument, or a dynamic rank, which adapts to each layer, by passing a float between 0 and 1 to the `rank` argument. Dynamic ranks can potentially be more efficient (same performance with fewer parameters).
+
+### Fixed LoRA rank
+
+The usage of [`save_as_lora`] is relatively straightforward:
+
+```python
+from peft import get_peft_model, save_as_lora
+
+# first load and train your non-LoRA PEFT model as normal
+base_model = ...
+non_lora_config = ...
+model = get_peft_model(base_model, non_lora_config)
+# check that this PEFT method can indeed be converted to LoRA
+assert model.supports_lora_conversion()
+...  # train the model
+
+# the rank of the LoRA adapter that you want to convert to
+target_rank = 64
+# save as a LoRA checkpoint
+save_as_lora(output_path, model, rank=target_rank)
+```
+
+This will create a LoRA checkpoint at `output_path` that you can load like any other LoRA adapter, or use in downstream packages such as Diffusers or vLLM.
+
+The [`convert_to_lora`] function is useful if you don't want to save the converted LoRA adapter but instead want to use the converted weights right away, for example to perform evaluations:
+
+```python
+from peft import convert_to_lora, get_peft_model, set_peft_model_state_dict
+
+base_model = ...
+non_lora_config = ...
+model = get_peft_model(base_model, non_lora_config)
+...  # train the model
+
+# get the lora config and state dict of the converted lora model
+lora_config, lora_state_dict = convert_to_lora(model, rank=target_rank)
+# reload the base model, or use model.unload()
+base_model = ...
+# apply the lora config to the base model
+lora_model = get_peft_model(base_model, lora_config)
+# load the LoRA weights onto the base model
+set_peft_model_state_dict(lora_model, state_dict)
+```
+
+### Dynamic LoRA rank
+
+In the examples above, we used a fixed LoRA rank for conversion. However, it is conceivable that some layers don't require a high rank to be accurately converted, while other layers require a higher rank. To accomodate this, PEFT offers the option to pass a float between 0 and 1 as the `rank` argument. Let's say you pass `rank=0.5`. This means that for each layer, the rank for the LoRA adapter is chosen such that the LoRA adapter explains 50% of the variance in weight introduced by original adapter. In more technical terms, under the hood we perform a [Singular Value Decomposition](https://en.wikipedia.org/wiki/Singular_value_decomposition) on the weight contribution of the adapter and then take the top singular values that, when normalized, sum up to the passed value.
+
+```python
+# set a dynamic rank by passing a float
+threshold = 0.7
+# save as a LoRA checkpoint
+save_as_lora(output_path, model, rank=threshold)
+# get the lora config and state dict directly:
+lora_config, lora_state_dict = convert_to_lora(model, rank=threshold)
+# inspect the different ranks per layer:
+print(lora_config.rank_pattern)
+```
+
+Using this type of dynamic LoRA rank can be useful if the contribution of the different layers varies a lot. The disadvantage is that it could mean that some layers will have a very high LoRA rank, which can lead to memory spikes. Please test what works best for your use case.
+
+### LoRA to LoRA conversion
+
+It is also possible to convert a LoRA adapter into another LoRA adapter. Why would you want to do that? There is one reason, namely if you want to reduce the rank of the LoRA adapter. If, after training, you want to shrink the LoRA adapter, use [`save_as_lora`] or [`convert_to_lora`] and pass a smaller rank. This will give you a new LoRA adapter that has a smaller memory and storage footprint.
+
+## Metrics
+
+### Non-LoRA to LoRA conversion
+
+Of course, converting one PEFT adapter into another adapter is a lossy process. The new adapter will most likely not perform as well as the initial adapter. Therefore, it is highly advised to **evaluate the converted LoRA adapter**. This way, you can make sure that the converted adapter performs well enough for your use case. The general rule applies that the higher the rank of the LoRA adaper, the better it will approximate your initial adapter. This means that the converted LoRA adapter may require more parameters than the original adapter to achieve a similar performanace.
+
+To give an example, here are some numbers that were derived on the [PEFT MetaMathQA benchmark](https://github.com/huggingface/peft/tree/main/method_comparison/MetaMathQA). For this, a [LoHa](https://huggingface.co/docs/peft/package_reference/loha) was used to fine-tune `meta-llama/Llama-3.2-3B` on MetaMathQA and evaluated on GSM8K. The initial LoKr adapter had rank 32, resulting in 18,350,080 trainable parameters, and a test accuracy of 41.85%. Evaluation required 12.25 GB of memory. The checkpoint was converted into LoRA with different values for the `rank`. The resulting outcome is:
+
+| rank | trainable parameters | test accuracy (%) | accuracy change | memory reserved (max, GB) | memory increase |
+|------|---------------------:|------------------:|----------------:|--------------------------:|----------------:|
+| 8    |              2293760 |             37.60 |           -4.25 |                     12.41 |            0.16 |
+| 16   |              4587520 |             38.89 |           -2.96 |                     12.15 |           -0.10 |
+| 32   |              9175040 |             40.11 |           -1.74 |                     12.41 |            0.16 |
+| 64   |             18350080 |             39.20 |           -2.65 |                     12.18 |           -0.07 |
+|      |                      |                   |                 |                           |                 |
+| 0.4  |              2428928 |             37.60 |           -4.25 |                     12.41 |            0.16 |
+| 0.5  |              4761600 |             40.18 |           -1.67 |                     12.41 |            0.16 |
+| 0.6  |              8857600 |             39.42 |           -2.43 |                     12.41 |            0.16 |
+| 0.7  |             16230400 |             39.04 |           -2.81 |                     12.15 |           -0.10 |
+
+As you can see, we can attain a test accuracy that comes close to the original LoHa adapter if the rank is sufficiently high. Choosing the right rank is a tradeoff between model performance and model efficiency. To reproduce this experiment, follow the script at https://github.com/huggingface/peft/tree/main/scripts/evaluate-lora-conversion.py.
+
+Note that the number of trainable parameters cannot be translated one to one into memory usage. Some PEFT methods require more, some less memory, even with the same number of trainable parameters. Therefore, even if after conversion, the LoRA adapter has more parameters than the original one, it could still be more memory efficient when serving.
+
+### LoRA to LoRA conversion
+
+Similar to the experiment above, we can also evaluate LoRA to LoRA conversion (i.e. LoRA compression). Here, we start with a LoRA adapter of rank 64 trained on the same setup as above with RS-LoRA. The initial adapter has 18,350,080 trainable parameters, a test accuracy of 52.92%, and requires 12.58 GB of memory for evaluation. The following table shows the results of converting this adapter to LoRA adapters of smaller rank:
+
+| rank | trainable parameters | test accuracy (%) | accuracy change | memory reserved (max, GB) | memory increase |
+|------|---------------------:|------------------:|----------------:|--------------------------:|----------------:|
+| 8    |              2293760 |             43.37 |           -9.55 |                     12.38 |           -0.20 |
+| 16   |              4587520 |             48.90 |           -4.02 |                     12.38 |           -0.20 |
+| 32   |              9175040 |             51.48 |           -1.44 |                     12.49 |           -0.09 |
+| 48   |             13762560 |             52.01 |           -0.91 |                     12.38 |           -0.20 |
+|      |                      |                   |                 |                           |                 |
+| 0.5  |              2150400 |             44.12 |           -8.80 |                     12.37 |           -0.21 |
+| 0.6  |              3082240 |             47.54 |           -5.38 |                     12.37 |           -0.21 |
+| 0.7  |              4448256 |             50.49 |           -2.43 |                     12.37 |           -0.21 |
+| 0.8  |              6510592 |             50.11 |           -2.81 |                     12.37 |           -0.21 |
+| 0.9  |             10022912 |             51.55 |           -1.37 |                     12.38 |           -0.20 |
+| 0.95 |             12976128 |             52.62 |           -0.30 |                     12.39 |           -0.19 |
+
+So for instance for rank 0.95, we can close the accuracy gap to just 0.3 percentage points while reducing the number of parameters by 30%. Also note that these compressed LoRAs can be better than directly training them on the lower rank -- e.g. for rank 32, training directly results in a test accuracy of 48.22%, while conversion from rank 64 results in 51.48%.
+
+## Caveats
+
+There are some limitations to the LoRA conversion. As mentioned above, a reduction in performance is expected and the converted LoRA will most likely be less parameter efficient than the original adapter. Morever, LoRA conversion has these limitations:
+
+- Right now, only adapters applied to linear layers can be converted.
+- Not all PEFT methods currently support LoRA conversion.
+
+If there is a lot of demand to extend LoRA conversion, please let us know and we will make it work with more layer types and PEFT methods.
+
+## API
+
+### Convert a non-LoRA model to a LoRA model, return the `LoraConfig` and `state_dict`
+
+[[autodoc]] tuners.lora.conversion.convert_to_lora
+    - all
+
+### Convert a non-LoRA model to a LoRA model, save the adapter checkpoint and config at the given path
+
+[[autodoc]] tuners.lora.conversion.save_as_lora
+    - all

scripts/evaluate-lora-conversion.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+# Copyright 2025-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Script to evaluate a PEFT checkpoint converted into a LoRA on GSM8K
+
+To run this script, first train a PEFT model on MetaMathQA as described here:
+
+https://github.com/huggingface/peft/tree/main/method_comparison/MetaMathQA
+
+Call the script with the `-v` (verbose) option. When that run finishes, it will save a checkpoint of that model and
+print a message like this: "Saved PEFT checkpoint to ...". Use this path as the `--path` argument to this script.
+
+Example usage:
+
+```bash
+# Convert to LoRA with rank 8 and evaluate it
+python evaluate-lora-conversion.py --path /path/to/peft/checkpoint --rank 8
+# Convert to LoRA with dynamic rank (50% singular value threshold) and evaluate it
+python evaluate-lora-conversion.py --path /path/to/peft/checkpoint --rank 0.5
+# Evaluate the original PEFT model without LoRA conversion
+python evaluate-lora-conversion.py --path /path/to/peft/checkpoint
+```
+
+The script will report the evaluation accuracy, maximum CUDA memory reserved, and evaluation time for the converted LoRA
+model.
+
+"""
+
+import argparse
+import importlib.util
+import os
+import sys
+import time
+
+import torch
+from transformers import AutoModelForCausalLM
+
+from peft import PeftModel, convert_to_lora, get_peft_model, set_peft_model_state_dict
+
+
+root = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+
+spec = importlib.util.spec_from_file_location("data", os.path.join(root, "method_comparison", "MetaMathQA", "data.py"))
+mm_data = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mm_data)
+sys.modules["data"] = mm_data
+
+spec = importlib.util.spec_from_file_location(
+    "utils", os.path.join(root, "method_comparison", "MetaMathQA", "utils.py")
+)
+mm_utils = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mm_utils)
+sys.modules["utils"] = mm_utils
+
+spec = importlib.util.spec_from_file_location("run", os.path.join(root, "method_comparison", "MetaMathQA", "run.py"))
+mm_run = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mm_run)
+
+
+def noop(*args, **kwargs):
+    pass
+
+
+def evaluate_model(model, tokenizer, ds_test):
+    torch.cuda.empty_cache()
+    torch.cuda.reset_peak_memory_stats()
+    tic = time.perf_counter()
+    predictions, responses = mm_run.evaluate(
+        model=model,
+        tokenizer=tokenizer,
+        ds=ds_test,
+        batch_size=50,
+        generate_kwargs={"max_length": 800, "max_new_tokens": 300, "pad_token_id": tokenizer.eos_token_id},
+        use_tqdm=True,
+    )
+    toc = time.perf_counter()
+    accuracy_peft = mm_utils.get_accuracy(predictions=predictions, responses=responses)
+    cuda_mem_reserved_max = torch.cuda.memory_reserved(0)
+    print(f"Evaluation Accuracy: {100 * accuracy_peft:.2f}%")
+    print(f"Max CUDA Memory Reserved: {cuda_mem_reserved_max / (1024**3):.2f} GB")
+    print(f"Evaluation Time: {toc - tic:.0f} seconds".format(toc - tic))
+
+
+def main(path_peft_model: str, rank: int | float | None) -> None:
+    model_id = "meta-llama/Llama-3.2-3B"
+    tokenizer = mm_utils.get_tokenizer(model_id=model_id, max_seq_length=768)
+    _, _, ds_test = mm_data.get_train_valid_test_datasets(
+        tokenizer=tokenizer, query_template="Question: {query} Think step by step.\nAnswer:", print_fn=noop
+    )
+
+    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(0)
+    model = PeftModel.from_pretrained(model, path_peft_model)
+    if rank is None:
+        print("Evaluating the original PEFT model without LoRA conversion...")
+        model.set_adapter("default")
+        model.print_trainable_parameters()
+        model.eval()
+        evaluate_model(model, tokenizer, ds_test)
+        return
+
+    print(f"Converting PEFT model to LoRA with rank={rank}...")
+    tic = time.perf_counter()
+    lora_config, lora_state_dict = convert_to_lora(model, rank=rank, progressbar=True)
+    toc = time.perf_counter()
+    print(f"Conversion completed in {toc - tic:.0f} seconds.".format(toc - tic))
+
+    del model
+    torch.cuda.empty_cache()
+    model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(0)
+
+    model = get_peft_model(model, lora_config)
+    model.print_trainable_parameters()
+
+    load_result = set_peft_model_state_dict(model, lora_state_dict)
+    assert not load_result.unexpected_keys, (
+        f"Unexpected keys when loading LoRA state dict: {load_result.unexpected_keys}"
+    )
+
+    del lora_state_dict
+    model.eval()
+    evaluate_model(model, tokenizer, ds_test)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Evaluate a PEFT checkpoint converted into a LoRA on GSM8K")
+    parser.add_argument(
+        "--path",
+        type=str,
+        required=True,
+        help="Path to the input PEFT checkpoint",
+    )
+    parser.add_argument(
+        "--rank",
+        required=False,
+        default=None,
+        help="Rank for the LoRA decomposition (int, float, or None for no conversion)",
+    )
+
+    args = parser.parse_args()
+    if args.rank is not None:
+        if "." in str(args.rank):
+            args.rank = float(args.rank)
+        else:
+            args.rank = int(args.rank)
+    main(args.path, args.rank)

src/peft/__init__.py

@@ -117,9 +117,11 @@
     WaveFTModel,
     XLoraConfig,
     XLoraModel,
+    convert_to_lora,
     create_arrow_model,
     get_eva_state_dict,
     initialize_lora_eva_weights,
+    save_as_lora,
 )
 from .tuners.cartridge.utils import (
     compose_cartridge_adapters,
@@ -244,6 +246,7 @@
     "bloom_model_postprocess_past_key_value",
     "cast_mixed_precision_params",
     "compose_cartridge_adapters",
+    "convert_to_lora",
     "create_arrow_model",
     "get_eva_state_dict",
     "get_layer_status",
@@ -259,6 +262,7 @@
     "prepare_model_for_kbit_training",
     "prompt_embeddings_from_past_key_values",
     "replace_lora_weights_loftq",
+    "save_as_lora",
     "set_peft_model_state_dict",
     "shift_tokens_right",
 ]

src/peft/peft_model.py

@@ -1618,6 +1618,21 @@ def create_or_update_model_card(self, output_dir: str):
         card.text = "\n".join(lines)
         card.save(filename)
 
+    def supports_lora_conversion(self, adapter_name: str = "default") -> bool:
+        """
+        Whether it is possible for the adapter of this model to be converted to LoRA.
+
+        Normally, this works if the PEFT method is additive, i.e. W' = W_base + delta_weight.
+        """
+        peft_config = self.active_peft_config
+        if peft_config.is_prompt_learning:
+            return False
+
+        if not hasattr(self.base_model, "supports_lora_conversion"):
+            return False
+
+        return self.base_model.supports_lora_conversion()
+
 
 class PeftModelForSequenceClassification(PeftModel):
     """

src/peft/tuners/__init__.py

@@ -35,9 +35,11 @@
     LoraConfig,
     LoraModel,
     LoraRuntimeConfig,
+    convert_to_lora,
     create_arrow_model,
     get_eva_state_dict,
     initialize_lora_eva_weights,
+    save_as_lora,
 )
 from .miss import MissConfig, MissModel
 from .mixed import MixedModel
@@ -132,7 +134,9 @@
     "WaveFTModel",
     "XLoraConfig",
     "XLoraModel",
+    "convert_to_lora",
     "create_arrow_model",
     "get_eva_state_dict",
     "initialize_lora_eva_weights",
+    "save_as_lora",
 ]