Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
49fab86
[WIP] Add LoRA multihead attention module
BenjaminBossan Jan 5, 2024
d8e9589
Make style
BenjaminBossan Jan 5, 2024
0e188a3
Remove commented code
BenjaminBossan Jan 5, 2024
b409d81
Remove assignment of weight to new module
BenjaminBossan Jan 5, 2024
173062c
Make state_dict and named_parameters work
BenjaminBossan Jan 5, 2024
1e007f5
Extend test coverage a bit
BenjaminBossan Jan 8, 2024
557c4a1
Clean ups after reviewer feedback:
BenjaminBossan Jan 9, 2024
add1f51
Reviewer feedback: removed another unnecessary arg
BenjaminBossan Jan 9, 2024
e44e030
Make style
BenjaminBossan Jan 9, 2024
8d62579
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Jan 9, 2024
c5d8a6b
Apply LoRA also to the out_proj of MHA
BenjaminBossan Jan 12, 2024
9dc4a4d
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Feb 7, 2024
c3fb2ce
Fix bug with incorrectly set gradient
BenjaminBossan Feb 7, 2024
17d407b
Fix failing tests
BenjaminBossan Feb 7, 2024
4cbf6e9
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Feb 26, 2024
e0cae11
Move to pytest style asserts
BenjaminBossan Feb 26, 2024
52c8d9b
Fix safe merging code
BenjaminBossan Feb 26, 2024
977c84b
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Mar 11, 2024
96d376d
No need to set bias for MHA anymore, see #1530
BenjaminBossan Mar 11, 2024
0c17476
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Mar 26, 2024
4b8db0c
Fix style
BenjaminBossan Mar 26, 2024
7e91712
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan May 21, 2024
e12070b
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Jul 25, 2024
7b6c7cb
Remove duplicate merge
BenjaminBossan Jul 25, 2024
e6ab8ed
Raise error for multi adapter batch inference
BenjaminBossan Jul 25, 2024
8ec6c3c
Raise error for DoRA + MHA
BenjaminBossan Jul 25, 2024
f6ba465
Fix error when adding multiple adapters to MHA
BenjaminBossan Jul 25, 2024
fb18886
Better way of param initialization
BenjaminBossan Jul 26, 2024
4ff2ec3
Add tests for broken loading and workaround
BenjaminBossan Jul 26, 2024
d1f6ab2
make style
BenjaminBossan Jul 26, 2024
65363be
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Sep 3, 2024
7ba2e68
Fix wrong merge conflict resolution in test
BenjaminBossan Sep 4, 2024
6ef04b0
Ensure that base weights have requires_grad False
BenjaminBossan Sep 4, 2024
07c7240
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Sep 4, 2024
cc3ac3d
Remove xpass-ing test
BenjaminBossan Sep 4, 2024
03c466f
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Sep 12, 2024
e558caa
MAINT: Give stale bot permissions for PRs too (#2064)
BenjaminBossan Sep 12, 2024
38f4a98
ENH BOFT don't save boft_P buffer (#2050)
sywangyi Sep 13, 2024
7e5c61d
FIX Command line args in PiSSA preprocess (#2053)
keakon Sep 13, 2024
183bf52
MNT Update deprecated evaluation_strategy (#1664)
muellerzr Sep 13, 2024
b970607
ENH Multi adapters in same batch: modules_to_save (#1990)
saeid93 Sep 17, 2024
732e8e7
FIX Bug that prevents BOFT from loading 2 adapters (#2068)
BenjaminBossan Sep 18, 2024
79e2b38
TST Skip some quantization tests on XPU (#2074)
faaany Sep 18, 2024
61e6934
Improve test coverage for initialization of MHA
BenjaminBossan Sep 18, 2024
ced2f15
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Oct 14, 2024
4c31bbc
Fix bug with unloading multihead attention layer
BenjaminBossan Oct 21, 2024
1dbb9a5
Fix bug in unloading
BenjaminBossan Oct 22, 2024
e094234
Fix for low_cpu_mem_usage
BenjaminBossan Nov 1, 2024
e90af48
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Nov 1, 2024
30a08e7
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Nov 1, 2024
09f5ea6
Add tests for init_empty_weights
BenjaminBossan Nov 26, 2024
6a83bd7
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Nov 26, 2024
3b0471a
Merge branch 'main' into feat-add-lora-multihead-attention
BenjaminBossan Dec 9, 2024
465a85e
Add MHA to modules unsupported by EVA
BenjaminBossan Dec 9, 2024
266f9da
Add comment on why/how empty init works
BenjaminBossan Jan 6, 2025
39e755e
Expose attributes of underlying MHA module
BenjaminBossan Jan 6, 2025
4857858
Apply suggestions from code review
BenjaminBossan Jan 6, 2025
74cbba6
Remove trailing whitespace
BenjaminBossan Jan 6, 2025
14deb9f
Linting..
BenjaminBossan Jan 6, 2025
ba2a8dd
Reviewer comment: Add comments for clarification
BenjaminBossan Jan 8, 2025
ac10b18
Reviewer feedback: Remove q_proj_weight
BenjaminBossan Jan 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Reviewer feedback: removed another unnecessary arg
  • Loading branch information
BenjaminBossan committed Jan 9, 2024
commit add1f5169fc4a35549ce7db4acfa48dafd7e5636
2 changes: 0 additions & 2 deletions src/peft/tuners/lora/layer.py
Original file line number Diff line number Diff line change
Expand Up @@ -706,7 +706,6 @@ def __init__(
r: int = 0,
lora_alpha: int = 1,
lora_dropout: float = 0.0,
is_target_conv_1d_layer: bool = False,
init_lora_weights: Union[bool, str] = True,
use_rslora: bool = False,
**kwargs,
Expand All @@ -720,7 +719,6 @@ def __init__(

self._active_adapter = adapter_name
self.update_layer(adapter_name, r, lora_alpha, lora_dropout, init_lora_weights, use_rslora)
self.is_target_conv_1d_layer = is_target_conv_1d_layer

def merge(self, safe_merge: bool = False, adapter_names: Optional[List[str]] = None) -> None:
"""
Expand Down
4 changes: 0 additions & 4 deletions tests/test_initialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,6 @@ def test_lora_linear_init_gaussian(self):
normal = self.get_normal(0.0, 1 / config.r)
_, p_value = stats.kstest(weight_A.detach().flatten().cpu().numpy(), normal.flatten().cpu().numpy())

# import matplotlib.pyplot as plt
# x = weight_A.detach().flatten().cpu().numpy()
# breakpoint()

self.assertGreater(p_value, 0.5)

# check that weight A is *not* from a uniform distribution
Expand Down