Skip to content

Remove hardcode flash-attn disable setting #2342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merrymercy merged 9 commits into from
Sep 1, 2023
Merged

Remove hardcode flash-attn disable setting #2342

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e6f8a22
Minor fix on code style
merrymercy Aug 24, 2023
e1f92ef
merge
Trangle Aug 24, 2023
adeaea4
Merge remote-tracking branch 'upstream/main'
Trangle Aug 25, 2023
41f0a74
Merge remote-tracking branch 'upstream/main'
Trangle Aug 25, 2023
b856594
Merge remote-tracking branch 'upstream/main'
Trangle Aug 27, 2023
2714582
Merge remote-tracking branch 'upstream/main'
Trangle Aug 28, 2023
43b8e82
Merge remote-tracking branch 'upstream/main'
Trangle Aug 29, 2023
f0b64ab
Merge remote-tracking branch 'upstream/main'
Trangle Aug 30, 2023
0aaa87e
update
Trangle Aug 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion fastchat/model/model_adapter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1339,7 +1339,8 @@ def load_model(self, model_path: str, from_pretrained_kwargs: dict):
model_path,
trust_remote_code=True,
)
config.use_flash_attn = False
# NOTE: if you use the old version of model file, please remove the comments below
# config.use_flash_attn = False
config.fp16 = True
generation_config = GenerationConfig.from_pretrained(
model_path, trust_remote_code=True
Expand Down