Skip to content

Rwkv v4 merge main #371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
CPFLAME wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Rwkv v4 merge main #371

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
3cfa0ac
update
QiaolingChen00 Aug 4, 2022
22fe34d
update
QiaolingChen00 Aug 4, 2022
36e8c08
refine code for running && replace RWKV_V4 to RWKV_v4
CPFLAME Aug 4, 2022
40cfb28
fix tensor parallel not work
CPFLAME Aug 4, 2022
853e383
add wkv and changeunwork big parain wkv
QiaolingChen00 Aug 10, 2022
a730659
update wkv
QiaolingChen00 Aug 12, 2022
7e16bcb
update
QiaolingChen00 Aug 12, 2022
ad05984
update
QiaolingChen00 Aug 16, 2022
3532271
uadd checkpoint config to reduce memo
QiaolingChen00 Aug 18, 2022
8e00484
revise model
QiaolingChen00 Aug 18, 2022
75c9489
fix loss not align bug
BBuf Aug 25, 2022
d87e565
reverse
BBuf Aug 25, 2022
bf52123
Merge branch 'main' of github.com:Oneflow-Inc/libai into rwkv_v4_merg…
CPFLAME Aug 26, 2022
0796847
fix rwkv libai code
CPFLAME Aug 26, 2022
f135eec
fix rwkv libai code step2
CPFLAME Aug 26, 2022
8c16400
flake code
CPFLAME Aug 26, 2022
2e7c81f
del file
CPFLAME Aug 26, 2022
057e6e5
del write json in dataset.py
CPFLAME Aug 26, 2022
6faf9f2
del unuuse code
CPFLAME Aug 26, 2022
c8f8c6f
refine loss compare code
CPFLAME Aug 26, 2022
2966c35
fix config
CPFLAME Aug 26, 2022
d5d2b8e
flake code
CPFLAME Aug 26, 2022
a04ea65
add readme
CPFLAME Aug 26, 2022
a54fdfa
add bf16 in config
CPFLAME Aug 26, 2022
c38cf15
add error info in graph base
CPFLAME Aug 26, 2022
4c98f99
add training command in readme.md
CPFLAME Aug 26, 2022
0c788c1
test
Sep 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add bf16 in config
  • Loading branch information
@CPFLAME
CPFLAME committed Aug 26, 2022
commit a54fdfa04e0f47fed5d43b524d5180b49a25186e
5 changes: 4 additions & 1 deletion projects/RWKV_v4/configs/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,15 @@
train.train_iter = 0
train.train_epoch = 1

train.amp.enabled = True
train.amp.type = "bf16"

train.rdma_enabled = False
train.activation_checkpoint.enabled = True

train.input_placement_device = "cpu"
train.dist.data_parallel_size = 1
train.dist.tensor_parallel_size = 2
train.dist.tensor_parallel_size = 1
train.dist.pipeline_parallel_size = 1
train.dist.pipeline_num_layers = model.n_layer

Expand Down