scripts: add script to sync kernel changes #13

anakryiko · 2019-02-16T23:06:24Z

This script automates the process of applying libbpf-relevant changes
from kernel repository on top of current state of libbpf repository.

It uses CHECKPOINT-COMMIT file to keep track of last commit in kernel
repo up to which libbpf is in sync with. If there are any new libbpf
changes in kernel repository, script extracts them, preserving original
commit metadata. It also creates a "sync commit" using cover letter as
a template, which nicely summarizes changes since last sync with kernel.

Usage: ./scripts/sync-kernel.sh <linux-repo> <libbpf-repo>

If it succeeds, script will create a bunch of local commits in
in separate branch, which can be easily pushed into github
to create a pull request.

Script tries to clean up after itself, except in case of failure. But it
doesn't clean up timestamped branches it creates in both kernel and
libbpf repositories for now. We can add that later.

Signed-off-by: Andrii Nakryiko [email protected]

yonghong-song · 2019-02-17T01:19:19Z

scripts/sync-kernel.sh

+echo "PATCHSET: /tmp/libbpf-${SUFFIX}.patchset"
+
+cd ${LIBBPF_REPO}
+git co -b libbpf-sync-${SUFFIX}


I assume this is git checkout?

Ah, yes, it's my local alias. I'm so used to it that didn't even realize it's not standard feature, will fix.

yonghong-song · 2019-02-17T01:22:46Z

Thanks for implementation. This is in the right direction for automation!

I tried to add a few artificial patches in bpf-next and got the following failures:

Ref 'refs/heads/libbpf-2019-02-17T01-17-25.489Z' was rewritten
Rewrite a63440df54bf4159b9389a70450579f990095c29 (3/3) (0 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/libbpf-2019-02-17T01-17-25.489Z' was rewritten
/tmp/libbpf-2019-02-17T01-17-25.489Z.patches/0000-cover-letter.patch
/tmp/libbpf-2019-02-17T01-17-25.489Z.patches/0001-fix1.patch
/tmp/libbpf-2019-02-17T01-17-25.489Z.patches/0002-fix-3.patch
[libbpf-2019-02-17T01-17-25.489Z b5ea6613c83e] sync: latest libbpf changes from kernel
 1 file changed, 1 insertion(+)
 create mode 100644 CHECKPOINT-COMMIT
PATCHSET: /tmp/libbpf-2019-02-17T01-17-25.489Z.patchset
Switched to a new branch 'libbpf-sync-2019-02-17T01-17-25.489Z'
Applying: fix1
error: patch failed: src/btf.h:4
error: src/btf.h: patch does not apply
Patch failed at 0001 fix1
Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
-bash-4.4$

The command line I tried:

-bash-4.4$ ./scripts/sync-kernel.sh /home/yhs/work/net-next .

in the libbpf repo directory.

anakryiko · 2019-02-17T01:29:56Z

Can you publish full output and patches you applied on top of bpf-next, I'll try to repro locally. Also what was HEAD of libbpf and kernel?
This happened to me only when master on libbpf and linux repos were out of sync (thus patches didn't apply clean). When I was testing I was even able to re-apply quite a long history of commits on top of some older commits in libbpf, but the trickiest part was correlating commits in linux and libbpf.

yonghong-song · 2019-02-17T01:34:08Z

The kernel part:

commit 6bcf3db00a5931ccbd78730d85ab6124f826d9ba (HEAD -> copy)
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:09:37 2019 -0800

    patch3
    
    Signed-off-by: Yonghong Song <[email protected]>

commit add7beebf02a190ccd74ac4a221105090408e9e9
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:09:03 2019 -0800

    fix 3
    
    Signed-off-by: Yonghong Song <[email protected]>

commit 0ccb09e4c26f3d0189f689b3596f6c4b6655a35a
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:08:33 2019 -0800

    fix1
    
    Signed-off-by: Yonghong Song <[email protected]>

commit b251f9f63a3bea7864bf627960926d978e97814d (bpf-next/master, bpf-next2)
Author: Peter Oskolkov <[email protected]>
Date:   Fri Feb 15 09:51:35 2019 -0800

    bpf: make LWTUNNEL_BPF dependent on INET

Individual commits:

-bash-4.4$ git show 6bcf3db00a5931ccbd78730d85ab6124f826d9ba
commit 6bcf3db00a5931ccbd78730d85ab6124f826d9ba (HEAD -> copy)
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:09:37 2019 -0800

    patch3
    
    Signed-off-by: Yonghong Song <[email protected]>

diff --git a/t.c b/t.c
new file mode 100644
index 000000000000..f349bb94c37e
--- /dev/null
+++ b/t.c
@@ -0,0 +1 @@
+iinwe filees
-bash-4.4$ git show add7beebf02a190ccd74ac4a221105090408e9e9
commit add7beebf02a190ccd74ac4a221105090408e9e9
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:09:03 2019 -0800

    fix 3
    
    Signed-off-by: Yonghong Song <[email protected]>

diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c
index ade1c32fb083..e96428c9a47b 100644
--- a/tools/lib/bpf/btf.c
+++ b/tools/lib/bpf/btf.c
@@ -13,6 +13,8 @@
 #include "libbpf.h"
 #include "libbpf_util.h"
 
+#error no
+
 #define max(a, b) ((a) > (b) ? (a) : (b))
 #define min(a, b) ((a) < (b) ? (a) : (b))
 
-bash-4.4$ git show 0ccb09e4c26f3d0189f689b3596f6c4b6655a35a
commit 0ccb09e4c26f3d0189f689b3596f6c4b6655a35a
Author: Yonghong Song <[email protected]>
Date:   Sat Feb 16 17:08:33 2019 -0800

    fix1
    
    Signed-off-by: Yonghong Song <[email protected]>

diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h
index 94bbc249b0f1..c57d42385806 100644
--- a/tools/lib/bpf/btf.h
+++ b/tools/lib/bpf/btf.h
@@ -4,6 +4,8 @@
 #ifndef __LIBBPF_BTF_H
 #define __LIBBPF_BTF_H
 
+#error no
+
 #include <linux/types.h>
 
 #ifdef __cplusplus
-bash-4.4$

yonghong-song · 2019-02-17T01:34:39Z

Next time I will remember to go to a repo I can push to github.

anakryiko · 2019-02-17T01:37:41Z

What was sha on libbpf side?

yonghong-song · 2019-02-17T01:41:40Z

libbpf side

commit 8521d84455f7bfd17b088ea642767e370939185a (HEAD -> script)
Author: Andrii Nakryiko <[email protected]>
Date:   Sat Feb 16 14:57:53 2019 -0800

    scripts: add script to sync kernel changes
    
    This script automates the process of applying libbpf-relevant changes
    from kernel repository on top of current state of libbpf repository.
    
    It uses CHECKPOINT-COMMIT file to keep track of last commit in kernel
    repo up to which libbpf is in sync with. If there are any new libbpf
    changes in kernel repository, script extracts them, preserving original
    commit metadata. It also creates a "sync commit" using cover letter as
    a template, which nicely summarizes changes since last sync with kernel.
    
    Usage: ./scripts/sync-kernel.sh <linux-repo> <libbpf-repo>
    
    If it succeeds, script will create a bunch of local commits in
    <libbpf-repo> in separate branch, which can be easily pushed into github
    to create a pull request.
    
    Script tries to clean up after itself, except in case of failure. But it
    doesn't clean up timestamped branches it creates in both kernel and
    libbpf repositories for now. We can add that later.
    
    Signed-off-by: Andrii Nakryiko <[email protected]>

commit f0bcba631dec4540fc6ab2cd0a0923a111cf4cf2
Author: yonghong-song <[email protected]>
Date:   Thu Feb 7 21:58:07 2019 -0800

    sync with latest bpf-next (#10)
    
    sync with latest bpf-next. tested with fb internal testcase and bcc.
    
    Signed-off-by: Yonghong Song <[email protected]>

commit b19c6dcf623a7adc9e538ddbe2964c2f58dd2417
Author: Yonghong Song <[email protected]>
Date:   Mon Feb 4 19:42:31 2019 -0800

    sync with bpf-next
    
    Sync with the following bug fix:
      commit a8a1f7d09cfc7e18874786c7634c9e71384fcd4e (HEAD -> bpf-next2, bpf-next/master)
      Author: Stanislav Fomichev <[email protected]>
      Date:   Mon Feb 4 16:20:55 2019 -0800
    
        libbpf: fix libbpf_print
    
        With the recent print rework we now have the following problem:
        pr_{warning,info,debug} expand to __pr which calls libbpf_print.
        libbpf_print does va_start and calls __libbpf_pr with va_list argument.
        In __base_pr we again do va_start. Because the next argument is a
        va_list, we don't get correct pointer to the argument (and print noting
        in my case, I don't know why it doesn't crash tbh).
    
        ......
    
        Signed-off-by: Stanislav Fomichev <[email protected]>
    
    Signed-off-by: Yonghong Song <[email protected]>

anakryiko · 2019-02-17T01:51:19Z

ah, I get it, it's the logic of updating CHECKPOINT-COMMIT file, it never exists in linux repo, I have a simpler idea on how to handle that. Thanks for catching this!

anakryiko · 2019-02-17T02:19:50Z

ahah, so there are two problems:

CHECKPOINT-COMMIT should be updated/commited in libbpf repo, not in linux repo (adding it in linux makes it harder to generate correct patch).
You used "." as libbpf-repo, script doesn't handle that properly. Try with absolute path for now, I'll think how to properly fix relative paths meanwhile.

I fixed first in updated pull request. Will fix second soon.

BTW, if you are going to try again, keep in mind that you might need to do "git am --abort" in linux repo, and checkout your old HEAD (copy branch).

This script automates the process of applying libbpf-relevant changes from kernel repository on top of current state of libbpf repository. It uses CHECKPOINT-COMMIT file to keep track of last commit in kernel repo up to which libbpf is in sync with. If there are any new libbpf changes in kernel repository, script extracts them, preserving original commit metadata. It also creates a "sync commit" using cover letter as a template, which nicely summarizes changes since last sync with kernel. Usage: ./scripts/sync-kernel.sh <linux-repo> <libbpf-repo> If it succeeds, script will create a bunch of local commits in <libbpf-repo> in separate branch, which can be easily pushed into github to create a pull request. Script tries to clean up after itself, except in case of failure. But it doesn't clean up timestamped branches it creates in both kernel and libbpf repositories for now. We can add that later. Signed-off-by: Andrii Nakryiko <[email protected]>

anakryiko · 2019-02-17T02:29:39Z

Ok, fixed second problem as well by switching back to original working dir and then doing cd LIBBPF_REPO. Now works as expected:

Ref 'refs/heads/libbpf-2019-02-17T02-25-37.047Z' was rewritten
Rewrite 146c3786b73a6b20c524feaebd7196203dea5533 (3/3) (0 seconds passed, remaining 0 predicted)
Ref 'refs/heads/libbpf-2019-02-17T02-25-37.047Z' was rewritten
/tmp/libbpf-2019-02-17T02-25-37.047Z.patches/0000-cover-letter.patch
/tmp/libbpf-2019-02-17T02-25-37.047Z.patches/0001-fix-1.patch
/tmp/libbpf-2019-02-17T02-25-37.047Z.patches/0002-fix-3.patch
PATCHSET: /tmp/libbpf-2019-02-17T02-25-37.047Z.patchset
Switched to a new branch 'libbpf-sync-2019-02-17T02-25-37.047Z'
Applying: fix 1
Applying: fix 3
[libbpf-sync-2019-02-17T02-25-37.047Z 022af37] sync: latest libbpf changes from kernel
 1 file changed, 1 insertion(+), 1 deletion(-)
Success! 2 commits applied.

State after running script:

commit 022af3793929ce6ab4abd1f178b4b67e03039fba (HEAD -> libbpf-sync-2019-02-17T02-25-37.047Z)
Author: Andrii Nakryiko <[email protected]>
Date:   Sat Feb 16 18:26:12 2019 -0800

    sync: latest libbpf changes from kernel

    Syncing latest libbpf commits from kernel repository.
    Baseline commit:   789f6bab849e04ea029c09b81dc8401dc0268cf9
    Checkpoint commit: 6911e801fa1da0e54b200af8df6908b9fc1c391d

    Andrii Nakryiko (2):
      fix 1
      fix 3

     src/btf.c | 2 ++
     src/btf.h | 2 ++
     2 files changed, 4 insertions(+)

    --
    2.17.1

commit 7825e77a9f04bd272b6aa6e5bf6ffdd239361da5
Author: Andrii Nakryiko <[email protected]>
Date:   Sat Feb 16 17:44:23 2019 -0800

    fix 3

commit 480c282161177b66931fe5175e88115f7394f791
Author: Andrii Nakryiko <[email protected]>
Date:   Sat Feb 16 17:41:06 2019 -0800

    fix 1

commit 88d35a3d80b52017bba32ecb3a4a9d412ac45a9d (kernel-sync-script)
Author: Andrii Nakryiko <[email protected]>
Date:   Sat Feb 16 14:57:53 2019 -0800

    scripts: add script to sync kernel changes

<SNIP>

4ast · 2019-02-17T02:34:59Z

scripts/sync-kernel.sh

+git rebase --onto ${SQUASHED_BASELINE_COMMIT} ${BASELINE_COMMIT} libbpf-${SUFFIX}
+
+# Move all libbpf files into libbpf-projection directory
+git filter-branch --prune-empty -f --tree-filter '                                                                             \


didn't know such git command exist :)

it's a "with great power comes great responsibility" type of command :)

anakryiko · 2019-02-17T03:03:13Z

@yonghong-song, you can check how sync commit looks like here: https://github.com/libbpf/libbpf/compare/master...anakryiko:libbpf-sync-2019-02-17T02-58-38.109Z?expand=1

I'm not yet creating that pull request, because we should probably land this one first and then I'll rebase (otherwise we'll have conflicts on CHECKPOINT-COMMIT file).

yonghong-song · 2019-02-17T06:09:23Z

I got a failure like below:

-bash-4.4$ ./scripts/sync-kernel.sh /home/yhs/work/net-next /home/yhs/work/libbpf
WORKDIR:     /home/yhs/work/libbpf
LINUX REPO:  /home/yhs/work/net-next
LIBBPF REPO: /home/yhs/work/libbpf
SUFFIX: 2019-02-17T06-04-31.687Z
BASELINE COMMIT: 789f6bab849e04ea029c09b81dc8401dc0268cf9
CHECKPOINT COMMIT: 6bcf3db00a5931ccbd78730d85ab6124f826d9ba
SQUASHED BASELINE COMMIT: 81818e2fe001f6e7b23d01ef4e3d9c38655ecfa8
It looks like 'git am' is in progress. Cannot rebase.
-bash-4.4$

Any idea what's the issue? I have the same bpf-next hack earlier.

anakryiko · 2019-02-17T06:14:50Z

Yeah, due to that initial bug, your linux repository now has git am in progress. Just execute git am --abort && git checkout master in your linux repo.

yonghong-song · 2019-02-17T06:22:23Z

thanks. this time succeeded! Look likes need to learn a few git advanced commands :-)

Fixes ``` ./out/bpf-object-fuzzer: Running 1 inputs 1 time(s) each. Running: CORPUS/036ff286c13e4590646c7ef59435ec642432da8e elf_begin.c:232:20: runtime error: member access within misaligned address 0x000001655e71 for type 'Elf64_Shdr', which requires 8 byte alignment 0x000001655e71: note: pointer points here 00 00 00 7f 45 4c 46 02 02 01 00 00 00 07 fb 00 1d 00 00 6c 69 63 65 42 fb 00 41 00 57 03 00 20 ^ #0 0x574d51 in get_shnum /home/libbpf/elfutils/libelf/elf_begin.c:232:20 #1 0x574d51 in file_read_elf /home/libbpf/elfutils/libelf/elf_begin.c:296:19 #2 0x569c2c in __libelf_read_mmaped_file /home/libbpf/elfutils/libelf/elf_begin.c:559:14 #3 0x58e812 in elf_memory /home/libbpf/elfutils/libelf/elf_memory.c:49:10 #4 0x4905b4 in bpf_object__elf_init /home/libbpf/src/libbpf.c:1255:9 #5 0x4905b4 in bpf_object_open /home/libbpf/src/libbpf.c:7104:8 libbpf#6 0x49144e in bpf_object__open_mem /home/libbpf/src/libbpf.c:7171:20 libbpf#7 0x483018 in LLVMFuzzerTestOneInput /home/libbpf/fuzz/bpf-object-fuzzer.c:16:8 libbpf#8 0x439389 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x439389) libbpf#9 0x419e2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x419e2f) libbpf#10 0x421aee in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/libbpf/out/bpf-object-fuzzer+0x421aee) libbpf#11 0x410f96 in main (/home/libbpf/out/bpf-object-fuzzer+0x410f96) libbpf#12 0x7f153e21255f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f) libbpf#13 0x7f153e21260b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b) libbpf#14 0x410fe4 in _start (/home/libbpf/out/bpf-object-fuzzer+0x410fe4) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior elf_begin.c:232:20 in ``` and ``` ./out/bpf-object-fuzzer: Running 1 inputs 1 time(s) each. Running: CORPUS/446b578d82c47fe177de6fd675f4cb6bae8d1ea9 elf_begin.c:485:40: runtime error: addition of unsigned offset to 0x000002277e70 overflowed to 0x0000021d7e6f #0 0x5748f1 in file_read_elf /home/libbpf/elfutils/libelf/elf_begin.c:485:40 #1 0x569c2c in __libelf_read_mmaped_file /home/libbpf/elfutils/libelf/elf_begin.c:559:14 #2 0x58e812 in elf_memory /home/libbpf/elfutils/libelf/elf_memory.c:49:10 #3 0x4905b4 in bpf_object__elf_init /home/libbpf/src/libbpf.c:1255:9 #4 0x4905b4 in bpf_object_open /home/libbpf/src/libbpf.c:7104:8 #5 0x49144e in bpf_object__open_mem /home/libbpf/src/libbpf.c:7171:20 libbpf#6 0x483018 in LLVMFuzzerTestOneInput /home/libbpf/fuzz/bpf-object-fuzzer.c:16:8 libbpf#7 0x439389 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x439389) libbpf#8 0x419e2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x419e2f) libbpf#9 0x421aee in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/libbpf/out/bpf-object-fuzzer+0x421aee) libbpf#10 0x410f96 in main (/home/libbpf/out/bpf-object-fuzzer+0x410f96) libbpf#11 0x7f753e38255f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f) libbpf#12 0x7f753e38260b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b) libbpf#13 0x410fe4 in _start (/home/libbpf/out/bpf-object-fuzzer+0x410fe4) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior elf_begin.c:485:40 in ```

Fixes ``` ./out/bpf-object-fuzzer: Running 1 inputs 1 time(s) each. Running: CORPUS/036ff286c13e4590646c7ef59435ec642432da8e elf_begin.c:232:20: runtime error: member access within misaligned address 0x000001655e71 for type 'Elf64_Shdr', which requires 8 byte alignment 0x000001655e71: note: pointer points here 00 00 00 7f 45 4c 46 02 02 01 00 00 00 07 fb 00 1d 00 00 6c 69 63 65 42 fb 00 41 00 57 03 00 20 ^ #0 0x574d51 in get_shnum /home/libbpf/elfutils/libelf/elf_begin.c:232:20 #1 0x574d51 in file_read_elf /home/libbpf/elfutils/libelf/elf_begin.c:296:19 #2 0x569c2c in __libelf_read_mmaped_file /home/libbpf/elfutils/libelf/elf_begin.c:559:14 #3 0x58e812 in elf_memory /home/libbpf/elfutils/libelf/elf_memory.c:49:10 #4 0x4905b4 in bpf_object__elf_init /home/libbpf/src/libbpf.c:1255:9 #5 0x4905b4 in bpf_object_open /home/libbpf/src/libbpf.c:7104:8 #6 0x49144e in bpf_object__open_mem /home/libbpf/src/libbpf.c:7171:20 #7 0x483018 in LLVMFuzzerTestOneInput /home/libbpf/fuzz/bpf-object-fuzzer.c:16:8 #8 0x439389 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x439389) #9 0x419e2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x419e2f) #10 0x421aee in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/libbpf/out/bpf-object-fuzzer+0x421aee) #11 0x410f96 in main (/home/libbpf/out/bpf-object-fuzzer+0x410f96) #12 0x7f153e21255f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f) #13 0x7f153e21260b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b) #14 0x410fe4 in _start (/home/libbpf/out/bpf-object-fuzzer+0x410fe4) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior elf_begin.c:232:20 in ``` and ``` ./out/bpf-object-fuzzer: Running 1 inputs 1 time(s) each. Running: CORPUS/446b578d82c47fe177de6fd675f4cb6bae8d1ea9 elf_begin.c:485:40: runtime error: addition of unsigned offset to 0x000002277e70 overflowed to 0x0000021d7e6f #0 0x5748f1 in file_read_elf /home/libbpf/elfutils/libelf/elf_begin.c:485:40 #1 0x569c2c in __libelf_read_mmaped_file /home/libbpf/elfutils/libelf/elf_begin.c:559:14 #2 0x58e812 in elf_memory /home/libbpf/elfutils/libelf/elf_memory.c:49:10 #3 0x4905b4 in bpf_object__elf_init /home/libbpf/src/libbpf.c:1255:9 #4 0x4905b4 in bpf_object_open /home/libbpf/src/libbpf.c:7104:8 #5 0x49144e in bpf_object__open_mem /home/libbpf/src/libbpf.c:7171:20 #6 0x483018 in LLVMFuzzerTestOneInput /home/libbpf/fuzz/bpf-object-fuzzer.c:16:8 #7 0x439389 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x439389) #8 0x419e2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/libbpf/out/bpf-object-fuzzer+0x419e2f) #9 0x421aee in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/libbpf/out/bpf-object-fuzzer+0x421aee) #10 0x410f96 in main (/home/libbpf/out/bpf-object-fuzzer+0x410f96) #11 0x7f753e38255f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f) #12 0x7f753e38260b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b) #13 0x410fe4 in _start (/home/libbpf/out/bpf-object-fuzzer+0x410fe4) SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior elf_begin.c:485:40 in ```

ASAN reports an use-after-free in btf_dump_name_dups: ERROR: AddressSanitizer: heap-use-after-free on address 0xffff927006db at pc 0xaaaab5dfb618 bp 0xffffdd89b890 sp 0xffffdd89b928 READ of size 2 at 0xffff927006db thread T0 #0 0xaaaab5dfb614 in __interceptor_strcmp.part.0 (test_progs+0x21b614) libbpf#1 0xaaaab635f144 in str_equal_fn tools/lib/bpf/btf_dump.c:127 libbpf#2 0xaaaab635e3e0 in hashmap_find_entry tools/lib/bpf/hashmap.c:143 libbpf#3 0xaaaab635e72c in hashmap__find tools/lib/bpf/hashmap.c:212 libbpf#4 0xaaaab6362258 in btf_dump_name_dups tools/lib/bpf/btf_dump.c:1525 libbpf#5 0xaaaab636240c in btf_dump_resolve_name tools/lib/bpf/btf_dump.c:1552 libbpf#6 0xaaaab6362598 in btf_dump_type_name tools/lib/bpf/btf_dump.c:1567 libbpf#7 0xaaaab6360b48 in btf_dump_emit_struct_def tools/lib/bpf/btf_dump.c:912 libbpf#8 0xaaaab6360630 in btf_dump_emit_type tools/lib/bpf/btf_dump.c:798 libbpf#9 0xaaaab635f720 in btf_dump__dump_type tools/lib/bpf/btf_dump.c:282 libbpf#10 0xaaaab608523c in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:236 libbpf#11 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 libbpf#12 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 libbpf#13 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 libbpf#14 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 libbpf#15 0xaaaab5d65990 (test_progs+0x185990) 0xffff927006db is located 11 bytes inside of 16-byte region [0xffff927006d0,0xffff927006e0) freed by thread T0 here: #0 0xaaaab5e2c7c4 in realloc (test_progs+0x24c7c4) libbpf#1 0xaaaab634f4a0 in libbpf_reallocarray tools/lib/bpf/libbpf_internal.h:191 libbpf#2 0xaaaab634f840 in libbpf_add_mem tools/lib/bpf/btf.c:163 libbpf#3 0xaaaab636643c in strset_add_str_mem tools/lib/bpf/strset.c:106 libbpf#4 0xaaaab6366560 in strset__add_str tools/lib/bpf/strset.c:157 libbpf#5 0xaaaab6352d70 in btf__add_str tools/lib/bpf/btf.c:1519 libbpf#6 0xaaaab6353e10 in btf__add_field tools/lib/bpf/btf.c:2032 libbpf#7 0xaaaab6084fcc in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:232 libbpf#8 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 libbpf#9 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 libbpf#10 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 libbpf#11 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 libbpf#12 0xaaaab5d65990 (test_progs+0x185990) previously allocated by thread T0 here: #0 0xaaaab5e2c7c4 in realloc (test_progs+0x24c7c4) libbpf#1 0xaaaab634f4a0 in libbpf_reallocarray tools/lib/bpf/libbpf_internal.h:191 libbpf#2 0xaaaab634f840 in libbpf_add_mem tools/lib/bpf/btf.c:163 libbpf#3 0xaaaab636643c in strset_add_str_mem tools/lib/bpf/strset.c:106 libbpf#4 0xaaaab6366560 in strset__add_str tools/lib/bpf/strset.c:157 libbpf#5 0xaaaab6352d70 in btf__add_str tools/lib/bpf/btf.c:1519 libbpf#6 0xaaaab6353ff0 in btf_add_enum_common tools/lib/bpf/btf.c:2070 libbpf#7 0xaaaab6354080 in btf__add_enum tools/lib/bpf/btf.c:2102 libbpf#8 0xaaaab6082f50 in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:162 libbpf#9 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 libbpf#10 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 libbpf#11 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 libbpf#12 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 libbpf#13 0xaaaab5d65990 (test_progs+0x185990) The reason is that the key stored in hash table name_map is a string address, and the string memory is allocated by realloc() function, when the memory is resized by realloc() later, the old memory may be freed, so the address stored in name_map references to a freed memory, causing use-after-free. Fix it by storing duplicated string address in name_map. Fixes: 919d2b1dbb07 ("libbpf: Allow modification of BTF and add btf__add_str API") Signed-off-by: Xu Kuohai <[email protected]> Signed-off-by: Andrii Nakryiko <[email protected]> Acked-by: Martin KaFai Lau <[email protected]> Link: https://lore.kernel.org/bpf/[email protected]

ASAN reports an use-after-free in btf_dump_name_dups: ERROR: AddressSanitizer: heap-use-after-free on address 0xffff927006db at pc 0xaaaab5dfb618 bp 0xffffdd89b890 sp 0xffffdd89b928 READ of size 2 at 0xffff927006db thread T0 #0 0xaaaab5dfb614 in __interceptor_strcmp.part.0 (test_progs+0x21b614) #1 0xaaaab635f144 in str_equal_fn tools/lib/bpf/btf_dump.c:127 #2 0xaaaab635e3e0 in hashmap_find_entry tools/lib/bpf/hashmap.c:143 #3 0xaaaab635e72c in hashmap__find tools/lib/bpf/hashmap.c:212 #4 0xaaaab6362258 in btf_dump_name_dups tools/lib/bpf/btf_dump.c:1525 #5 0xaaaab636240c in btf_dump_resolve_name tools/lib/bpf/btf_dump.c:1552 #6 0xaaaab6362598 in btf_dump_type_name tools/lib/bpf/btf_dump.c:1567 #7 0xaaaab6360b48 in btf_dump_emit_struct_def tools/lib/bpf/btf_dump.c:912 #8 0xaaaab6360630 in btf_dump_emit_type tools/lib/bpf/btf_dump.c:798 #9 0xaaaab635f720 in btf_dump__dump_type tools/lib/bpf/btf_dump.c:282 #10 0xaaaab608523c in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:236 #11 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 #12 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 #13 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 #14 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 #15 0xaaaab5d65990 (test_progs+0x185990) 0xffff927006db is located 11 bytes inside of 16-byte region [0xffff927006d0,0xffff927006e0) freed by thread T0 here: #0 0xaaaab5e2c7c4 in realloc (test_progs+0x24c7c4) #1 0xaaaab634f4a0 in libbpf_reallocarray tools/lib/bpf/libbpf_internal.h:191 #2 0xaaaab634f840 in libbpf_add_mem tools/lib/bpf/btf.c:163 #3 0xaaaab636643c in strset_add_str_mem tools/lib/bpf/strset.c:106 #4 0xaaaab6366560 in strset__add_str tools/lib/bpf/strset.c:157 #5 0xaaaab6352d70 in btf__add_str tools/lib/bpf/btf.c:1519 #6 0xaaaab6353e10 in btf__add_field tools/lib/bpf/btf.c:2032 #7 0xaaaab6084fcc in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:232 #8 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 #9 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 #10 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 #11 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 #12 0xaaaab5d65990 (test_progs+0x185990) previously allocated by thread T0 here: #0 0xaaaab5e2c7c4 in realloc (test_progs+0x24c7c4) #1 0xaaaab634f4a0 in libbpf_reallocarray tools/lib/bpf/libbpf_internal.h:191 #2 0xaaaab634f840 in libbpf_add_mem tools/lib/bpf/btf.c:163 #3 0xaaaab636643c in strset_add_str_mem tools/lib/bpf/strset.c:106 #4 0xaaaab6366560 in strset__add_str tools/lib/bpf/strset.c:157 #5 0xaaaab6352d70 in btf__add_str tools/lib/bpf/btf.c:1519 #6 0xaaaab6353ff0 in btf_add_enum_common tools/lib/bpf/btf.c:2070 #7 0xaaaab6354080 in btf__add_enum tools/lib/bpf/btf.c:2102 #8 0xaaaab6082f50 in test_btf_dump_incremental tools/testing/selftests/bpf/prog_tests/btf_dump.c:162 #9 0xaaaab6097530 in test_btf_dump tools/testing/selftests/bpf/prog_tests/btf_dump.c:875 #10 0xaaaab6314ed0 in run_one_test tools/testing/selftests/bpf/test_progs.c:1062 #11 0xaaaab631a0a8 in main tools/testing/selftests/bpf/test_progs.c:1697 #12 0xffff9676d214 in __libc_start_main ../csu/libc-start.c:308 #13 0xaaaab5d65990 (test_progs+0x185990) The reason is that the key stored in hash table name_map is a string address, and the string memory is allocated by realloc() function, when the memory is resized by realloc() later, the old memory may be freed, so the address stored in name_map references to a freed memory, causing use-after-free. Fix it by storing duplicated string address in name_map. Fixes: 919d2b1dbb07 ("libbpf: Allow modification of BTF and add btf__add_str API") Signed-off-by: Xu Kuohai <[email protected]> Signed-off-by: Andrii Nakryiko <[email protected]> Acked-by: Martin KaFai Lau <[email protected]> Link: https://lore.kernel.org/bpf/[email protected]

Original change: undetermined Change-Id: I32955427576e59c77aeb84dcaab6aa72f1862490

…cf7c79f Original change: undetermined Change-Id: If01a0204b1e18e197116dcf03de0f3ce7e07f69e

…cf7c79f am: 656f471 Original change: undetermined Change-Id: I201cf3dc6afcaba2094d05fa8df88b1709a9ea29

When the binary path is excessively long, the generated probe_name in libbpf exceeds the kernel's MAX_EVENT_NAME_LEN limit (64 bytes). This causes legacy uprobe event attachment to fail with error code -22. The fix reorders the fields to place the unique ID before the name. This ensures that even if truncation occurs via snprintf, the unique ID remains intact, preserving event name uniqueness. Additionally, explicit checks with MAX_EVENT_NAME_LEN are added to enforce length constraints. Before Fix: ./test_progs -t attach_probe/kprobe-long_name ...... libbpf: failed to add legacy kprobe event for 'bpf_testmod_looooooooooooooooooooooooooooooong_name+0x0': -EINVAL libbpf: prog 'handle_kprobe': failed to create kprobe 'bpf_testmod_looooooooooooooooooooooooooooooong_name+0x0' perf event: -EINVAL test_attach_kprobe_long_event_name:FAIL:attach_kprobe_long_event_name unexpected error: -22 test_attach_probe:PASS:uprobe_ref_ctr_cleanup 0 nsec libbpf#13/11 attach_probe/kprobe-long_name:FAIL libbpf#13 attach_probe:FAIL ./test_progs -t attach_probe/uprobe-long_name ...... libbpf: failed to add legacy uprobe event for /root/linux-bpf/bpf-next/tools/testing/selftests/bpf/test_progs:0x13efd9: -EINVAL libbpf: prog 'handle_uprobe': failed to create uprobe '/root/linux-bpf/bpf-next/tools/testing/selftests/bpf/test_progs:0x13efd9' perf event: -EINVAL test_attach_uprobe_long_event_name:FAIL:attach_uprobe_long_event_name unexpected error: -22 libbpf#13/10 attach_probe/uprobe-long_name:FAIL libbpf#13 attach_probe:FAIL After Fix: ./test_progs -t attach_probe/uprobe-long_name libbpf#13/10 attach_probe/uprobe-long_name:OK libbpf#13 attach_probe:OK Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED ./test_progs -t attach_probe/kprobe-long_name libbpf#13/11 attach_probe/kprobe-long_name:OK libbpf#13 attach_probe:OK Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED Fixes: 46ed5fc33db9 ("libbpf: Refactor and simplify legacy kprobe code") Fixes: cc10623c6810 ("libbpf: Add legacy uprobe attaching support") Signed-off-by: Hengqi Chen <[email protected]> Signed-off-by: Feng Yang <[email protected]> Signed-off-by: Andrii Nakryiko <[email protected]> Link: https://lore.kernel.org/bpf/[email protected]

When the binary path is excessively long, the generated probe_name in libbpf exceeds the kernel's MAX_EVENT_NAME_LEN limit (64 bytes). This causes legacy uprobe event attachment to fail with error code -22. The fix reorders the fields to place the unique ID before the name. This ensures that even if truncation occurs via snprintf, the unique ID remains intact, preserving event name uniqueness. Additionally, explicit checks with MAX_EVENT_NAME_LEN are added to enforce length constraints. Before Fix: ./test_progs -t attach_probe/kprobe-long_name ...... libbpf: failed to add legacy kprobe event for 'bpf_testmod_looooooooooooooooooooooooooooooong_name+0x0': -EINVAL libbpf: prog 'handle_kprobe': failed to create kprobe 'bpf_testmod_looooooooooooooooooooooooooooooong_name+0x0' perf event: -EINVAL test_attach_kprobe_long_event_name:FAIL:attach_kprobe_long_event_name unexpected error: -22 test_attach_probe:PASS:uprobe_ref_ctr_cleanup 0 nsec #13/11 attach_probe/kprobe-long_name:FAIL #13 attach_probe:FAIL ./test_progs -t attach_probe/uprobe-long_name ...... libbpf: failed to add legacy uprobe event for /root/linux-bpf/bpf-next/tools/testing/selftests/bpf/test_progs:0x13efd9: -EINVAL libbpf: prog 'handle_uprobe': failed to create uprobe '/root/linux-bpf/bpf-next/tools/testing/selftests/bpf/test_progs:0x13efd9' perf event: -EINVAL test_attach_uprobe_long_event_name:FAIL:attach_uprobe_long_event_name unexpected error: -22 #13/10 attach_probe/uprobe-long_name:FAIL #13 attach_probe:FAIL After Fix: ./test_progs -t attach_probe/uprobe-long_name #13/10 attach_probe/uprobe-long_name:OK #13 attach_probe:OK Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED ./test_progs -t attach_probe/kprobe-long_name #13/11 attach_probe/kprobe-long_name:OK #13 attach_probe:OK Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED Fixes: 46ed5fc33db9 ("libbpf: Refactor and simplify legacy kprobe code") Fixes: cc10623c6810 ("libbpf: Add legacy uprobe attaching support") Signed-off-by: Hengqi Chen <[email protected]> Signed-off-by: Feng Yang <[email protected]> Signed-off-by: Andrii Nakryiko <[email protected]> Link: https://lore.kernel.org/bpf/[email protected]

yonghong-song reviewed Feb 17, 2019

View reviewed changes

anakryiko force-pushed the kernel-sync-script branch from 8521d84 to 4091954 Compare February 17, 2019 02:03

anakryiko force-pushed the kernel-sync-script branch from 4091954 to 88d35a3 Compare February 17, 2019 02:27

4ast reviewed Feb 17, 2019

View reviewed changes

yonghong-song merged commit 8e42f42 into libbpf:master Feb 17, 2019

anakryiko deleted the kernel-sync-script branch February 17, 2019 06:33

danielocfb pushed a commit to danielocfb/libbpf that referenced this pull request Feb 15, 2023

scripts: add script to sync kernel changes (libbpf#13) am: 8e42f42

cf7c79f

Original change: undetermined Change-Id: I32955427576e59c77aeb84dcaab6aa72f1862490

danielocfb pushed a commit to danielocfb/libbpf that referenced this pull request Feb 15, 2023

scripts: add script to sync kernel changes (libbpf#13) am: 8e42f42 am: …

656f471

…cf7c79f Original change: undetermined Change-Id: If01a0204b1e18e197116dcf03de0f3ce7e07f69e

danielocfb pushed a commit to danielocfb/libbpf that referenced this pull request Feb 15, 2023

scripts: add script to sync kernel changes (libbpf#13) am: 8e42f42 am: …

e21e2d9

…cf7c79f am: 656f471 Original change: undetermined Change-Id: I201cf3dc6afcaba2094d05fa8df88b1709a9ea29

originalvojak mentioned this pull request Sep 8, 2023

Failing to attach USDT program to malloc #726

Closed

scripts: add script to sync kernel changes #13

scripts: add script to sync kernel changes #13

Uh oh!

Conversation

anakryiko commented Feb 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yonghong-song Feb 17, 2019

Choose a reason for hiding this comment

Uh oh!

anakryiko Feb 17, 2019

Choose a reason for hiding this comment

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

4ast Feb 17, 2019

Choose a reason for hiding this comment

Uh oh!

anakryiko Feb 17, 2019

Choose a reason for hiding this comment

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

anakryiko commented Feb 17, 2019

Uh oh!

yonghong-song commented Feb 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anakryiko commented Feb 16, 2019 •

edited

Loading