Segmentation fault on Ubuntu 22.04.2 LTS

[andrey-admin]

Hello,

Got Segmentation fault (core dumped) when trying to run bpftune on Linux 5.19.0-1026-gcp kernel with:

[949460.456403] bpftune[82605]: segfault at 0 ip 00007f146066fde2 sp 00007fff34453140 error 4 in tcp_cong_tuner.so[7f146066f000+2000]
[949460.456415] Code: 45 a8 0f b6 40 40 83 f0 01 84 c0 0f 84 97 00 00 00 e8 c3 f7 ff ff 48 89 45 c8 48 8b 45 a8 48 8b 55 c8 48 89 50 48 48 8b 45 c8 <48> 8b 10 48 8b 45 a8 48 89 50 38 e8 8e f4 ff ff 48 8b 55 c8 48 8b

in dmesg.

How i can fix that?

Thanks.

[alan-maguire]

thanks for reportig - can you retry with the latest main branch? I ran into a segmentation fault on ubuntu and pushed a fix that resolved it. if it's still there, can you attach the stack associated with the core dump from gdb and i'll try and figure out what's going on.

[andrey-admin]

@alan-maguire same issue but with that in dmesg:

[953685.243427] bpftune[85946]: segfault at 0 ip 00007f29a426fde2 sp 00007ffc218d6980 error 4 in tcp_cong_tuner.so[7f29a426f000+2000]
[953685.243437] Code: 45 a8 0f b6 40 40 83 f0 01 84 c0 0f 84 97 00 00 00 e8 c3 f7 ff ff 48 89 45 c8 48 8b 45 a8 48 8b 55 c8 48 89 50 48 48 8b 45 c8 <48> 8b 10 48 8b 45 a8 48 89 50 38 e8 8e f4 ff ff 48 8b 55 c8 48 8b

[andrey-admin]

dump.zip

[alan-maguire]

thanks! i can't reproduce it so can you try running "gdb bpftune ", and once in gdb run "bt" to get a stack backtrace? you might need to run "sudo sysctl -w kernel.core_pattern=core.%f.%p" first to get core files of form core.bpftune..

[andrey-admin]

https://pastebin.com/76XQ30ac

[alan-maguire]

thanks; the crash is happening in bpftune_bpf_init(); would you be able to run "bpftune -ds" to see if we can see what is happening with bpf open/load/attach?

[andrey-admin]

https://pastebin.com/PbFnpyFi

[alan-maguire]

i suspect the issue is https://lore.kernel.org/bpf/20211008000309.43274-7-andrii@kernel.org/ where the bpf skeleton generation does not like the .rodata.cst16 section . It may be that a newer bpftool might help; i'm using bpftool 5.15 on ubuntu from the linux-tools package synced to the kernel version. however we may also be able to work around this; you could try making the following chages to tcp_cong_tuner.bpf.c and rebuilding:

diff --git a/src/tcp_cong_tuner.bpf.c b/src/tcp_cong_tuner.bpf.c
index 77957b3..ab6661a 100644
--- a/src/tcp_cong_tuner.bpf.c
+++ b/src/tcp_cong_tuner.bpf.c
@@ -40,7 +40,7 @@ static __always_inline bool
retransmit_threshold(struct remote_host *remote_host,
u32 segs_out, u32 total_retrans)
{

•   const char bbr[CONG_MAXNAME] = "bbr";
  

•   static const char bbr[4] = "bbr";
    __u64 now;
  
    if (!remote_host)
  

@@ -188,7 +188,7 @@ int BPF_PROG(cong_retransmit, struct sock *sk, struct sk_buff *skb)
struct tcp_sock *tp = (struct tcp_sock *)sk;
struct in6_addr *key = &sin6->sin6_addr;
__u32 segs_out = 0, total_retrans = 0;

•   const char bbr[CONG_MAXNAME] = "bbr";
  

•   static const char bbr[4] = "bbr";
    int id = TCP_CONG_BBR;
    struct net *net;
  

that was enough to get rid of the .rodata.cst16 section (it's replaced with .rodata.str1.1 that bpftool can handle).

[alan-maguire]

patch got mangled but replaces

const char bbr[CONG_MAXNAME] = "bbr";

...with

static const char bbr[4] = "bbr";

...in the two places it is declared in tcp_cong_tuner.bpf.c

[pavlinux]

static volatile const char const bbr[4] = {'b', 'b', 'r', '\0'}; B-)

[andrey-admin]

can you put patch in attach, please?

[alan-maguire]

0001-avoid-.rodata.cst16-sections-in-tcp_cong_tuner.bpf.c.patch.gz

[andrey-admin]

still sigfault.

#0 0x00007fa2a93a2de2 in init (tuner=0x558225754260) at tcp_cong_tuner.c:58
58 bpftuner_bpf_init(tcp_cong, tuner, NULL);
(gdb) bt
#0 0x00007fa2a93a2de2 in init (tuner=0x558225754260) at tcp_cong_tuner.c:58
#1 0x00007fa2a99efa88 in bpftuner_init (path=0x7fff10a1bfb0 "/usr/lib64/bpftune//tcp_cong_tuner.so") at libbpftune.c:655
#2 0x0000558225735e32 in init (library_dir=0x5582257374aa "/usr/lib64/bpftune/") at bpftune.c:199
#3 0x0000558225736541 in main (argc=2, argv=0x7fff10a1c478) at bpftune.c:391

[andrey-admin]

last strings from -ds:

bpftune: libbpf: prog 'cong_retransmit': found data map 5 (tcp_cong.bss, sec 8, off 0) for insn 157
bpftune: libbpf: sec '.reltp_btf/tcp_retransmit_skb': relo #4: insn #179 against 'init_net'
bpftune: libbpf: prog 'cong_retransmit': found extern #0 'init_net' (sym 34) for insn #179
bpftune: libbpf: sec '.reltp_btf/tcp_retransmit_skb': relo #5: insn #182 against 'bpftune_init_net'
bpftune: libbpf: prog 'cong_retransmit': found data map 5 (tcp_cong.bss, sec 8, off 0) for insn 182
bpftune: libbpf: sec '.reltp_btf/tcp_retransmit_skb': relo #6: insn #190 against 'ring_buffer_map'
bpftune: libbpf: prog 'cong_retransmit': found map 0 (ring_buffer_map, sec 9, off 0) for insn #190
bpftune: libbpf: sec '.reliter/tcp': collecting relocation for section(5) 'iter/tcp'
bpftune: libbpf: sec '.reliter/tcp': relo #0: insn #37 against 'remote_host_map'
bpftune: libbpf: prog 'bpftune_cong_iter': found map 3 (remote_host_map, sec 9, off 80) for insn #37
bpftune: libbpf: sec '.reliter/tcp': relo #1: insn #52 against 'remote_host_map'
bpftune: libbpf: prog 'bpftune_cong_iter': found map 3 (remote_host_map, sec 9, off 80) for insn #52
bpftune: libbpf: sec '.reliter/tcp': relo #2: insn #57 against 'remote_host_map'
bpftune: libbpf: prog 'bpftune_cong_iter': found map 3 (remote_host_map, sec 9, off 80) for insn #57
bpftune: libbpf: sec '.reliter/tcp': relo #3: insn #135 against 'debug'
bpftune: libbpf: prog 'bpftune_cong_iter': found data map 5 (tcp_cong.bss, sec 8, off 0) for insn 135
bpftune: libbpf: sec '.reliter/tcp': relo #4: insn #139 against '.rodata'
bpftune: libbpf: prog 'bpftune_cong_iter': found data map 4 (tcp_cong.rodata, sec 11, off 0) for insn 139
bpftune: libbpf: failed to find skeleton map '.rodata.str1.1'
Segmentation fault (core dumped)

[alan-maguire]

can you check bpftool, clang versions ("bpftool --version", "clang --version"? ubuntu with bpftool v5.15 and clang v14 work fine for me, even with the .rodata.str1.1 sections.