bpftune should tune it's own fd limit

[withinboredom]

I noticed that bpftune uses quite a bit of file descriptors and keeps bumping up against the soft-limits. Ideally, it should be able to inspect process limits and adjust them accordingly. This could potentially spiral out of control, so I suspect there is probably some other back-pressure to keep it under control, but I have no idea what that would be (maybe no more than its fair share of open files from the total max?)

root@delta ~ # lsof -p 950 | wc -l
1050
root@delta ~ # cat /proc/950/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             256549               256549               processes
Max open files            1024                 524288               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       256549               256549               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
root@delta ~ # ulimit -n
1048576

[alan-maguire]

The source of the fd usage is largely around network namespaces, but it's possible you're experiencing an fd leak too. the general approach is intended to be get an fd for netns access, use it then close it. so while this is a bit more costly in terms of time, it means we don't have too many fds open. do you have hundreds of network namespaces on this system? if not i strongly suspect it's an fd leak somewhere in the code.

[withinboredom]

There are thousands of network namespaces on this machine :) as it is a (bare metal) production machine as part of a kubernetes cluster. I'll turn up the number of fds to something reasonable for my situation and see if that helps.

[withinboredom]

Instead of "something reasonable" I gave it a limit of 65k and let it run for about 10 minutes. I would have expected this graph to level out by then, but it just kept on going up.

It does seem there is a leak. Needless to say, it still seems to function at its lower limit, so it might not be an actual leak but just some quirky behavior or an accounting issue. If there were an actual leak, I would expect it to crash when the limit is hit and be unable to open any new descriptor.

[alan-maguire]

Thanks for collecting this data! I'll add a test which creates a large number of network namespaces and see if I can reproduce + fix the leak.

[alan-maguire]

I've created a test which reproduces fd leaks when large numbers of netns are present, and confirmed the leaks I see go away with the fix. if you could try building updated main branch with pr #106 that would be great. thanks!

[withinboredom]

Confirmed that this fixes it @alan-maguire. I'm now seeing ~100 fd's open consistently. I don't have a pretty graph because it is essentially just a flat line.

[alan-maguire]

excellent, thanks for the quick turnaround on testing! i see ~130 open fds, mostly for bpf programs, maps and links so that's in the ballpark of what i'd expect. thanks again!

[withinboredom]

Thanks to you as well for the quick fixes! This is now deployed to the whole fleet, and I can't wait to watch this project grow!