Assembly eBPF
=============

This is a hand-written eBPF tool tree.  The eBPF programs are assembly
source files.  The build is plain make.

The first target is a set of small tracing programs that can grow into a
full system inspection tool: process activity, files, network paths, and
kernel events.  The eBPF side stays assembly.


BUILDING
--------

Preferred assembler:

        bpf-unknown-none-as

Build with a BPF GNU assembler:

        make

If the BPF GNU assembler is not installed, clang can assemble the same eBPF
sources:

        make LLVM=1

Inspect generated objects:

        make dump

Clean generated files:

        make clean


RUNNING
-------

The asm-ebpf loader loads and attaches the assembled programs:

        su -c ./asm-ebpf

It keeps running until interrupted.  It needs the usual privileges for loading
eBPF programs and opening tracing events.

On a terminal the loader shows a live full-screen view, refreshed in place.
Four overview tabs switch with 1, 2, 3 and 4: the tree of tracked processes
drawn with the usual branch characters, the busiest pids, the network
connections from procfs with the owning pid and command and the socket state,
and a log of recent exec calls newest first with the full command line.  The
exec log is kept because a short-lived command execs and exits before it can be
found in the pid list, so it would otherwise never be seen; the log holds the
last commands so their argv is there to read after the fact.  In the pid
view p freezes the list so the arrows can move the highlight without the numbers
shifting underneath.  In the pid view the up
and down arrows move the highlight and enter opens a detail view for that pid;
its header shows the parent chain (who spawned it), the upper half keeps a
per-kind tally, and the lower half lists its operations with repeats collapsed
into a counter, so a flood of one syscall cannot scroll the rare, useful events
away.  The detail view narrows the in-kernel filter to the chosen pid, and f
cycles which kinds the kernel emits (all, no read/write, net+file only), so a
flooding syscall can be dropped before it ever leaves the kernel and the rare
events keep flowing; d toggles the write-buffer preview.  While the detail view
is open the overview tabs only count that pid; counters resume when it closes.
Lists longer than the screen scroll: the arrows and page keys move through the
tree, the connections and a pid's operations, and a footer shows the position.
In the detail view r reads memory from the process itself: type a user-space
address and a program on the syscall entry tracepoint, which runs in the
target's own context, copies bytes from that address with bpf_probe_read_user
and hands them back, shown as a hex and ascii dump that refreshes on every
syscall the process makes.  This is only possible because the probe executes as
the traced task, so its address space is the current one; the read is armed for
the inspected pid alone and cleared on leaving the view.  The probe sits on
every syscall entry, so it is a system-wide cost while the tool runs.

In the detail view m switches the lower half from the operation list to a
behaviour fingerprint: a table of which syscall followed which, prev -> next,
ordered by how often the pair occurred.  It is built from the pid's own event
stream, so a process that reads then writes in a loop and one that connects then
sends look different at a glance, and a familiar program that starts doing
something new stands out.  The same syscall arrives once per per-cpu program
copy and shares its nanosecond timestamp, so the count follows the timestamp and
each syscall is counted once.

Keys: q quit or leave detail, esc leave detail, p pause, f kinds, d data,
x inject, m fingerprint and r read-memory (detail), / filter and t tripwire (pids),
arrows/pgup/pgdn scroll.  The per-kind counts and latency histogram print on exit.
The terminal handling and rendering are hand-written x86-64 assembly in tui.S.
When stdout is not a terminal, or with --no-tui, the loader streams one line
per event instead:

Each line starts with seconds since the first event, then the event kind, the
pid and tid, the command, and an argument: a path with open flags, a file
descriptor, a signal, a mapping length, a connect or sendto destination as
addr:port (IPv4 or IPv6), or the tcp_connect endpoints as saddr -> daddr.

The openat, connect and accept programs are split across the sys_enter and
sys_exit tracepoints: the enter side stashes the arguments in a map keyed by
thread, and the exit side pairs them with the return value, so a line can end
with = <fd> or = -ENOENT.

The write program also copies the first bytes of the buffer, shown as a quoted
preview; d toggles it in the detail view for privacy.

The openat exit program keeps an fd-to-path index in a map keyed by thread and
descriptor, so the read, write and close programs no longer show a bare number:
the loader looks the path up and prints it.  One program fills the index and
others read it, which is the kind of cross-program state only eBPF maps allow.

The kernel probes capture the kernel call stack at the moment they fire with a
stack-trace map, and the loader resolves the frames through /proc/kallsyms, so a
tcp_connect line ends with the network path that reached it, such as
[tcp_connect<-__inet_stream_connect<-__sys_connect].  Kernel addresses are only
readable as root, which the loader already needs.

Besides the syscall tracepoints the tree attaches kernel probes on tcp_connect,
tcp_close, vfs_open, vfs_unlink and tcp_conn_request.  Probes on symbols the
running kernel does not export are skipped.

The tool does not only watch: two probes act on the kernel.  In the detail view
x arms fault injection for the inspected pid: probes on the openat and connect
handlers, which the kernel marks error-injectable, call bpf_override_return so
the chosen process fails those calls with -EACCES while the real handler never
runs.  It is scoped to that one pid and cleared on leaving the view.  In the pid
view t arms a tripwire: a second program on the openat entry tracepoint reads
the opened path with bpf_probe_read_user_str, compares it against a path typed
into the config map, and calls bpf_send_signal to SIGKILL any process that opens
it.  Both need the
kernel built with error injection and signal helpers, and both undo themselves
when the loader exits.  Rewriting a call's outcome and killing on a match are
things a probe can do only in the kernel; this is where the tool stops being a
pure observer.

The loader filters out its own events.  Trailing names attach only the matching
hooks; -p limits output to one pid and -n to one comm:

        su -c './asm-ebpf exec open'
        su -c './asm-ebpf -n bash -p 1234'

The pid and comm filters run inside the eBPF programs through a config map.
Run with -h for the full option list.

The eBPF programs also keep running totals in kernel maps: every program
atomically counts its events by kind, and the read, write, sendto and recvfrom
programs accumulate the requested byte counts.  Each program also counts its
events per pid.  On exit the loader prints the per-kind event counts, the byte
totals for the I/O kinds, and the busiest pids.

The execve program walks the argv array in the kernel: it reads the argument
vector pointer from the tracepoint, follows the first four user pointers, and
copies each string with bpf_probe_read_user_str into its own slot, so the event
carries the command line and not just the program path.  The loader joins the
slots, so an exec shows curl -Lo /dev/null <url> rather than a bare /usr/bin/curl.
Each argument is capped and a trailing ... marks a longer line.

The execve program stores a start timestamp keyed by pid, and the process exit
program looks it up to report the process lifetime as life=<ms> on the exit
line, then drops the entry.

A separate program pair times read syscalls: the enter program stores a
timestamp keyed by thread, the exit program computes the delta, buckets it by
power of two entirely in assembly, and accumulates a histogram map.  The loader
prints the latency histogram on exit.

A program on the scheduler's sched_switch tracepoint measures off-cpu time: the
time a thread spends blocked, waiting rather than running.  When a thread is
switched out in a sleeping state it stamps a start time keyed by thread; when it
is switched back in the program adds the elapsed time to a per-thread total.
The detail view sums these totals across a process's threads and shows the
milliseconds it has spent blocked, which is where I/O and lock waits hide that
the syscall counts alone do not reveal.  This runs on every context switch, so
it is the busiest probe in the tree.


TOOLCHAIN
---------

GNU as works only when binutils was built with the BPF target.  A normal host
assembler such as /usr/bin/as for x86_64 is not enough.

Common package names to look for:

        binutils-bpf
        binutils-bpf-unknown-none
        bpf-unknown-none-binutils

Some distributions do not ship this target.  In that case build binutils with:

        --target=bpf-unknown-none

and make sure bpf-unknown-none-as is in PATH.

Clang support needs a build with the BPF backend enabled.  The command used by
this tree is:

        clang -target bpf -x assembler-with-cpp