Build terminal.so on CI, don't include a binary

[vindarel]

see #2060

[petebu]

I have a proposed solution for terminal.so (and the other single-file C deps e.g. async-process and tree-sitter) here:
petebu@8346396
but it does depend on upstreaming a change to cffi to support using pkg-config to find includes and libs.