Packaging: Toolchain for clean builds

[ra-id-j]

Summary

This Ideas thread aims to collect all the discussions concerning packaging of lem using the standard GNU Build System: Autoconf, Automake and Libtool. The discussion aims to incorporate the clean build logic encapsulated in recent flake.nix commits. Ideas from packaging of lem in Guix will also be considered. Contributions from packaging for FreeBSD and OpenBSD are also strongly encouraged.

Background

2025-09-16
Commit be73d0d: Add terminal.so
@jgarte: Hi @Sasanidas @vindarel, what do you think if instead we have a Makefile that would build this binary instead of including a binary directly that we don't know how it was compiled?

2025-09-18
PR #1964: Remove binary files
@mldulaney: Open source software should, by definition, not be distributing binary files. Given the xz backdoor, these increase distrust.

Continuing the discussion below.

[ra-id-j]

Guix build for terminal.c

Reference: comment on build invocation of terminal.c in Guix. Thanks @jgarte.

$COMPILER extensions/terminal/terminal.c -L <lib-dir> -lvterm -Wl,-Bdynamic -o <shared-lib> -shared -fPIC -lutil

Dockerfile build for terminal.c

Comparing the previous to the equivalent build invocation as documented in the Dockerfile:

gcc terminal.c -Wl,-Bstatic -lvterm -Wl,-Bdynamic -o terminal.so -shared -fPIC -lutil

I note that Guix does not compile a static library.

TODO: Decision on static vs dynamic shared library

[ra-id-j]

Contributed lisp build snippet for terminal.so

#1964 (comment)

The path is hardcoded to lib/<OS>/<ARCH>/terminal.so. This is probably best delegated to the configure autoconf script to figure out.

---

;; thanks to April & May on Discord
(let* ((source (asdf:system-relative-pathname :lem-terminal "terminal.c"))
       (lib (asdf:system-relative-pathname
             :lem-terminal (format nil
                                   "~(lib/~A/~A/terminal.so~)"
                                   (uiop:operating-system)
                                   (uiop:architecture)))))
  (unless (probe-file lib)
    (ensure-directories-exist lib)
    #+darwin
    (uiop:run-program (format nil "cc ~A -I/opt/homebrew/include -L/opt/homebrew/lib -lvterm -o ~A -shared -fPIC" 
                              source lib))
    #-darwin
    (uiop:run-program (format nil "cc ~A -lvterm -o ~A -shared -fPIC" source lib))))

=>

cc /home/vindarel/lisp-projects/lem/extensions/terminal/terminal.c -lvterm -o /home/vindarel/lisp-projects/lem/extensions/terminal/lib/linux/x64/terminal.so -shared -fPIC

terminal.c:297:34: error: ‘VTermScreenCellAttrs’ has no member named ‘conceal’
  297 |   return terminal->lastCell.attrs.conceal;
[…]

[ra-id-j]

Nix build framework for lem

Reference: #2019

Important comment by @tani:

Lem when used with Nix relies heavily on Nix's Common Lisp library collection. We have no dependency on qlot.

TODO: List the build steps from flake.nix

[ra-id-j]

Nix Build Script: build-lem.lisp

Reference:

lem/flake.nix

Lines 30 to 48 in 6d259d0

	mkBuildScript =
	{
	entryPoint ? "lem:main",
	}:
	pkgs.writeText "build-lem.lisp" ''
	(defpackage :nix-cl-user (:use :cl))
	(in-package :nix-cl-user)
	;; Load ASDF
	(load "${lem-base.asdfFasl}/asdf.${lem-base.faslExt}")
	(asdf:initialize-output-translations '(:output-translations :disable-cache :inherit-configuration))
	;; Load Systems
	(mapcar #'asdf:load-system (uiop:split-string (uiop:getenv "systems")))
	;; Dump Image
	(setf uiop:*image-entry-point* #'${entryPoint})
	(uiop:dump-image "lem" :executable t :compression t)
	'';

The heavy lifting is all done by asdf and uiop.

[ra-id-j]

lem-project/webview

The lem webview library is currently modeled similarly to the basic example of upstream webview. The main difference is that the CMake file builds a library instead of an executable.

FetchContent

The upstream project source is implicitly fetched through CMake FetchContent.

https://github.com/lem-project/webview/blob/76715d325c44cbd410b3f1359b09e7e8bd2226eb/c/CMakeLists.txt#L10-L14

FetchContent_Declare(
    webview
    GIT_REPOSITORY https://github.com/webview/webview
    GIT_TAG 0.12.0)
FetchContent_MakeAvailable(webview)

main.c

The main.c is only a placeholder for the upstream header file. Since there is no new C or C++ code contained within, we can probably compile the upstream webview code directly as a shared libwebview.so library.

Alternative webview shared library build

We can also build a shared library directly from the upstream webview source code with the following cmake invocation.

cmake -B build -S . \
  -D WEBVIEW_BUILD_DOCS=OFF \
  -D WEBVIEW_BUILD_EXAMPLES=OFF \
  -D WEBVIEW_BUILD_AMALGAMATION=OFF \
  -D WEBVIEW_BUILD_TESTS=OFF

If the resulting libwebview.so shared library is sufficient, we can probably simplify the lem webview source code by removing the c subdirectory entirely. This alternative approach does not produce a libexample.so file, which to me seems to be a side artifact of the example code for making a customised webview library (in other words, an irrelevant artifact).

[petebu]

For terminal.so, cffi already ships with some c toolchain support that can build this at deployment time. It might be simplest to just use that for c libraries included with Lem.

For external deps like libwebview it's best just to trigger the build system they ship with (e.g. cmake).

[ra-id-j]

Thanks for the comment.

Indeed for the upstream webview dependency, it's best to let cmake do its job. We just need a simple makefile for the lisp wrapper (webview.lisp and so on). I'm thinking of submitting a PR which brings in the upstream webview code as an explicit git submodule to replace the implicit FetchContent arrangement described above.

[petebu]

Including the upstream webview dep as a git submodule sounds good. I would pin it to a release tag too. Presumably you want to do that in lem-project/webview and not lem-project/lem?

Then we should be able to delete c code in webview/c like you say above.

[ra-id-j]

copied from #2105 (comment):

I have a proposed solution for terminal.so (and the other single-file C deps e.g. async-process and tree-sitter) here:
petebu@8346396
but it does depend on upstreaming a change to cffi to support using pkg-config to find includes and libs.

looks like a very elegant solution based on system "cffi-toolchain"!
(Does the CFFI user manual even document this?!)