Distributing a CFFI wrapper for a custom C lib - common-lisp

Question

I've built a library that wraps custom C code and thinking about the best way to build the shared library as part of the ASDF load. The makefile is conditionalised for various OSs, so it could be as simple as uiop:run-program ..., but I thought I'd ask here if there were a more standard idiom for this.
Because the C code is specific to this application, it won't be available through a package manager and must be built specifically for each users machine. I'm fine with documenting a manual build, but if I can smooth things for the user I will. I notice that Python seems to have some kind of automated way of building libs for their CFFI and wonder if there's something for CL.

Answer 1

For an answer to my own question: there seems to be neither a de-facto way of doing this (based on a search of github asd files), nor a definitive method in the ASDF best practices, though there are some ideas to be gleaned from that document.
I'll put my implementation out as a suggested idiom for this use case, along with some possible alternatives. Hopefully some of the ASDF experts here will correct any misunderstandings.
;; Define a makefile as a type of source file for the system
(defclass makefile (source-file) ((type :initform "m")))
;; tell ASDF how to compile it
(defmethod perform ((o load-op) (c makefile)) t)
(defmethod perform ((o compile-op) (c makefile))
(let* ((lib-dir (system-relative-pathname "cephes" "scipy-cephes"))
(lib (make-pathname :directory `(:relative ,(namestring lib-dir))
:name "libmd"
:type #+unix "so" #+(or windows win32) "dll"))
(built (probe-file (namestring lib))))
(if built
(format *error-output* "Library ~S exists, skipping build" lib)
(format *error-output* "Building ~S~%" lib))
(unless built
(run-program (format nil "cd ~S && make" (namestring lib-dir)) :output t))))
(defsystem "cephes"
:description "Wrapper for the Cephes Mathematical Library"
:version (:read-file-form "version.sexp")
:license "MS-PL"
:depends-on ("cffi")
:serial t
:components ((:module "libmd"
:components ((:makefile "makefile")))
(:file "package")
(:file "init")
(:file "cephes")))
This works fine, on both MS Windows and UNIX. Adding a method to perform seems to be the most common method on github.
An alternative might be to use a build-op, as described in building a system. The description
Some systems offer operations that are neither loading in the current
image, nor testing. Whichever operation a system is meant to be used
with, you may use it with:
(asdf:make :foobar)
This will invoke build-op, which in turn will depend on the
build-operation for the system, if defined, or load-op if not.
Therefore, for usual Lisp systems that want you to load them, the
above will be equivalent to (asdf:load-system :foobar), but for other
Lisp systems, e.g. one that creates a shell command-line executable,
(asdf:make ...) will do the Right Thing™, whatever that Right Thing™
is.
suggest to me that this is rather close to the idea of building a C library, and it would map nicely to the mental model of using a makefile and the asdf:make command. I didn't find too many examples in the wild of this being used though and technically we are loading the C lib into the existing image.
Another point that could be reconsidered is the detection of an existing shared library to avoid the rebuild. make will avoid recompiling if the shared library exists, but will still call the linker again. This causes errors because it can't write to the shared library when it's in use, at least on MS Windows. The ASDF example used Lisp code to detect the existence of the library and avoiding recompilation, but an alternative might be to use output-files.
The ASDF docs are a bit muddled on the purpose of output-files and there are no examples that make their intentions clear, but in the manual section on creating new operations we have:
output-files If your perform method has any output, you must define a
method for this function. for ASDF to determine where the outputs of
performing operation lie.
which suggests that defining the shared library (libmd.so or libmd.dll) is the recommended way to avoid a recompilation if the output-files already exists.
Finally, the C library could be considered a secondary system, cephes/libmd in this case, and added to the :depends-on clause in the main system. The section on other secondary systems demonstrates building an executable this way, with build-op. Except for the fact that this is building an executable and hard-codes ".exe" it seems to map well onto the use case:
To build an executable, define a system as follows (in this case, it's
a secondary system, but it could also be a primary system). You will
be able to create an executable file foobar-command by evaluating
(asdf:make :foobar/executable):
(defsystem "foobar/executable"
:build-operation program-op
:build-pathname "foobar-command" ;; shell name
:entry-point "foobar::start-foobar" ;; thunk
:depends-on ("foobar")
:components ((:file "main")))
The build-pathname gives the name of the executable; a .exe type will
be automatically added on Windows.
I didn't use this method because the secondary system would look almost exactly like the primary one does now, but would be slightly less understandable.