How to find all available systems? - common-lisp

There are good functions I use to study Common Lisp projects:
CL-USER> (list-all-packages)
CL-USER> (describe (asdf:find-system "asdf"))
How to list all systems know for asdf, quicklisp or sbcl?
I've tried to dig it from documentation but did not find it yet.

All systems registered in ASDF:
(asdf:registered-systems)
I found that one by typing asdf:systems and letting auto-completion suggests a name. The symbol is exported, so it is fair game. Apparently it is undocumented.
Quicklisp has a notion of distributions, dists.
(ql-dist:all-dists)
Each dist has different versions (http://blog.quicklisp.org/2011/08/going-back-in-dist-time.html):
(ql-dist:available-versions (ql-dist:dist "quicklisp"))
Each dist provides systems:
(ql-dist:provided-systems (ql-dist:dist "quicklisp"))
Each system has a release, you can list all releases:
(ql-dist:provided-releases (ql-dist:dist "quicklisp"))
Conforming implementation have a list of *MODULES*, which is useful notably for systems that are available as built-ins by your implementation; for SBCL:
CL-USER> (require 'sb-mpfr)
("SB-MPFR" "SB-GMP")
CL-USER> *modules*
("SB-GMP" "SB-MPFR" ...)

Also, you can use to have a convenient search through all quickdist's systems, lookup their documentation and dependencies:
http://quickdocs.org

Related

Distributing a CFFI wrapper for a custom C lib

I've built a library that wraps custom C code and thinking about the best way to build the shared library as part of the ASDF load. The makefile is conditionalised for various OSs, so it could be as simple as uiop:run-program ..., but I thought I'd ask here if there were a more standard idiom for this.
Because the C code is specific to this application, it won't be available through a package manager and must be built specifically for each users machine. I'm fine with documenting a manual build, but if I can smooth things for the user I will. I notice that Python seems to have some kind of automated way of building libs for their CFFI and wonder if there's something for CL.
For an answer to my own question: there seems to be neither a de-facto way of doing this (based on a search of github asd files), nor a definitive method in the ASDF best practices, though there are some ideas to be gleaned from that document.
I'll put my implementation out as a suggested idiom for this use case, along with some possible alternatives. Hopefully some of the ASDF experts here will correct any misunderstandings.
;; Define a makefile as a type of source file for the system
(defclass makefile (source-file) ((type :initform "m")))
;; tell ASDF how to compile it
(defmethod perform ((o load-op) (c makefile)) t)
(defmethod perform ((o compile-op) (c makefile))
(let* ((lib-dir (system-relative-pathname "cephes" "scipy-cephes"))
(lib (make-pathname :directory `(:relative ,(namestring lib-dir))
:name "libmd"
:type #+unix "so" #+(or windows win32) "dll"))
(built (probe-file (namestring lib))))
(if built
(format *error-output* "Library ~S exists, skipping build" lib)
(format *error-output* "Building ~S~%" lib))
(unless built
(run-program (format nil "cd ~S && make" (namestring lib-dir)) :output t))))
(defsystem "cephes"
:description "Wrapper for the Cephes Mathematical Library"
:version (:read-file-form "version.sexp")
:license "MS-PL"
:depends-on ("cffi")
:serial t
:components ((:module "libmd"
:components ((:makefile "makefile")))
(:file "package")
(:file "init")
(:file "cephes")))
This works fine, on both MS Windows and UNIX. Adding a method to perform seems to be the most common method on github.
An alternative might be to use a build-op, as described in building a system. The description
Some systems offer operations that are neither loading in the current
image, nor testing. Whichever operation a system is meant to be used
with, you may use it with:
(asdf:make :foobar)
This will invoke build-op, which in turn will depend on the
build-operation for the system, if defined, or load-op if not.
Therefore, for usual Lisp systems that want you to load them, the
above will be equivalent to (asdf:load-system :foobar), but for other
Lisp systems, e.g. one that creates a shell command-line executable,
(asdf:make ...) will do the Right Thing™, whatever that Right Thing™
is.
suggest to me that this is rather close to the idea of building a C library, and it would map nicely to the mental model of using a makefile and the asdf:make command. I didn't find too many examples in the wild of this being used though and technically we are loading the C lib into the existing image.
Another point that could be reconsidered is the detection of an existing shared library to avoid the rebuild. make will avoid recompiling if the shared library exists, but will still call the linker again. This causes errors because it can't write to the shared library when it's in use, at least on MS Windows. The ASDF example used Lisp code to detect the existence of the library and avoiding recompilation, but an alternative might be to use output-files.
The ASDF docs are a bit muddled on the purpose of output-files and there are no examples that make their intentions clear, but in the manual section on creating new operations we have:
output-files If your perform method has any output, you must define a
method for this function. for ASDF to determine where the outputs of
performing operation lie.
which suggests that defining the shared library (libmd.so or libmd.dll) is the recommended way to avoid a recompilation if the output-files already exists.
Finally, the C library could be considered a secondary system, cephes/libmd in this case, and added to the :depends-on clause in the main system. The section on other secondary systems demonstrates building an executable this way, with build-op. Except for the fact that this is building an executable and hard-codes ".exe" it seems to map well onto the use case:
To build an executable, define a system as follows (in this case, it's
a secondary system, but it could also be a primary system). You will
be able to create an executable file foobar-command by evaluating
(asdf:make :foobar/executable):
(defsystem "foobar/executable"
:build-operation program-op
:build-pathname "foobar-command" ;; shell name
:entry-point "foobar::start-foobar" ;; thunk
:depends-on ("foobar")
:components ((:file "main")))
The build-pathname gives the name of the executable; a .exe type will
be automatically added on Windows.
I didn't use this method because the secondary system would look almost exactly like the primary one does now, but would be slightly less understandable.

What's the meaning of "#+" in the code of cl-mysql? [duplicate]

This question already has answers here:
operator #+ and #- in .sbclrc
(2 answers)
Closed 6 years ago.
Recently I tried to read code about cl-mysql, but got stuck with the #+.
Tried to google it, but not work, so turn to here
(defun make-lock (name)
#+sb-thread (sb-thread:make-mutex :name name)
#+ecl (mp:make-lock :name name)
#+armedbear (ext:make-thread-lock)
#+ (and clisp mt) (mt:make-mutex :name name)
#+allegro (mp:make-process-lock :name name))
And looks like it is for different backend lisp compiler. But still no idea why write something like this.
Anyone can help me make it clear, thx.
#+ is a reader-macro that checks if a keyword is in the special variable *FEATURES*. If it isn't there, the following form will be skipped over (by the reader; the compiler will never see it). There is also #- which does the opposite.
There are some things that aren't part of the Common Lisp standard, but are important enough that all (or most) implementations provide a non-standard extension for them. When you want to use them in code that needs to work on multiple implementations, you have to use read-time conditionals to provide the correct code for the current implementation. Mutexes (and threads in general) are one of those things.
Of course there may be features provided by third party libraries as well. The contents of *FEATURES* will look something like this:
(:SWANK :QUICKLISP :SB-BSD-SOCKETS-ADDRINFO :ASDF-PACKAGE-SYSTEM :ASDF3.1
:ASDF3 :ASDF2 :ASDF :OS-UNIX :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :64-BIT
:64-BIT-REGISTERS :ALIEN-CALLBACKS :ANSI-CL :ASH-RIGHT-VOPS
:C-STACK-IS-CONTROL-STACK :COMMON-LISP :COMPARE-AND-SWAP-VOPS
:COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :ELF :FLOAT-EQL-VOPS
:FP-AND-PC-STANDARD-SAVE :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS
:INTEGER-EQL-VOP :INTERLEAVED-RAW-SLOTS :LARGEFILE :LINKAGE-TABLE :LINUX
:LITTLE-ENDIAN :MEMORY-BARRIER-VOPS :MULTIPLY-HIGH-VOPS :OS-PROVIDES-DLADDR
:OS-PROVIDES-DLOPEN :OS-PROVIDES-GETPROTOBY-R :OS-PROVIDES-POLL
:OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T :PACKAGE-LOCAL-NICKNAMES
:PRECISE-ARG-COUNT-ERROR :RAW-INSTANCE-INIT-VOPS :SB-DOC :SB-EVAL :SB-FUTEX
:SB-LDB :SB-PACKAGE-LOCKS :SB-SIMD-PACK :SB-SOURCE-LOCATIONS :SB-TEST
:SB-THREAD :SB-UNICODE :SBCL :STACK-ALLOCATABLE-CLOSURES
:STACK-ALLOCATABLE-FIXED-OBJECTS :STACK-ALLOCATABLE-LISTS
:STACK-ALLOCATABLE-VECTORS :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS
:UNIX :UNWIND-TO-FRAME-AND-CALL-VOP :X86-64)
So if you wanted to write code that depends on Quicklisp for example, you could use #+quicklisp. If you wanted code that is only run if Quicklisp is not available, you'd use #-quicklisp.
You can also use a boolean expression of features. For example,
#+(or sbcl ecl) (format t "Foo!")
would print Foo! on either SBCL or ECL.
#+(and sbcl quicklisp) (format t "Bar!")
would only print Bar! on SBCL that has Quicklisp available.
One could imagine that we can write:
(defun make-lock (name)
(cond ((member :sb-thread *features)
(sb-thread:make-mutex :name name))
((member :ecl *features*)
(mp:make-lock :name name))
...))
But that does usually not work, because we can't read symbols when their package is not existing and some packages are implementation/library/application specific. Packages are not created at read time in a lazy/automatic fashion.
In Common Lisp, reading a symbol of a package, which does not exist, leads to an error:
CL-USER 1 > (read-from-string "foo:bar")
Error: Reader cannot find package FOO.
1 (continue) Create the FOO package.
2 Use another package instead of FOO.
3 Try finding package FOO again.
4 (abort) Return to level 0.
5 Return to top loop level 0.
In your example sb-thread:make-mutex is a symbol which makes sense in SBCL, but not in Allegro CL. Additionally the package SB-THREAD does not exist in Allegro CL. Thus Allegro CL needs to be protected from reading it. In this case, the symbol sb-thread:make-mutex will only be read, if the the feature sb-thread is present on the cl:*features* list. Which is likely only for SBCL, or a Lisp which claims to have sb-threads available.
The feature expressions here prevents the Lisp from trying to read symbols with unknown packages - the packages are unknown, because the respective software is not loaded or not available.

Conditional Windows comments in Common Lisp

What's the canonical Windows/Not-windows conditional comment in Common Lisp?
#-win32 (non-windows-stuff) #+win32 (usually-some-horrific-hack)
seems to work correctly on SBCL and LispWorks (whether the underlying platform is 32 or 64 bit), but CCL only seems to understand
#-windows (non-windows-stuff) #+windows (usually-some-horrific-hack)
How do I dispatch on platform in a portable (across implementations) way? Is there a reference for standard comment flags somewhere?
There is no canonical element of *features* that works across all implementations. The trivial-features project normalizes features across platforms and implementations so, after loading trivial-features, you can use #+windows anywhere to mean Windows.
#+ and #- tests the presence of the symbol in the *features* list. Try checking it's value on the repl of the implementations/platforms you're targeting to see if you can find something in common.
You can also use #+(or windows win32) to test for either of them.
I use:
#+(OR WIN32 WINDOWS MSWINDOWS) ...
Also
#+(and (not windows) (or win32 mswindows))(pushnew :windows *features*)
and then
#+windows ...

Defining aliases to standard Common Lisp functions?

Lisp is said to enable redefinitions of its core functions.
I want to define an alias to the function cl:documentation function, such that
(doc 'write 'function) === (documentation 'write 'function)
How can this be done and made permanent in SBCL?
Creating an Alias
You are not trying to redefine (i.e., change the definition of) the system function documentation, you want to define your own function with a shorter name which would do the same thing as the system function.
This can be done using fdefinition:
(setf (fdefinition 'doc) #'documentation)
How to make your change "permanent" in common lisp
There is no standard way, different implementation may do it differently, but, generally speaking, there are two common ways.
Add code to an init file - for beginners and casual users
SBCL
CLISP
Clozure
ECL
The code in question will be evaluated anew every time lisp starts.
Pro:
Easy to modify (just edit file)
Takes little disk space
Normal lisp invocation captures the change
Con:
Evaluated every time you start lisp (so, slows start up time if the code is slow)
Save image - for heavy-weight professionals
SBCL
CLISP
Clozure
ECL - not supported
The modified lisp world is saved to disk.
Pro:
Start uptime is unaffected
Con:
Requires re-dumping the world on each change
Lisp image is usually a large file (>10MB)
Must specify the image at invocation time
Even though #sds has already answered pretty thoroughly I just wanted to add that the utility library serapeum has defalias
I use a simple macro for this:
(defmacro alias (to fn)
`(setf (fdefinition ',to) #',fn))
e.g.
(alias neg -) => #<Compiled-function ... >
(neg 10) => -10
Other answers include detail about how to make this permanent.

Evolving a lisp image

I love the idea of image-based languages, and lately I've been toying with Common Lisp via sbcl. I've read in a few places about how through being able to save and load back an image of the virtual machine, you can evolve an application or set of apps running on that image.
I get how to load code into an image and get it running, slime makes this sort of thing very nice, but my question is this: How can I tell what functions are defined in an image? Let's say I want to make an update to a function days or months after it has been running and I can't remember the name. Is there any way to get to the code or even just the names of the functions defined in an the image?
Now, I do write the code out into source and load it in via the repl, so I have a copy there, but it seems like this would be an obvious feature.
Common Lisp has the idea of packages. Packages are kind of a registry for symbols and are used as namespaces for symbols. You can ask Common Lisp for a list of all packages.
CL-USER 1 > (list-all-packages)
(#<The SQL-COMMON package, 0/4 internal, 28/32 external>
#<The LOOP package, 245/256 internal, 3/4 external>
#<The COMM package, 0/4 internal, 940/1024 external>
#<The REG package, 41/64 internal, 0/4 external>
...)
Each packages stores the interned symbols in some data structure. You can ask Common Lisp which symbols are interned in a package.
CL-USER 2 > (loop for symbol being
each external-symbol in (find-package "COMMON-LISP")
collect symbol)
(MAKE-ARRAY INVOKE-DEBUGGER STRING-TRIM ...)
To make that easier, Common Lisp provides functions APROPOS and APROPOS-LIST.
CL-USER 3 > (apropos "MAKE-LOCK")
MP::INTERNAL-MAKE-LOCK (defined)
MP:MAKE-LOCK (defined)
WWW-UTILS:MAKE-LOCK (defined)
MAKE-LOCK
RESOURCES::MAKE-LOCK (defined)
MINIPROC:MAKE-LOCK (defined)
Functions, Classes, etc. use symbols as their identifier. You can also ask a symbol, which function it denotes.
CL-USER 4 > (symbol-function 'www-utils:make-lock)
#<Function WWW-UTILS:MAKE-LOCK 41E006A69C>
Sometimes a Common Lisp also records the definition of functions. Then the function FUNCTION-LAMBDA-EXPRESSION can be used to retrieve 'it'.
CL-USER 5 > (defun foo (a) (* (sin a) a))
FOO
CL-USER 6 > (pprint (function-lambda-expression 'foo))
(LAMBDA (A)
(DECLARE (SYSTEM::SOURCE-LEVEL #<EQ Hash Table{0} 41403151C3>))
(DECLARE (LAMBDA-NAME FOO))
(* (SIN A) A))
But usually nowadays Common Lisp implementations don't use the recorded definitions, but record the locations of the source for each Lisp construct.
Most Common Lisp implementations can track the source locations in an implementation specific way.
The Common Lisp standard defines a function ED.
CL-USER 7 > (ed 'www-utils:make-lock)
This calls an editor (internal or external) and should open the source code for that function. To make that work, Common Lisp needs to keep track of the source location for each function. Next the editor needs to have access to that source. Sometimes the location recorded is an absolute path /Users/joswig/lisp/utils.lisp . If the editor wants to open that file, it should be accessible. But it is also possible to use logical pathnames like http:server;utils.lisp . This is then translated into a real physical pathname. This translation can later be configured. So it would be possible to move a Lisp to a different machine with different pathnames, configure the logical pathname HTTP and then the Lisp still finds all source code, even though it is on a different machine with a different file system structure. So, to make it work may need some configuration. But it is a very useful feature and it is widely used.
How the recording of source code and how the recording of source locations work is implementation dependent and is a feature of the respective Lisp in combination with its development environment. Better Lisp implementations have a lot of features in this area.
You can also use do-symbols or do-external-symbols if you prefer:
example:
>> (do-external-symbols (s (find-package :foo-package)) (print s))
FOO-PACKAGE:XXX
FOO-PACKAGE:YYY
FOO-PACKAGE:ZZZ
NIL
Where XXX, YYY & ZZZ are all external symbols in the package :foo-package.

Resources