From How to specify additional files as prerequisites that would cause ASDF to recompile the program, I learned that the ASDF static-file directive can be used to list non-Lisp files that a Common Lisp program depends on. For example:
(defsystem "myprog"
:components ((:static-file "1.txt")
(:static-file "2.txt")
(:file "myprog" :depends-on ("1.txt" "2.txt"))))
This requires the programmer to explicitly list out the static files. If there are many static files, and all of them reside in a directory and its sub-directories, it may become impractical to list out all the static files by hand. Is there a way for ASDF to consider all the files in a directory and its sub-directories as static files?
In fact, you can just use the same :static-file "directive" to indicate a dependency on the files in some directory. Or rather the presence of the directory itself (possibly, with some files in it). Theoretically, you could add a separate :static-dir dependency class with some special processing. But the question is what will this special processing constitute? If you don't want to manually specify an exact list of files (which can already be done with (:module "dir" :components ((:static-file "file1") ...))) what other additional processing do you want for this dependency class?
Related
I use ADSF to organise my projects.
(asdf:defsystem :sender
:version "0.1.4"
:serial t
:depends-on (:cl-json :dexador :clsql :clsql-sqlite3)
:components ((:file "packages")
(:file "validation")
(:file "sender")))
But when I open a new repl (slime, emacs), I have to go to the packages.lisp file and evaluate the form before I can change the repl to that package with (in-package :sender).
Is there a way to make the slime repl remember my packages?
My current thinking is that I need to "run" adsf to load all my files. If so, how?
UPDATE:
Actually, I need to compile all the files manually before I can actively use them in the repl. I believe I am using asdf incorrectly.
From a .asd system definition, the steps would be:
compile the .asd file, so that you Lisp image knows about the system
C-c C-k or programmatically: (asdf:load-asd "system").
LOAD it somehow: (ql:quickload "system") to install dependencies
or (asdf:load-system "system"), but it will fail if your project has unknown dependencies, Quicklisp would download them.
avoid using cl:load for asdf systems, load-asd / load-system will do more complex stuff.
use it on the REPL. The package that was defined in a packages.lisp file should be available, because this file was declared in the .asd.
You can add stuff to your ~/.sbclrc (or similar for other implementations). Here I tell ASDF where a project lives, so I can quickload it without manually compiling the .asd before:
(pushnew "/home/vince/projets/ciel/" asdf:*central-registry* :test #'equal)
This is an "old" style not encouraged anymore by the ASDF documentation.
Or simply create a symlink for your project to ~/quicklisp/local-projects/ or ~/common-lisp/.
I wouldn't "load-system" my project in the init file, because that's a side effect that would show sometimes when I don't want it (like building an image: if that involves reading my init file, I'd have an unwanted system in it).
Saving a core image with a ton of dependencies is cool. Use it with sbcl --core …. The advantage is that it starts up instantly in the REPL, when loading all projects and dependencies would take a few seconds.
Pointers
ASDF
ASDF cookbook
ASDF manual
ASDF best practices
ASDF Build System explained
Overview
A system is an object which describes a library (or an application) which consists of: a name, a version, a list of files and subsystems, and a list of other systems it depends on. There is more.
If you want to have Lisp know a certain system, then we need to execute its defsystem form either manually or find it automatically. That ONLY makes the system definition known.
If you want to load the system's files, then you need to LOAD the SYSTEM. If you want to compile the system's files files, the you need to COMPILE the system.
So, if you want to use a package (which is a namespace), then you need to run the DEFPACKAGE form. If that form is describe in a file which is component of a system, then you need to load that system.
ASDF : a popular build system for Common Lisp
system -> library, application. It is a bunch of files and dependencies.
system definition -> not the system itself, but a description of it. Use DEFSYSTEM to describe a system
system operations -> like load-system, compile-system and others.
system registry -> Typically we want ASDF to find the systems by name. See the manual or a tutorial for ASDF.
lisp file -> some file we can compile or load
fasl file -> some compiled file we can load
package -> a namespace for symbols, that's actually a built-in feature for Common Lisp
Quicklisp -> a library manager with a selection of curated Common Lisp libraries/applications. The libraries can be loaded over the Internet. Be aware that there is little to no security when loading foreign code from the Internet.
Use a system, its components and dependencies
If you want to use a system you need to load it first. Manually or automatically.
That loading can be done automatically:
For example in an init-file for your Lisp you can load all systems you want/need. Alternative write a function which loads all systems you want.
saving an image. Some Lisps support saving an image. Thus one can load all interesting systems and then save an image. If one starts that image later, all those systems are already in memory. (side problem: getting newer versions loaded)
Assume 1000 files with extension .xhtml are in directory input, and that a certain subset of those files (with output paths in $(FILES), say) need to be transformed via xslt to files with the same name in directory output. A simple make rule would be:
$(FILES): output/%.xhtml : input/%.xhtml
saxon s:$< o:$# foo.xslt
This works, of course, doing the transform one file at a time. The problem is that I want to use saxon's batch processing to do all the files at one time, since, given the number of files, that would be much faster, considering the overhead of loading java and saxon for each file. Saxon allows the -s (source) option to be a directory and processes all files in that directory, placing the results with the same name in the directory specified in the -o: option.
I'm aware of the well-known technique to get GNU make to do a single command to update multiple files by using pattern rules:
output/%.xhtml: input/%.xhtml
saxon s:input -o:output foo.xslt
But in my case this suffers from two problems. First, it will run the transform on all files in the input directory, not just the ones that have changed; and second, it will not limit the transform to the subset of files specified in $(FILES). The GNU make feature of running a recipe given in a pattern rule only once for all matched targets does not work in the case of so-called "static pattern rules" (see [here]), as the rule given at the top of the post is known.
In order to use the saxon batching feature, I need to create a temporary directory, copy to it only those files to be processed, then run the transform with that temporary directory as the input directory. I tried creating a temporary directory, and remember its name using a target-specific variable for future use, using
$(FILES): TMPDIR:=$(shell mktemp -d)
but this creates a new temporary directory for every single target that is out-of-date. In any case, I'm not sure how to structure the rule that would then copy the necessary files into that directory. I don't want to create the temporary directory at the time the makefile is parsed, since I have a non-recursive make system that will parse all make files, even those not related to the current top-level target, and don't want to create the temporary directory for situations in which it is not necessary/will not be used.
I'm well aware that many questions have been asked on SO in the past about creating multiple files from a single input; one solution is (non-static) pattern rules; other solutions involve phony targets. However, in this case I'm stuck as to how to put all this together.
I can identify the files that changed and copy them using the static pattern rule
$(FILES): output/%.xhtml : input/%.xhtml
TMPDIR=`mktemp -d`
cp $< $(TMPDIR)
but actually I would prefer to copy the files with a single cp command, whereas this copies them one by one. Perhaps there is some application here of cp -u?
I also considered using an ad-hoc extension for those files needing updating but could not see how to get this to work either. I'm about to give up and just run the saxon transform on all files when any of them have changed, but is there any better way?
Personally, I wouldn't try to do this from the command line. That's partly because I'm not a shell scripting wizard. I'm not an Ant wizard either, but because the requirement is to process files that haven't changed, this seems to fall very much into Ant territory. On the other hand, Ant will recompile the stylesheet for each transformation, which is an overhead you might want to avoid; if that's the case then your best bet is probably to write a little Java application. It's probably only 100 lines or less.
Final possibility is to do the processing within Saxon: that is, a single transformation that reads multiple input files using the collection() function and generates multiple result files using xsl:result-document. Saxon (commercial editions) offers an extension function last-modified that allows you to filter the files to be processed. With 1000 files you might also want the extension function saxon:discard-document() to prevent the heap filling.
Personally, I like your original one-compiler-per-file formulation. Does not this work well with make's -j n flag?
You can of course batch up files by copying, and then running saxon at the end. Recursive make (ugh!) can sort out the ordering. Something like:
.PHONY: all
all:
rm -rf tmpdir
${MAKE} tmpdir/sentinel
saxon -s:tmpdir -o:output foo.xslt
tmpdir/sentinel: $(FILES) ; touch $#
$(FILES): output/%.xhtml: input/%.xhtml
ln $< $(patsubst input/%,tmpdir/%,$<)
This does work, though I am very queasy about lying to make (the static pattern rule purports to create the target in output/, but in fact does its dirty deed in tmpdir/).
Note in the recipe for tmpdir/sentinel, that $? is correctly set to the list of output files that are out of date. This might be useful if you can pass a bunch of files to saxon rather than a folder.
I think one issue here is that 'saxon' supports either one file or all files in a directory, so isn't suitable for batch processing without copying to temporary directories.
Otherwise, this is quite simple to do by using a timestamp marker file as a proxy target. For example:
output/.timestamp : $(FILES)
mkdir -p $(#D)
$(COMMAND) -outputdir=output $?
touch $#
The three commands are:
Ensure that the output directory exists.
Run the batch command on files newer than the timestamp file.
Update the timestamp file (creating it if necessary).
Remembering that each line of a command is executed in its own subshell, and that if any command line fails, then subsequent lines are not invoked.
This approach is useful with Java builds.
I have a project with some folders which happen to contain source files with the same names.
My source tree looks like this:
project.pro
foo/
conflict.h
conflict.cpp
bar/
conflict.h
conflict.cpp
some.h
other.h
files.h
main.cpp
Per default, qmake generates a Makefile which will produce a build tree like this:
conflict.o
main.o
target
Where conflict.o is the object file resulting for both foo/conflict.cpp and foo/conflict.h.
I can't to change their names because they are generated using an external tool and forcing different file names would imply to change their contents, so this is not an option.
I also don't want to use qmake SUBDIRS template because this would imply that (1) every subdir is built separately as a library and thus very much complicate the overall build process (in my eyes at least) and (2) in the top level directory I can't have any source files. Or am I wrong?
Can't I just tell qmake to write the object files into separate directories within the build directory? So my build tree will look like this:
foo/
conflict.o
bar/
conflict.o
main.o
target
Or are there any other solutions neither requiring to rename the source files nor introducing something complicated like static libraries? I just can't believe that Qt didn't solve this (in my eyes simple) problem for years. (I already hat this problem 4 years ago but could rename the files in that project, while here I can't.)
If it's important: I use Qt 4.8 on both Ubuntu with G++ and Windows with mingw32.
Are you tied to qmake? If not, an alternative could be to use cmake. I just verified your usecase with a simple CMakeLists.txt like
cmake_minimum_required (VERSION 2.6)
project (conflict)
add_executable(conflict foo/conflict.cpp bar/conflict.cpp main.cpp)
which even included a source file in the top level directory (main.cpp). This properly builds the executable - the object files are created in sub directories like
./CMakeFiles/conflict.dir/main.cpp.o
./CMakeFiles/conflict.dir/bar/conflict.cpp.o
./CMakeFiles/conflict.dir/foo/conflict.cpp.o
cmake also includes support for Qt4, to automatically pull in the required include paths and libraries. It might require some effort to migrate from qmake to cmake, but given the requirements you have I would give it a try.
I am trying to load my project using a self-made .asd file, but somehow asdf does not recognize that the .asd file should also be the root-directory of the project and therefore resuling in an error like failed to find the TRUENAME of /home/$USER/rel-path-to-file where rel-path-to-file is the path to the corresponding file, relative to the loaded .asd file.
This is what I did:
(push "path-to-project-directory-of-asd-file" asdf:*central-registry*)
then
(asdf:load-system 'project-name)
It DOES find my .asd file but somehow not the .lisp files it shall load.
The asdf has been configured and installed by quicklisp. Therefore I also "installed" the project into quicklisp using a symlink, this worked as well. It DOES find the .asd file using ql:quickload :name but still doesn't use the directory containing the .asd file as the project-root.
How can I fix this?
Lisp pathnames are historically such that they require you to add a slash after directory name (I was told this is so due to the operating systems which were created before I was even born), so in order to tell Lisp that you target a directory you must end the name with the slash.
See asdf:system-relative-pathname.
I am new to cl, and I just learned to install packages using asdf-install, but I don't know how it works, I wonder how the package can be installed manully, then I could understand the use of the files in the root directory of the source code, thanks.
Short answer: Just use quicklisp.
Long answer: if you want to understand, how the package, or - more precisely - ASDF system, is laid out, that's a good idea. Actually, there's nothing hard about that.
Every ASDF system should have a system definition file with .asd extension. This file names other file of the system with their paths relative to the .asd file, their types (by default: lisp source code) and dependencies. Your Lisp should know where to find the system definition file. In ASDF there are 2 ways to inform Lisp about it: adding the directory, in which you store the file or symlink to it, to asdf:*central-registry* list or setting up special configuration files (called source-registry - more on that in ASDF manual).
Now if you want to install the system by hand, just download its sources, extract them into some directory (like in /home/user/lib/lisp/ - you may get /home/user/lib/lisp/cl-ppcre-2.3.1/, inside which there's cl-ppcre.asd). To let your Lisp find out about it just (push "/home/user/lib/lisp/cl-ppcre-2.3.1/" asdf:*central-registry*) (and don't forget the trailing slash - it's required), and then you can load the system with (asdf:oos 'asdf:load-op :cl-ppcre).
You might also setup a special dir, where you'll symlink your existing systems, like /home/user/.lisp/ and add it to *central-registry* at Lisp startup type (e.g. in .sbclrc). Now if you want to temporarily override some of the system linked in this dir, say, with a newer version, you don't need to unlink anything - just push the path to alternative system to *central-registry*.
Quicklisp does all that for you and more...