I use a macro called use-db and change-to-path in a lot of my projects.
(defmacro use-db (db project-name &body query)
`(progn
(change-to-path ,project-name)
(clsql:connect ,db :database-type :sqlite3)
(unwind-protect (progn ,#query)
(clsql:disconnect :database ,db))))
(defmacro change-to-path (project-name)
`(uiop:chdir (merge-pathnames (write-to-string ,project-name) "/Users/vince/quicklisp/local-projects/")))
As you can see, change to path changes the repl dir. Then use-db reads the sqlite db from the root of the project folder. This works fine.
However, when I try to read a db as part of a route (easy-routes + hunchentoot):
(defroute test-file ("/results" :method :post)
()
(get-one-col-as-list-db #'first "firstname"))
(defun get-one-col-as-list-db (fn tablename)
(ki:use-db "new.db" 'custom-sender
(mapcar fn
(clsql:query
(concatenate 'string "Select * from " tablename)))))
I get the following error:
Error in SB-POSIX:CHDIR: No such file or directory (2)
without using the change-to-path macro, clsql function would note find the database.
Is there a better way to use sqlite (from the file system) and manage the pathing for specific db files?
To change the current working directory, temporarily, use uiop:with-current-directory:
(uiop:with-current-directory ("/tmp/")
(uiop:getcwd))
(edit) chdir has a side effect: it effectively changes the current working directory (uiop:getcwd), while with-current-directory doesn't, or it does but only temporarily for the code under it. For your sanity you want to avoid chdir's side effets (or I do, I want to work at a project root). It could have an effect on bare-bones Hunchentoot, if it looks at a www/ directory under the project root, but in your case I'm not sure.
Your use-db mimics a well-known pattern: get a resource, and be sure to release it in case of errors. These macros are often named "with-something", just like "with-current-directory". Does CLSQL have such a macro? It does: with-database.
Why is your change-to-path a macro? It can be a function that concatenates strings and calls chdir.
Inside use-db, "project-name" is an unknown variable. Look at the compiler warnings:
; caught WARNING:
; undefined variable: COMMON-LISP-USER::PROJECT-NAME
How do you use the two anyways?
Error in SB-POSIX:CHDIR: No such file or directory (2) […] without using the change-to-path macro
what is the directory it tries to go to? You'd have a tip on how things are not working.
I try to never rely on the process working directory, because from Common Lisp I am much happier using *default-pathname-defaults* (no side effects!). This is a special variable with a long name, and what I want is to use merge-pathnames in most of the cases, so I have this macro:
(defmacro with-path (pathname &body body)
`(let ((*default-pathname-defaults* (merge-pathnames ,pathname)))
,#body))
It is just a local binding of a special variable so unwinding is trivial, and it comes with all the benefits of CL pathnames (and some of its drawbacks but that's fine).
On the Posix side of things, programs want a native namestring so I also use this a lot:
(defun fullname (pathname)
(osicat:native-namestring (merge-pathnames pathname)))
I do also have a concept of projects in my environment so it looks as follows. Note that I try to carry the special variable until the last moment, until it needs to be given to other processes, like:
(within-project projects:my-project
(with-path "build/"
(run "make")))
Here run grabs the current *default-pathname-defaults* for its :directory argument (using sb-ext:run-program but this is a detail).
Since it is a pathname, some components can be wild too, if that makes sense in your use case. For example, you can avoid giving a ".db" suffix if you prefer:
(defmacro using-db ((name db) &body body)
`(with-path #P"*.db"
(with-database (,name (list (fullname ,db)) :database-type :sqlite3)
,#body)))
For example, here I enter the "/tmp/" directory and open database "test.db":
(with-path "/tmp/"
(using-db (database "test")
database)))
#<CLSQL-SQLITE3:SQLITE3-DATABASE /tmp/test.db CLOSED {100DD6CE63}>
Edit: with respect to the following code:
(uiop:chdir
(merge-pathnames (write-to-string project-name)
"/Users/vince/quicklisp/local-projects/")))
Note that you can use ASDF to access files relative to systems:
(asdf:system-relative-pathname :cl-ppcre "ppcre.lisp")
#P"..../quicklisp/software/cl-ppcre-20220220-git/ppcre.lisp"
Related
How do I change the path that .fas files are placed in? By lisp code and not configuration file.
I am doing like this:
(asdf:disable-output-translations)
(push *default-pathname-defaults* asdf:*central-registry*)
(setq asdf:*central-registry*
(append asdf:*central-registry*
(directory (merge-pathnames
(make-pathname :directory '(:relative :wild))
*default-pathname-defaults*))))
With ECL, it generates the .fas and .o files in the source code directory being compiled, but I don't want it to be in the project directory, which is on a path that I go through, to better organize myself here.
Something like:
(asdf:set-path-compiled-files "/home/noloop/lisp-armv7a-libs/builds")
asdf:*central-registry* is for the location of SOURCE, there isn't a single location for compiled results because asdf2 explicitly handles you using multiple implementations, which have incompatible fasl formats (this is a point of improvement in asdf2 vs the previous state of affairs).
According to [1] (asdf:initialize-output-translations list) can be used to set the translation programmatically, and (asdf:ensure-output-translations) can be used to see the current value
As mentioned in the DSL, you must include one and only one of :inherit-configuration or :ignore-inherited-configuration.
It's not clear if you also need to set one of :enable-user-cache or :disable-cache but choosing one seems sensible.
As stated in [2] the order matters.
On my machine, with sbcl:
cl-user> (asdf::initialize-output-translations '(:output-translations (t #p"/tmp/asdf/") :disable-cache :ignore-inherited-configuration))
Gives:
((#P"/home/peter/Programming/unix-built/sbcl/lib/sbcl/**/*.*" T)
(#P"/tmp/asdf/" T) (T #P"/tmp/asdf/") (T T)
(#P"/home/peter/.cache/common-lisp/sbcl-2.0.0.104-b5a4454ca-linux-x64/**/*.*"
T)
(T
#P"/home/peter/.cache/common-lisp/sbcl-2.0.0.104-b5a4454ca-linux-x64/**/*.*"))
And with that I see FASLs in /tmp/asdf when I try and load a system.
1: http://soc.if.usp.br/manual/cl-asdf/asdf/Controlling-where-ASDF-saves-compiled-files.html#Controlling-where-ASDF-saves-compiled-files
2: https://gitlab.common-lisp.net/asdf/asdf/issues/22
I have a directory name (as string) with a tilde: ~/projects.
I want to get its fullpath: /home/user/projects. How do I do that ?
The goal is to pass it to uiop:run-program, that doesn't seem to do the right thing©.
With this answer: How to translate (make-pathname :directory '(:absolute :home "directoryiwant") into absolute path
(merge-pathnames
(make-pathname
:directory '(:relative "~/projects"))
(user-homedir-pathname))
#P"/home/me/~/projects/"
=> WRONG
Thank you.
edit I'll share more context.
I wanted to run a program through uiop:launch-program. I had a user-defined list of directories such as ~/projects. Using it as is created the ./~/projects directory instead of /home/user/projects.
truename doesn't work if the directory doesn't exist.
On SBCL, (namestring "~/doesntexist") returns also its tilde.
merge-pathnames didn't work, still the tilde problem.
Feeding ensure-directories-exist with this result created a directory named ~.
Given the answers, I had no choice but to adapt the logic to expand the directory name of a directory we actually want to exist.
;; Create a directory
;; Ensure its name (string) ends with a slash.
(setf mydir
(str:concat (string-right-trim (list #\/) mydir)
"/"))
(ensure-directories-exist base)
Then I could use its truename.
General remarks about ~
Your Lisp implementation may or may not support tilde syntax.
If it does (e.g. CCL, ABCL, CLISP, ECL, LispWorks), then truename would consistently expand to a filename:
(truename "~/projects")
=> /home/user/projects
If your implementation doesn't, or if you want to code portably, you have to merge relatively to (user-homedir-pathname):
(truename (merge-pathnames #p"projects" (user-homedir-pathname)))
=> /home/user/projects
Note that the tilde, if it is supported, seems to only be supported for strings used as pathnames, and not in directory components; (:relative "~") does not work as you would expect, and refers to a directory literaly named "~".
Instead, at least for SBCL, the appropriate directory is (:absolute :home), or, if you want to refer to another user, you can wrap the component in a list:
(make-pathname :directory '(:absolute (:home "root")))
=> #P"~root/"
Notice how it only works if the :home form is just after :absolute, it doesn't work otherwise (see Home Directory Specifiers).
Expanding to non-existent pathnames
truename would require that the thing exists?
Yes, if you want to build the absolute path to a file that does not exist (yet), then you need to call truename on the part that exists, and merge with that.
In your case, that would be (truename "~/"), which is the same as (user-homedir-pathname).
As pointed out by Rainer Joswig, calling namestring on implementations other than SBCL returns an expanded pathname, translating ~ as /home/user. In SBCL you have to call sb-ext:native-namestring to obtain the same effect.
In other words, in order to expand to a filename that does not necessarily exist, you could write the following portability layer:
(defun expand-file-name (pathname)
(check-type pathname pathname)
(block nil
#+(or lispworks clozure cmu clisp ccl armedbear ecl)
(return (namestring pathname))
#+sbcl
(return (native-namestring pathname))
#+(not (or sbcl lispworks clozure cmu clisp ccl armedbear ecl))
(let ((expanded (namestring pathname)))
(prog1 expanded
(assert (not find #\~ expanded) ()
"Tilde not supported")))))
See also https://github.com/xach/tilde/blob/master/tilde.lisp for inspiration if your Lisp doesn't support the syntax.
There is a native-namestring function in uiop, which should be available in all implementations:
(uiop:native-namestring "~/projects")
=> /home/user/projects
Anselm Farber's solution, involving uiop:native-namestring breaks on some pathnames that don't have native-namestrings, like the following:
(uiop:native-namestring "~/Music/[Video] performance.mp4")
==>
The pathname #P"~/Music/[Video] performance.mp4"
does not have a native namestring because
of the :NAME component #<SB-IMPL::PATTERN (:CHARACTER-SET
. "Video")
" performance">.
[Condition of type SB-KERNEL:NO-NATIVE-NAMESTRING-ERROR]
Here is a direct solution that only uses pathname- functions:
(defun expand-user-homedir (f)
(let ((d (pathname-directory f)))
(if (and (eql (car d) :absolute)
(eql (cadr d) :home))
(make-pathname :directory (append (pathname-directory (user-homedir-pathname))
(cddr d))
:name (pathname-name f)
:type (pathname-type f))
f)))
My lisp file contains about 50 function and macro definitions. At the head of the file is:
(defpackage :utils
(:use :common-lisp))
(in-package :utils)
; ... ~50 defuns/defmacros follow along with some other code
I want to export all these 50 definitions. But no other symbols.
To write them all out in :export clauses inside defpackage seems very tedious.
I tried getting the necessary symbols by using loop's facility for listing symbols in a package. But both "symbol" and "present-symbol" don't output the needed definition symbols. I could somehow subtract "external-symbols" and inherited symbols. But this seems like the kind of thing everyone would need and there should be an established easy solution. I looked around and haven't found a straightforward answer.
How do I export all the definition symbols without having to write them out individually and manually?
scan symbols
You can scan all symbols and export all that have a function binding (this includes macro names):
(defun export-fbound-symbols (package)
"export symbols with function bindings"
(do-symbols (s package)
(when (fboundp s)
(export s package))))
This is usually not a very good idea: good software engineering practice is to limit the set of public interfaces.
be more selective
You can redefine defun and defmacro so that they automatically export their definienda.
See exporting.lisp in CLISP. E.g.,
(defmacro define-function (name lambda-list &body body)
`(progn
(export ',name)
(defun ,name ,lambda-list ,#body)))
Note that this ignores the possibility of (defun (setf foo) (...) ...).
No, this is not something everyone needs. Usually, you'd export them as you write (and actually need) them.
I'd probably use an ad hoc keyboard macro for this. You could also write a little Lisp function to read your file form by form and output the relevant symbols for copy-paste.
All I wanted was to load an initialisation file for swank which wouldn't affect my lisp when it's started without swank...
I first tried #+swank (defun...) in my file that's loaded from ccl-init (trying this on ccl 1.10 + windows), and soon realised it's sourced before swank is loaded (obviously).
My aim is to define a simple function in :cl-user everytime I start swank. I just ended up with a swank add-hook to load my init.lisp file, and since I want to define the function in cl-user, I tried this in the init.lisp:
(let ((current-package *package*))
(in-package :cl-user)
(defun cd (dir)
(swank:set-default-directory
(parse-namestring dir)))
(in-package current-package))
Now, I don't recall if a defun in let was allowed, but lisp doesn't complain for it, but rather tells me that cur-pck symbol doesn't exist, and it seems when we switch the package, cur-pck binding gets out of scope. I thought cur-pck is a lexical binding, and that it should be reachable from within the lexical region, being independent from a package, am I wrong?
Why do I switch packages? I'm thinking that loading this file from swank at some initialisation point will define things in some swank package, that's why I wanted to try switching to cl-user first, define the function symbol, and switch back to let swank do it's thing.
At this point I guess I need someone to tell me I'm approaching the problem from the wrong angle, and that I should better choose an easier solution.
Additionally, out of curiosity in case above is the complete wrong approach, is there a way to define a symbol in another package within a function or a closure?
Switching packages in a form has no direct effect on the form
Let's look at this:
(in-package "FOO")
(let ((x 10))
(in-package "BAR")
(setf x 20))
Which x does it set to 20? FOO::X or BAR::X?
Well, it is FOO::X. Switching packages during execution has no effect on the symbols already read. The whole let form is read at once and the *package* value is used for that. Having an IN-PACKAGE in the form itself has no effect on the form itself.
Symbol with a package prefix
If you want to use a symbol in a certain package, just write the package prefix:
cl-user:foo ; if FOO is exported and the package exists
or
cl-user::foo ; if foo is not exported and the package exists
for example:
(defun cl-user::cd (...) ...)
Computing with symbols
You can also compute new symbols in packages you don't know yet:
(let ((sym-name "A-NEW-SYMBOL")
(my-package-name "SOME-EXISTING-PACKAGE"))
(intern sym-name my-package-name))
If the package does not exist, you can create it.
You can also set the function of a computed symbol:
(setf (symbol-function (compute-a-function-symbol))
#'(lambda ()
'foo))
If you want to define a function in a different package that the current one you can use a qualified symbol for a name
(defun cl-user::cd (dir)
(swank:set-default-directory
(parse-namestring dir)))
The binding is not being "lost". To test so yourself add (princ cur-pck) before the in-package form.
If you try evaluating (in-package *package*) you will see why your code fails to switch packages. The in-package macro does not evaluate its argument. The code that would give us the code we would want to evaluate is:
(let ((cur-pck *package*))
(in-package :cl-user)
(defun cd (dir)
(swank:set-default-directory
(1+ 2)))
(princ cur-pck)
`(in-package ,cur-pck))
However, as Rainer Joswig noted in his answer the in-package has no effect on forms already read, so it wouldn't work as desired even as a macro.
A style nitpick, don't use abbreviations, write current-package.
IN-PACKAGE is a macro, not a function. The problem in your code is that saying (in-package cur-pck) tries not to switch into the package denoted by the cur-pck variable but to the package named CUR-PCK (which obviously does not exist).
You can set the package temporarily with
(let ((*package* (find-package :cl-user)))
(defun cd (dir)
...))
But then again, the easiest way to achieve what you are doing would be
(defun cl-user::cd (dir)
...)
which totally eliminates the need to set the current package.
Following this question: Strange symbols in filespec when calling load I tried my luck with pathnames, but, as you see, failed. Below is an example of the error, which I cannot explain:
This code does not work:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test #P"digit-recognition:digit-7.png")
(process-image-raw test)))
Neither does this:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test "digit-recognition:digit-7.png")
(process-image-raw test)))
But this code does:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test (translate-logical-pathname "digit-recognition:digit-7.png"))
(process-image-raw test)))
And so does this:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test (translate-logical-pathname #P"digit-recognition:digit-7.png"))
(process-image-raw test)))
Here's the "translator":
(setf (logical-pathname-translations "DIGIT-RECOGNITION")
`(("**;*.*" "/home/wvxvw/Projects/digit-recognition/**/*.*")))
And here's the error I'm getting:
Pathname components from SOURCE and FROM args to TRANSLATE-PATHNAME
did not match:
:NEWEST NIL
[Condition of type SIMPLE-ERROR]
Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT] Abort thread (#<THREAD "repl-thread" RUNNING {1003800113}>)
Backtrace:
0: (SB-IMPL::DIDNT-MATCH-ERROR :NEWEST NIL)
1: (SB-IMPL::TRANSLATE-COMPONENT :NEWEST NIL :NEWEST T)
2: (TRANSLATE-PATHNAME #P"DIGIT-RECOGNITION:DIGIT-7.PNG.NEWEST" #P"DIGIT-RECOGNITION:**;*.*" #P"/home/wvxvw/Projects/digit-recognition/**/*.*")
3: (TRANSLATE-LOGICAL-PATHNAME #P"DIGIT-RECOGNITION:DIGIT-7.PNG.NEWEST")
4: (SB-IMPL::QUERY-FILE-SYSTEM #P"DIGIT-RECOGNITION:DIGIT-7.PNG" :TRUENAME NIL)
5: (PROBE-FILE #P"DIGIT-RECOGNITION:DIGIT-7.PNG")
6: (CREATE-IMAGE-FROM-FILE #<unavailable argument> NIL)
7: (TEST-PROCESS-IMGAE-RAW)
I'm trying to read the Hyperspec section on translate-pathname, but I can make absolutely no sense of what it says, neither from the examples it shows. Let alone it, I can't even understand how there can possibly be an error if you transform a string by whatever rules you put in place, so far it's only one way transformation...
I'm trying to read SBCL sources for this function, but they are really lengthy, and trying to figure out the problem this way is taking huge amounts of time.
tl;dr How is it even possible that translate-logical-pathname called from user's code will produce something different to what is produced from that function if called from system code? This is not only non-portable, this is just outright broken.
EDIT:
Adding one more asterisk to the pattern on the left side, but not on the right solved this. But the purpose or logic of why is this necessary is beyond me.
I.e.
(setf (logical-pathname-translations "DIGIT-RECOGNITION")
`(("**;*.*.*" "/home/wvxvw/Projects/digit-recognition/**/*.*")))
This allows pathnames like digit-recognition:foo.bar.newest to succeed, just like digit-recognition:foo.bar but why is that asterisk a requirement flies beyond me. Also, why is the system function feels entitled to change the pathname to something else of what it was given?.. But just not to get you confused, with-image-from-file will only work with the path already expanded by translate-logical-pathname, it won't work otherwise.
EDIT2:
OK, it seems like this is the problem with cl-gd, instead of trying to expand the file name, it takes it literally. This code taken from create-image-from-file probably best answers my question:
(when (pathnamep file-name)
(setq file-name
#+:cmu (ext:unix-namestring file-name)
#-:cmu (namestring file-name)))
(with-foreign-object (err :int)
(with-cstring (c-file-name file-name)
(let ((image (ecase %type
((:jpg :jpeg)
(gd-image-create-from-jpeg-file c-file-name err))
I.e. instead of doing (namestring file-name) it has to do (namestring (trnaslate-logical-pathname file-name)). Duh...
Another way is to use TRUENAME, which returns the real file name. Normally this would not make a difference.
Image a file system with file versions (like the file systems of VMS, ...). If you have a logical pathname foo:bar;baz.png.newest, then it might translate to, say, /myfiles/images/baz.png~newest (again, just assume that it has version numbers). This still is not a real physical file. If such a Lisp system tries to open the file, it has to look into the file system to actually determine the newest file. That might be /myfiles/images/baz.png~42.
So, if you want to pass real physical filenames to external tools (like a C library), it might not be sufficient to expand the logical pathname, but it might be necessary to compute the truename - the real physical file.
The ability to deal with file versions comes from a time when file versions where quite common (see Versioning file system) with operating systems like ITS, VMS or the various Lisp Machine operating systems.
The main practical problem for this is that there is no common test suite for pathname operations for the various CL implementations and thus implementations differ in a lot of subtle details (especially when you need to deal with different file systems from different operating systems). Plus real file systems have complications - for example file names in Mac OS X use a special unicode encoding when dealing with Umlauts.