I have been struggling with reading out the contents of a resources directory in my lein project. I understand now (after doing it wrong for awhile) to use clojure.java.io/resource to pull out a resource, because just using the file system doesn't work when it is packaged as a jar:
> (require '[clojure.java.io :as io])
> (def zipzip (.openStream (io/resource "zip.zip")))
This returns a BufferedInputStream. What I want to do is take this zip file and unpack it to a local directory. I can't make a ZipFile out of it, but I can make a ZipInputStream. Unfortunately, while I can get ZipEntries out of this, I need a ZipFile to actually read the contents of the ZipEntry. I can do this:
> (-> zipzip ZipInputStream. .getNextEntry .getName)
This returns the name, but there is nothing in the api docs to get the actual contents of that ZipEntry with the ZipInputStream!
How do I write out the contents from this ZipInputStream to a local directory? (that also works when the code is packaged into a jar!)
You can simply read from the ZipInputStream after you got the next entry. Use the size information from the entry to read the content.
user=> (import 'java.util.zip.ZipInputStream)
java.util.zip.ZipInputStream
user=> (def zs (ZipInputStream. (io/input-stream "foo.zip")))
#'user/zs
user=> (def ze (.getNextEntry zs))
#'user/ze
user=> (.getName ze)
"foo.txt"
user=> (.getSize ze)
21
user=> (let [bytes (byte-array 21)] (.read zs bytes 0 21) (String. bytes "UTF-8"))
"Das ist ein Test!\r\n\r\n"
Related
I use a macro called use-db and change-to-path in a lot of my projects.
(defmacro use-db (db project-name &body query)
`(progn
(change-to-path ,project-name)
(clsql:connect ,db :database-type :sqlite3)
(unwind-protect (progn ,#query)
(clsql:disconnect :database ,db))))
(defmacro change-to-path (project-name)
`(uiop:chdir (merge-pathnames (write-to-string ,project-name) "/Users/vince/quicklisp/local-projects/")))
As you can see, change to path changes the repl dir. Then use-db reads the sqlite db from the root of the project folder. This works fine.
However, when I try to read a db as part of a route (easy-routes + hunchentoot):
(defroute test-file ("/results" :method :post)
()
(get-one-col-as-list-db #'first "firstname"))
(defun get-one-col-as-list-db (fn tablename)
(ki:use-db "new.db" 'custom-sender
(mapcar fn
(clsql:query
(concatenate 'string "Select * from " tablename)))))
I get the following error:
Error in SB-POSIX:CHDIR: No such file or directory (2)
without using the change-to-path macro, clsql function would note find the database.
Is there a better way to use sqlite (from the file system) and manage the pathing for specific db files?
To change the current working directory, temporarily, use uiop:with-current-directory:
(uiop:with-current-directory ("/tmp/")
(uiop:getcwd))
(edit) chdir has a side effect: it effectively changes the current working directory (uiop:getcwd), while with-current-directory doesn't, or it does but only temporarily for the code under it. For your sanity you want to avoid chdir's side effets (or I do, I want to work at a project root). It could have an effect on bare-bones Hunchentoot, if it looks at a www/ directory under the project root, but in your case I'm not sure.
Your use-db mimics a well-known pattern: get a resource, and be sure to release it in case of errors. These macros are often named "with-something", just like "with-current-directory". Does CLSQL have such a macro? It does: with-database.
Why is your change-to-path a macro? It can be a function that concatenates strings and calls chdir.
Inside use-db, "project-name" is an unknown variable. Look at the compiler warnings:
; caught WARNING:
; undefined variable: COMMON-LISP-USER::PROJECT-NAME
How do you use the two anyways?
Error in SB-POSIX:CHDIR: No such file or directory (2) […] without using the change-to-path macro
what is the directory it tries to go to? You'd have a tip on how things are not working.
I try to never rely on the process working directory, because from Common Lisp I am much happier using *default-pathname-defaults* (no side effects!). This is a special variable with a long name, and what I want is to use merge-pathnames in most of the cases, so I have this macro:
(defmacro with-path (pathname &body body)
`(let ((*default-pathname-defaults* (merge-pathnames ,pathname)))
,#body))
It is just a local binding of a special variable so unwinding is trivial, and it comes with all the benefits of CL pathnames (and some of its drawbacks but that's fine).
On the Posix side of things, programs want a native namestring so I also use this a lot:
(defun fullname (pathname)
(osicat:native-namestring (merge-pathnames pathname)))
I do also have a concept of projects in my environment so it looks as follows. Note that I try to carry the special variable until the last moment, until it needs to be given to other processes, like:
(within-project projects:my-project
(with-path "build/"
(run "make")))
Here run grabs the current *default-pathname-defaults* for its :directory argument (using sb-ext:run-program but this is a detail).
Since it is a pathname, some components can be wild too, if that makes sense in your use case. For example, you can avoid giving a ".db" suffix if you prefer:
(defmacro using-db ((name db) &body body)
`(with-path #P"*.db"
(with-database (,name (list (fullname ,db)) :database-type :sqlite3)
,#body)))
For example, here I enter the "/tmp/" directory and open database "test.db":
(with-path "/tmp/"
(using-db (database "test")
database)))
#<CLSQL-SQLITE3:SQLITE3-DATABASE /tmp/test.db CLOSED {100DD6CE63}>
Edit: with respect to the following code:
(uiop:chdir
(merge-pathnames (write-to-string project-name)
"/Users/vince/quicklisp/local-projects/")))
Note that you can use ASDF to access files relative to systems:
(asdf:system-relative-pathname :cl-ppcre "ppcre.lisp")
#P"..../quicklisp/software/cl-ppcre-20220220-git/ppcre.lisp"
Sounds deceptively easy. This doesn't work:
~/.sbclrc
(load #P"~/in-package.lisp")
~/in-package.lisp
(in-package 'stored-package)
The package change only applies to the file in-package.lisp itself.
Try different approach: store just the name.
(defmacro recall-package (&optional (filename #p"~/lisp-package.lisp"))
"IN-PACKAGE the contents of the FILENAME"
(let ((p (car (uiop:read-file-lines filename))))
`(in-package ,p)))
This works, but only from ~/.sbclrc. Files which it LOADs expand the macro within their own context, and so it doesn't work.
SBCL reads it's .sbclrc like this:
(restart-case
(flet ((process-init-file (kind specified-pathname default-function)
(awhen (or specified-pathname (funcall default-function))
(with-open-file (stream (if specified-pathname
(parse-native-namestring it)
(pathname it))
:if-does-not-exist nil)
(cond (stream
(sb-fasl::call-with-load-bindings
(lambda (stream kind) (load-as-source stream :context kind))
stream kind stream))
(specified-pathname
(cerror "Ignore missing init file"
"The specified ~A file ~A was not found."
kind specified-pathname)))))))
(unless no-sysinit
(process-init-file "sysinit" sysinit *sysinit-pathname-function*))
(unless no-userinit
(process-init-file "userinit" userinit *userinit-pathname-function*))
Using these fancy sb-fasl::call-with-load-bindings and sb-int:load-as-source yields similar results to the above.
You can't do this with load, because
load binds *readtable* and *package* to the values they held before loading the file.
Function load
This means that any changes made to the values of these variables within a file being loaded are local to the file. That's almost always a good thing. In particular it means that there is no way at all (or no portable way: if you had access to the guts of the dynamic binding mechanism of the implementation this might not be true) that any changes made to the current package (ie the dynamic value of *package*) can ever propagate up through calls to load.
If all you want to do is set the package based on some name in a file, then this is relatively easy, with something like the below:
(defpackage :empty-package
(:use))
(defun set-package-from-file (f)
(let ((pn
(with-standard-io-syntax
;; EP just in case the file does somehow manage to smash
;; *package*)
(let* ((ep (find-package :empty-package))
(*package* ep)
(*read-eval* nil))
(unwind-protect
(with-open-file (in f)
(string (read in)))
;; Clean up EP to avoid leakage
(do-symbols (s ep)
(unintern s ep)))))))
(let ((p (find-package pn)))
(unless p (error "no package ~A" pn))
(setf *package* p))))
This is probably both overly-protective and thus will contain some horrible unexpected bug which I should have thought about (I know it's not safe against interning symbols in other packages). However the idea is that the file contains a single string-designator which should be the package name.
If you had time on your hands you could fairly easily write a version of load which would not rebind *package* &c, and which would work for source files. I think you can't portably write one which would work for FASL files.
Here's one reason why the behaviour the language specifies is the right behaviour: it makes compilation a lot easier. Consider a file which contains:
(in-package ...)
(defun foo (...) ...)
(load ...)
(defun bar (...)
(foo ...)
...)
If *package* could propagate up through load then compiling this file would be, at best, interesting.
While trying to work through cl-sdl2-tutorial, I'm having trouble loading a bitmap due to a wrong current working directory.
I'd like to get a proper solution to the problem using relative path names.
A minimal working example:
Having modified the code of example two from above mentioned tutorial.
(defpackage #:sdl2-tutorial-2
(:use :common-lisp)
(:export :main))
(in-package :sdl2-tutorial-2)
(defparameter *screen-width* 640)
(defparameter *screen-height* 480)
(defmacro with-window-surface ((window surface) &body body)
`(sdl2:with-init (:video)
(sdl2:with-window (,window
:title "SDL2 Tutorial"
:w *screen-width*
:h *screen-height*
:flags '(:shown))
(let ((,surface (sdl2:get-window-surface ,window)))
,#body))))
(defun main(&key (delay 2000))
(format t " cwd: ~a, ~% dpd: ~a, ~& e-p: ~a, ~% pf: ~a, ~& load: ~a"
(sb-posix:getcwd)
*default-pathname-defaults*
(uiop:file-exists-p "hello_world.bmp")
(probe-file "hello_world.bmp")
(sdl2:load-bmp "hello_world.bmp"))
(with-window-surface (window screen-surface)
(let ((image (sdl2:load-bmp "hello_world.bmp")))
(break "1 here with ~a~%" image)
(setf image (sdl2:load-bmp "hello_world.bmp"))
(break "2 here with ~a~%" image)
(break "3 error: ~a~%" (sdl2-ffi.functions:sdl-get-error))
(sdl2:blit-surface image
nil
screen-surface
nil)
(sdl2:update-window window)
(sdl2:with-event-loop (:method :poll)
(:quit () t)
(:idle ()
(sdl2:delay delay))))))
Before compiling above code and running (main), I changed working directory in the REPL, via:
(sb-posix:chdir (truename "/test/cl-sdl2-tutorial/2/"))
(setf *default-pathname-defaults* (truename "/test/cl-sdl2-tutorial/2/"))
The above code prints, as expected, when running (main) in the REPL:
SDL2-TUTORIAL-2> (sdl2-tutorial-2:main)
0: (SDL2-TUTORIAL-2:MAIN)
cwd: /test/cl-sdl2-tutorial/2,
dpd: /test/cl-sdl2-tutorial/2/,
e-p: /test/cl-sdl2-tutorial/2/hello_world.bmp,
pf: /test/cl-sdl2-tutorial/2/hello_world.bmp,
load: #<SDL-SURFACE {#X7F5CBC018DD0}>
Problem:
The bitmap can not be found and therefore not loaded.
Calls to (sdl2:load-bmp "hello_world.bmp") always return a a zero pointer (#<SDL-SURFACE {#X00000000}>) and breakpoint 3 states:
3 error: Couldn't open /home/jue/hello_world.bmp
but evaling (sdl2:load-bmp "hello_world.bmp") during a break from breakpoints 1 or 2 or 3, is successful and continuing (main) displays the picture.
Questions:
Why is sdl2:load-bmp using the "wrong" working directory and why is it using the "correct" working directory during breakpoints?
How to make sdl2:load-bmp use the wanted working directory (instead of "/home/jue/") when using relative paths?
Remarks
(I'm using current released versions of sbcl, Emacs, sly on a Linux machine, if that matters. I'm only intermediate experienced with Common Lisp and its development environment, but advanced at elisp)
I suspect but don't know that the problem is that the sdl2 library is doing fanciness with threads, with the result that the working directory isn't what you think.
The solution to this in my experience is never to let the implementation second-guess you like that. In any case where there's some interface which says "do something to a file" give it an absolute pathname so it has no chance to do any thinking of its own. Do something like.
(defparameter *where-my-bitmaps-live* (merge-pathnames
(pathname "lib/bitmaps/")
(user-homedir-pathname)))
...
(defun load-a-bitmap (name)
(load-bmp (merge-pathnames (pathname name) *where-my-bitmaps-live*)))
And now it really has no excuse.
(You want to check that the above pathname-mergery is actually right: it seems to be for me but I forget the details of pathname rules every time I look away for more than a few minutes).
How do I change the path that .fas files are placed in? By lisp code and not configuration file.
I am doing like this:
(asdf:disable-output-translations)
(push *default-pathname-defaults* asdf:*central-registry*)
(setq asdf:*central-registry*
(append asdf:*central-registry*
(directory (merge-pathnames
(make-pathname :directory '(:relative :wild))
*default-pathname-defaults*))))
With ECL, it generates the .fas and .o files in the source code directory being compiled, but I don't want it to be in the project directory, which is on a path that I go through, to better organize myself here.
Something like:
(asdf:set-path-compiled-files "/home/noloop/lisp-armv7a-libs/builds")
asdf:*central-registry* is for the location of SOURCE, there isn't a single location for compiled results because asdf2 explicitly handles you using multiple implementations, which have incompatible fasl formats (this is a point of improvement in asdf2 vs the previous state of affairs).
According to [1] (asdf:initialize-output-translations list) can be used to set the translation programmatically, and (asdf:ensure-output-translations) can be used to see the current value
As mentioned in the DSL, you must include one and only one of :inherit-configuration or :ignore-inherited-configuration.
It's not clear if you also need to set one of :enable-user-cache or :disable-cache but choosing one seems sensible.
As stated in [2] the order matters.
On my machine, with sbcl:
cl-user> (asdf::initialize-output-translations '(:output-translations (t #p"/tmp/asdf/") :disable-cache :ignore-inherited-configuration))
Gives:
((#P"/home/peter/Programming/unix-built/sbcl/lib/sbcl/**/*.*" T)
(#P"/tmp/asdf/" T) (T #P"/tmp/asdf/") (T T)
(#P"/home/peter/.cache/common-lisp/sbcl-2.0.0.104-b5a4454ca-linux-x64/**/*.*"
T)
(T
#P"/home/peter/.cache/common-lisp/sbcl-2.0.0.104-b5a4454ca-linux-x64/**/*.*"))
And with that I see FASLs in /tmp/asdf when I try and load a system.
1: http://soc.if.usp.br/manual/cl-asdf/asdf/Controlling-where-ASDF-saves-compiled-files.html#Controlling-where-ASDF-saves-compiled-files
2: https://gitlab.common-lisp.net/asdf/asdf/issues/22
I want to provide additional information to the user during a restart-case for missing input files.
Is there a portable way to print out the files in the current directory?
Please take a look at File System Concepts, specifically at directory.
E.g.,
use (directory "~/*") to get the list of files in your home,
use (directory "~/*/") to get the list of directories in your home,
and (directory "~/**/*") to get the list of all files in all subdirectoris in your home
Note that directory may take implementation-specific arguments like :full.
If you are using asdf (you should!), you might want to take a look at uiop which comes with it.
PS. If your implementation does not support ~ in pathname to refer to the home directory, you can use user-homedir-pathname instead.
The reference to uiop was the hint I needed. This is what I use now in my test case and which works perfectly for me:
(defun file-is-missing ()
(let* ((cwd (uiop:getcwd))
(files (uiop:directory-files cwd)))
(format t "~&Current directory: ~a~%" cwd)
(dolist (file files)
(format t "~&~3T~a" file))
(format t "~&Provide new file: ")
(list (read-line))))
(defun read-file (file)
(restart-case
(with-open-file (s file :direction :input :if-does-not-exist :error)
(do ((l (read-line s) (read-line s nil 'eof)))
((eq l 'eof) "Reached end of file.")
(format t "~&*** ~A~%" l)))
(New-File (filename)
:interactive file-is-missing
(read-file filename))))