It seems there is NO ANSI standard way to execute an external program and get its output as the following SBCL special code does:
(defmacro with-input-from-program ((stream program program-args environment)
&body body)
"Creates an new process of the specified by PROGRAM using
PROGRAM-ARGS as a list of the arguments to the program. Binds the
stream variable to an input stream from which the output of the
process can be read and executes body as an implicit progn."
#+sbcl
(let ((process (gensym)))
`(let ((,process (sb-ext::run-program ,program
,program-args
:output :stream
:environment ,environment
:wait nil)))
(when ,process
(unwind-protect
(let ((,stream (sb-ext:process-output ,process)))
,#body)
(sb-ext:process-wait ,process)
(sb-ext:process-close ,process))))))
The following CCL code reports "ERROR: value # is not of the expected type (AND CCL::BINARY-STREAM INPUT-STREAM)"
#+clozure
(let ((process (gensym)))
`(let ((,process (ccl:run-program "/bin/sh" (list "-c" (namestring ,program))
:input nil :output :stream :error :stream
:wait nil)))
(when ,process
(unwind-protect
(let ((,stream (ccl::external-process-output-stream ,process)))
,#body)
;(ccl:process-wait (ccl:process-whostate ,process) nil)
(close (ccl::external-process-output-stream ,process))
(close (ccl::external-process-error-stream ,process))))))
I know little CCL. I want to know how i can modify this code to support CCL ?
Any suggestion is appreciated !
Apparently trivial-shell:shell-command doesn't allow exactly what you want (it executes the external command synchronously and returns the whole output).
You could look into CCL's run-program. See:
run-program;
Does there exist standard way to run external program in Common Lisp? (this is a question that is similar to your question);
external-program (suggested in one of the answers in the question above) is supported by Quicklisp and it seems to have better support for executing external programs.
You should use trivial-shell.
Trivial shell is a simple platform independent interface to the underlying Operating System.
Related
Sounds deceptively easy. This doesn't work:
~/.sbclrc
(load #P"~/in-package.lisp")
~/in-package.lisp
(in-package 'stored-package)
The package change only applies to the file in-package.lisp itself.
Try different approach: store just the name.
(defmacro recall-package (&optional (filename #p"~/lisp-package.lisp"))
"IN-PACKAGE the contents of the FILENAME"
(let ((p (car (uiop:read-file-lines filename))))
`(in-package ,p)))
This works, but only from ~/.sbclrc. Files which it LOADs expand the macro within their own context, and so it doesn't work.
SBCL reads it's .sbclrc like this:
(restart-case
(flet ((process-init-file (kind specified-pathname default-function)
(awhen (or specified-pathname (funcall default-function))
(with-open-file (stream (if specified-pathname
(parse-native-namestring it)
(pathname it))
:if-does-not-exist nil)
(cond (stream
(sb-fasl::call-with-load-bindings
(lambda (stream kind) (load-as-source stream :context kind))
stream kind stream))
(specified-pathname
(cerror "Ignore missing init file"
"The specified ~A file ~A was not found."
kind specified-pathname)))))))
(unless no-sysinit
(process-init-file "sysinit" sysinit *sysinit-pathname-function*))
(unless no-userinit
(process-init-file "userinit" userinit *userinit-pathname-function*))
Using these fancy sb-fasl::call-with-load-bindings and sb-int:load-as-source yields similar results to the above.
You can't do this with load, because
load binds *readtable* and *package* to the values they held before loading the file.
Function load
This means that any changes made to the values of these variables within a file being loaded are local to the file. That's almost always a good thing. In particular it means that there is no way at all (or no portable way: if you had access to the guts of the dynamic binding mechanism of the implementation this might not be true) that any changes made to the current package (ie the dynamic value of *package*) can ever propagate up through calls to load.
If all you want to do is set the package based on some name in a file, then this is relatively easy, with something like the below:
(defpackage :empty-package
(:use))
(defun set-package-from-file (f)
(let ((pn
(with-standard-io-syntax
;; EP just in case the file does somehow manage to smash
;; *package*)
(let* ((ep (find-package :empty-package))
(*package* ep)
(*read-eval* nil))
(unwind-protect
(with-open-file (in f)
(string (read in)))
;; Clean up EP to avoid leakage
(do-symbols (s ep)
(unintern s ep)))))))
(let ((p (find-package pn)))
(unless p (error "no package ~A" pn))
(setf *package* p))))
This is probably both overly-protective and thus will contain some horrible unexpected bug which I should have thought about (I know it's not safe against interning symbols in other packages). However the idea is that the file contains a single string-designator which should be the package name.
If you had time on your hands you could fairly easily write a version of load which would not rebind *package* &c, and which would work for source files. I think you can't portably write one which would work for FASL files.
Here's one reason why the behaviour the language specifies is the right behaviour: it makes compilation a lot easier. Consider a file which contains:
(in-package ...)
(defun foo (...) ...)
(load ...)
(defun bar (...)
(foo ...)
...)
If *package* could propagate up through load then compiling this file would be, at best, interesting.
Thanks to Common Lisp's powerful macro system, I can write lots of code template to generate functions avoid writing redundant code manually. What's more, it can generate different code based on configure file, so I can implement many kinds of feature just apply different configure file.
However, I have no idea how to deliver the project (It's a library):
In my opinion, maybe every config file corresponds to a package?
For example, there is a common lisp file common.lisp, it generate different functions based different configure file in compile-time.
It reads a.conf in compile-time and generate functions for PackageA and reads config b.conf in compile-time for PackageB. But in-place statement must specify only one package, the common.lisp can't both in Package A and B.
By the way, I still can't find out a proper method to get the configure path of project (So I can read and use it in compile-time to generate functions) I have tried *load-truename* for it points to the cache path which contains .fasl file on SBCLv2.0.1. But it looks like the staic files are not contained in it, so it doesn't works.
For macro-expansion the thing you care about is compile time, not load time, and the variables you want are therefore *compile-file-pathname* &/or *compile-file-truename*. ASDF likes to stash compiled files (and hence the files being loaded) somewhere known to it, which you can turn off (I do) but defaultly they end up somewhere far from their sources.
Here's an example macro which should (I have not really tested it) let you enable debugging output on a per-file basis. In real life it would be better to cache the read of the config file/s but this is mildly fiddly to get right.
(declaim (inline mutter))
(defun mutter (format &rest arguments)
(declare (ignore format arguments))
(values))
(defmacro maybe-debugging (&body forms)
(let ((config-file (and *compile-file-truename*
(make-pathname :name "debug"
:type "cf"
:defaults *compile-file-truename*))))
(multiple-value-bind (debugging cond)
(if (and config-file (probe-file config-file))
(ignore-errors
(with-standard-io-syntax
(let ((*read-eval* nil))
(with-open-file (in config-file)
(values (assoc (pathname-name *compile-file-truename*)
(read in)
:test #'string-equal)
nil)))))
(values nil nil))
(when cond
(warn "bogons reading ~A for ~A: ~A"
config-file *compile-file-truename* cond))
(if debugging
`(flet ((mutter (format &rest arguments)
(apply #'format *debug-io* format arguments)))
,#forms)
`(progn
,#forms)))))
For the single-source-file-resulting-in-multiple-object-files you could do something like this (note this repeats a variant of the above code):
(eval-when (:load-toplevel :compile-toplevel :execute)
(defvar *package-compilation-configuration*
nil
"Compile-time configuration for a package")
(defun package-config-value (key &optional (default nil))
(getf *package-compilation-configuration* key default)))
(declaim (inline mutter))
(defun mutter (format &rest args)
(declare (ignore format args))
(values))
(defmacro with-muttering (&body forms)
(if (package-config-value ':mutter)
`(flet ((mutter (fmt &rest args)
(apply #'format *debug-io* fmt args)))
,#forms)
`(progn
,#forms)))
(defun compile-file-for-package (file package &rest kws
&key (output-file nil output-file-p)
&allow-other-keys)
(with-muttering
(let* ((sf-pathname (pathname file))
(package-file (make-pathname :name (string package)
:type "cf"
:defaults sf-pathname))
(the-output-file
(if output-file-p
output-file
(compile-file-pathname
(make-pathname :name (format nil "~A-~A"
(pathname-name sf-pathname)
package)
:defaults sf-pathname))))
(*package-compilation-configuration*
(if (probe-file package-file)
(with-standard-io-syntax
(mutter "~&Compile ~A -> ~A using ~A~%"
sf-pathname the-output-file package-file)
(let ((*read-eval* nil))
(with-open-file (in package-file)
(read in))))
(progn
(mutter "~&Compile ~A -> ~A (no package)~%"
sf-pathname the-output-file)
nil))))
(apply #'compile-file file
:output-file the-output-file
kws))))
Then (compile-file-for-package "x.lisp" "y") will compile x.lisp having read configuration for package "y".
To use something like this in anger you would need to integrate it with ASDF and I don't know how to do that.
An alternative idea is just to use symlinks for the source files, and have the filename-dependent configuration depend on the symlink name, not the target name.
For my case:
project-a.asd:
(asdf:defsystem #:project-a
:components ((:static-file "my-config-file.conf")
(:static-file "common.lisp") ; shared common lisp file
(:file "project-a-package")
(:file "project-a-setup")
;; other components
)
)
project-a-setup.lisp:
(in-package #:project-a)
(eval-when (:compile-toplevel)
(defvar *mypackage* (find-package 'project-a))
(defvar *source-home* (path:dirname *compile-file-truename*))
;; read configure file
(defparameter *myconf*
(with-open-file (stream (merge-pathnames *source-home* #P"my-config-file.conf"))
(read stream)))
)
(load (merge-pathnames *source-home* #P"common.lisp"))
common.lisp:
(let ((*package* *mypackage*))
;; intern symbol
)
I am trying to add to a program I am writing, a feature whereby everything printed to the console, also gets added to a log file. This much can be done with broadcast streams. The problem is that the program may also need to abruptly exit from within a leaf function, and when I do this, the log file does not get created. This is what I have so far:
(catch 'quit
(with-open-file (log-stream "log.txt"
:direction :output
:if-exists :supersede
:if-does-not-exist :create)
(let ((*standard-output*
(make-broadcast-stream *standard-output* log-stream)))
(format t "abc~%")
(throw 'quit nil))))
When I run the above code (SBCL 1.4.2, Windows 7), the file log.txt does not get created. The same is true if I replace (throw 'quit nil) with (quit). However, if I remove that line altogether and just let the program exit by falling off the end of the file, the log file does get correctly created, which suggests it's a caching issue.
Is that the correct diagnosis? If so, is there a way to tell the compiler not to cache that file, or to exit with rather than without writing cached data?
This is the behaviour described in the standard for WITH-OPEN-FILE:
If a new output file is being
written, and control leaves abnormally, the file is aborted and the file system is left,
so far as possible, as if the file had never been opened.
The following explicitly closes the file:
(catch 'quit
(with-open-file (log-stream "/tmp/log.txt"
:direction :output
:if-exists :supersede
:if-does-not-exist :create)
(let ((*standard-output* (make-broadcast-stream *standard-output* log-stream)))
(unwind-protect (progn
(format t "abc~%")
(throw 'quit nil))
(finish-output)
(close log-stream :abort nil)))))
The :abort nil value is the default one, it is made explicit here for the sake of the answer.
I was hoping to experiment with cl-async to run a series of external programs with a large combinations of command line arguments. However, I can't figure out how to read the stdout of the processes launched with as:spawn.
I would typically use uiop which makes it easy to capture the process output:
(let ((p (uiop:launch-program ... :output :stream)))
(do-something-else-until-p-is-done)
(format t "~a~%" (read-line (uiop:process-info-output p))))
I've tried both :output :pipe and :output :stream options to as:spawn and executing (as:process-output process-object) in my exit-callback shows the appropriate pipe or async-stream objects but I can't figure out how to read from them.
Can anyone with experience with this library tell how to accomplish this?
So you go to your repl and type:
CL-USER> (documentation 'as:spawn 'function)
And you read whatever comes out (or put your point on the symbol and hit C-c C-d f). If you read it you’ll see that the format for the :input, etc arguments is either :pipe, (:pipe args...), :stream, or (:stream args...) (or some other options). And that :stream behaves similarly to :pipe but gives output of a different type and that for details of args one should look at PIPE-CONNECT so you go and look up the documentation for that. Well it tells you what the options are but it isn’t very useful. What’s the documentation/description of PIPE or STREAM? Well it turns out that pipe is a class and a subclass of STREAMISH. What about PROCESS that’s a class too and it has slots (and accessors) for things like PROCESS-OUTPUT. So what is a good plan for how to figure out what to do next? Here’s a suggestion:
Spawn a long running process (like cat foo.txt -) with :output :stream :input :pipe say
Inspect the result (C-c C-v TAB)
Hopefully it’s an instance of PROCESS. What is it’s output? Inspect that
Hopefully the output is a Gray stream (ASYNC-STREAM). Get it into your repl and see what happens if you try to read from it?
And what about the input? See what type that has and what you can do with it
The above is all speculation. I’ve not tried running any of this but you should. Alternatively go look at the source code for the library. It’s already on your computer and if you can’t find it it’s on GitHub. There are only about half a dozen source files and they’re all small. Just read them and see what you can learn. Or go to the symbol you want to know about and hit M-. to jump straight to its definition. Then read the code. Then see if you can figure out what to do.
I found the answer in the test suite. The output stream can only be processed asynchronously via a read call-back. The following is simple example for posterity
(as:start-event-loop
(lambda ()
(let ((bytes (make-array 0 :element-type '(unsigned-byte 8))))
(as:spawn "./test.sh" '()
:exit-cb (lambda (proc exit-status term-signal)
(declare (ignore proc exit-status term-signal))
(format t "proc output:~%~a"
(babel:octets-to-string bytes)))
:output (list :stream
:read-cb (lambda (pipe stream)
(declare (ignore pipe))
(let ((buf (make-array 128 :element-type '(unsigned-byte 8))))
(loop for n = (read-sequence buf stream)
while (plusp n) do
(setf bytes
(concatenate '(vector (unsigned-byte 8))
bytes
(subseq buf 0 n)))))))))))
with
$ cat test.sh
#!/bin/bash
sleep_time=$((1+$RANDOM%10))
echo "Process $$ will sleep for $sleep_time"
sleep $sleep_time
echo "Process $$ exiting"
yields the expected output
I try to get a practical development environment for R in Emacs, hoping to get auto-completion working as shown in http://www.emacswiki.org/emacs/ESSAuto-complete.
However, even in a minimal configuration, I can't get it working.
See what I get on http://screencast.com/t/qcyVwkECX. In fact, while AC does work (see completion menu appearing), it's like if there was no info from the R language itself, while ac-source-R is WELL added to ac-sources.
Do you understand what's happening?
Best regards.
PS- Here is my minimal Emacs configuration file for the demo:
;; Auto Completion
(add-to-list 'load-path "~/.emacs.d/elpa/auto-complete-20140824.1658/")
(add-to-list 'load-path "~/.emacs.d/elpa/popup-20140815.629/")
(when (require 'auto-complete-config)
(ac-config-default)
;; use `C-n/C-p' to select candidates
(setq ac-use-menu-map t)
(define-key ac-menu-map (kbd "C-n") 'ac-next)
(define-key ac-menu-map (kbd "C-p") 'ac-previous)
;; unbind some keys (inconvenient in iESS buffers)
(define-key ac-completing-map (kbd "M-n") nil)
(define-key ac-completing-map (kbd "M-p") nil)
;; set default sources
(setq ac-sources
(append '(ac-source-features
ac-source-functions
ac-source-yasnippet
ac-source-variables
ac-source-symbols)
ac-sources))
(setq ac-delay 0) ; faster than default 0.1
(setq ac-auto-show-menu 0.2)
(setq ac-quick-help-delay 0.5)
(setq ac-quick-help-height 10)
(setq ac-candidate-limit 100)
;; completion by TAB
(define-key ac-completing-map
(kbd "<tab>") 'ac-complete)
;; avoid Flyspell processes when auto completion is being started
(ac-flyspell-workaround))
;; ESS: Emacs Speaks Statistics
(add-to-list 'load-path "~/.emacs.d/elpa/ess-20140824.1452/lisp/")
(setq shell-file-name "zsh.exe")
(add-to-list 'auto-mode-alist '("\\.[rR]\\'" . R-mode))
(autoload 'R "ess-site" "Call 'R', the 'GNU S' system from the R Foundation." t)
(autoload 'R-mode "ess-site" "Major mode for editing R source." t)
(setq ess-ask-for-ess-directory nil)
(setq inferior-ess-same-window nil)
(setq ess-default-style 'DEFAULT)
(with-eval-after-load "ess-site"
;; use eldoc to report R function names
(require 'ess-eldoc)
(add-hook 'inferior-ess-mode-hook 'ess-use-eldoc))
PPS- In fact, ElDoc does not seem to work either!
UPDATE
I just discovered it almost works when the iESS buffer gets created, not when just editing R code in its own buffer. See http://screencast.com/t/fKRjLmIC6K0.
What would explain that iESS must be run first before it finally works?
Still, something that does not work is the completion on function arguments (like with the cat function on the page http://www.emacswiki.org/emacs/ESSAuto-complete).
Why does that not work?
After opening a R file with emacs, if you have this mode described into your emacs status bar:
(ESS[S] [none] ElDoc AS)
You can run this shortcut C-c C-s to attach a R session to you ESS[S] editor mode.
If you already have one or some R session open, emacs will ask you to choose the R session you want to use. Otherwise if you have not already open a R session then emacs will open a new one for you.
Next, you should have this information inside your status bar:
(ESS[S] [R db -] ElDoc AS)
and the completion should work.