Using stdout from shell script in common lisp - common-lisp

I am writing a common lisp program that needs to handle the output of a command. However, when I try to use the results in another function, I only get a NIL as return-value.
Here is the function I use to run commands:
(defun run-command (command &optional arguments)
(with-open-stream (pipe
(ext:run-program command :arguments arguments
:output :stream :wait nil))
(loop
:for line = (read-line pipe nil nil)
:while line :collect line)))
Which, when run by itself gives:
CL-USER> (run-command "ls" '("-l" "/tmp/test"))
("-rw-r--r-- 1 petergil petergil 0 2011-06-23 22:02 /tmp/test")
However, when I run it through a function, only NIL is returned:
(defun sh-ls (filename)
(run-command "ls" '( "-l" filename)))
CL-USER> (sh-ls "/tmp/test")
NIL
How can I use the results in my functions?

Try this:
(defun sh-ls (filename)
(run-command "ls" (list "-l" filename)))
The '("-l" filename) is quoting the list, AND the symbol 'filename', rather than evaluating filename.

You can also use backquote ` before sexpr and , before filename to evaluate it:
(defun sh-ls (filename)
(run-command "ls" `("-l" ,filename)))

Related

Cannot get input stream working in SBCL sb-ext:run-program

While the following works:
(let* ((i (make-string-input-stream "foo bar baz"))
(p (sb-ext:run-program "/bin/cat" '()
:input i :output *trace-output* :wait t)))
(sb-ext:process-close p))
the code below does not - it will stop after writing "001":
(let* ((_1 (format t "001~%"))
(p (sb-ext:run-program "/bin/cat" '()
:input :stream :output *trace-output* :wait t))
(_2 (format t "010~s~%" p))
(s (sb-ext:process-input p)))
(declare (ignore _1 _2))
(format s "foo bar baz~%")
(finish-output s)
(sb-ext:process-close p))
So it seems to silently leave execution in sb-ext:run-program.
This is with SBCL 1.3.6 on Ubuntu 16.04.1.
Any ideas? Thanks in advance, Frank
As I mentioned in the comments, the problem is the :WAIT T argument. It causes the call to SB-EXT:RUN-PROGRAM to not return until the child process exits.
In the first example you passed a string input stream to the child process. cat will read input from the stream, and when the input ends there will be a End of File, so cat exits. In the second example there is no input available for the program, so it's effectively an infinite loop (just like if you run cat on the command line, and don't give any input to it; it will never exit).
The solution is to use :WAIT NIL. You will also have to close the input stream with CLOSE, because otherwise there will be no EOF and cat keeps listening for more input. You'll also want to use SB-EXT:PROCESS-WAIT after closing the stream to wait for cat to exit itself.
(let* ((p (sb-ext:run-program "/bin/cat" '()
:input :stream
:output *standard-output*
:wait nil))
(s (sb-ext:process-input p)))
(format s "foo bar baz~%")
(finish-output s)
(close s)
(sb-ext:process-wait p)
(sb-ext:process-close p))
I'm not sure why you used *TRACE-OUTPUT* for the child output, so I changed it to *STANDARD-OUTPUT*.
Also, using FORMAT for debugging like that is kind of ugly. Common Lisp provides actual debugging tools. In this case you could use STEP:
(step (let* ((p (sb-ext:run-program "/bin/cat" '()
:input :stream
:output *standard-output*
:wait nil))
(s (sb-ext:process-input p)))
(format s "foo bar baz~%")
(finish-output s)
(close s)
(sb-ext:process-wait p)
(sb-ext:process-close p)))
This will put you in the debugger, showing the call being evaluated next. You can invoke the STEP-NEXT-restart to continue to the next call.
This is what works, as suggested by jkiiski:
(let* ((p (sb-ext:run-program "/bin/cat" '()
:input :stream
:output *standard-output*
:wait nil))
(s (sb-ext:process-input p)))
(format s "foo bar baz~%")
(finish-output s)
(sb-ext:process-wait p)
(sb-ext:process-close p))

common lisp: how to suppress newline or "soft return"

This code
(defun arabic_to_roman (filename)
(let ((arab_roman_dp '())
(arab nil)
(roman nil))
(with-open-file (in filename
:direction :input
:if-does-not-exist nil)
(when in
(loop for line = (read-line in nil)
while line do
(setq arab (subseq line 0 (search "=" line)))
(setq roman (subseq line (1+ (search "=" line)) (length line)))
(setf arab_roman_dp (acons arab roman arab_roman_dp))
;(format t "~S ~S~%" arab roman)
)))
(with-open-file (stream #p"ar_out.txt"
:direction :output
:if-exists :overwrite
:if-does-not-exist :create )
(write arab_roman_dp :stream stream :escape nil :readably nil))
'done!))
seems to work well. It takes a file with entries like this
1=I
2=II
...
and builds one large list of dotted pairs. However when I look at the output file, it seems as though soft returns or newlines have been inserted.
((4999 . MMMMCMXCIX) (4998 . MMMMCMXCVIII) (4997 . MMMMCMXCVII)
(4996 . MMMMCMXCVI) (4995 . MMMMCMXCV) (4994 . MMMMCMXCIV)
(4993 . MMMMCMXCIII) (4992 . MMMMCMXCII) (4991 . MMMMCMXCI) (4990 . MMMMCMXC)
...
I was expecting the output to look more like just one continuous line:
((4999 . MMMMCMXCIX) (4998 . MMMMCMXCVIII) (4997 . MMMMCMXCVII) (4996 . MMMCMXCVI) (4995 . MMMMCMXCV) (4994 . MMMMCMXCIV) (4993 . MMMMCMXCIII) (4992 . MMMMCMXCII) (4991 . MMMMCMXCI) (4990 . MMMMCMXC) ...
Is my code the way it is indeed throwing in newlines somehow? I've used the write version of princ which supposedly suppresses newlines. Later I want to read this file back into the program as just one big list, so I don't want newline issues.
It looks like the pretty-printer is being invoked (the default is implementation-dependent), to print it with indentation and human-readable line lengths. Use :pretty nil to disable this.
(write arab_roman_dp :stream stream :escape nil :readably nil :pretty nil)
A better way to write it:
use functions to create blocks of code, which can be combined
less side effects and fewer variables
no need to test for in
easy to understand control flow
Example:
(defun arabic_to_roman (filename)
(flet ((read-it ()
(with-open-file (in filename
:direction :input
:if-does-not-exist nil)
(loop for line = (read-line in nil)
for pos = (position #\= line)
while line collect (cons (subseq line 0 pos)
(subseq line (1+ pos))))))
(write-it (list)
(with-open-file (stream #p"ar_out.txt"
:direction :output
:if-exists :overwrite
:if-does-not-exist :create)
(write list :stream stream :escape nil :readably nil :pretty nil))))
(write-it (read-it))
'done-read-write))

How to interact with a process input/output in SBCL/Common Lisp

I have a text file with one sentence per line. I would like to lemmatize the worlds in each line using hunspell (-s option). Since I want to have the lemmas of each line separately, it wouldn't make sense to submit the whole text file to hunspell. I do need to send one line after another and have the hunspell output for each line.
Following the answers from How to process input and output streams in Steel Bank Common Lisp?, I was able to send the whole text file for hunspell one line after another but I was not able to capture the output of hunspell for each line. How interact with the process sending the line and reading the output before send another line?
My current code to read the whole text file is
(defun parse-spell-sb (file-in)
(with-open-file (in file-in)
(let ((p (sb-ext:run-program "/opt/local/bin/hunspell" (list "-i" "UTF-8" "-s" "-d" "pt_BR")
:input in :output :stream :wait nil)))
(when p
(unwind-protect
(with-open-stream (o (process-output p))
(loop
:for line := (read-line o nil nil)
:while line
:collect line))
(process-close p))))))
Once more, this code give me the output of hunspell for the whole text file. I would like to have the output of hunspell for each input line separately.
Any idea?
I suppose you have a buffering problem with the program you want to run. For example:
(defun program-stream (program &optional args)
(let ((process (sb-ext:run-program program args
:input :stream
:output :stream
:wait nil
:search t)))
(when process
(make-two-way-stream (sb-ext:process-output process)
(sb-ext:process-input process)))))
Now, on my system, this will work with cat:
CL-USER> (defparameter *stream* (program-stream "cat"))
*STREAM*
CL-USER> (format *stream* "foo bar baz~%")
NIL
CL-USER> (finish-output *stream*) ; will hang without this
NIL
CL-USER> (read-line *stream*)
"foo bar baz"
NIL
CL-USER> (close *stream*)
T
Notice the finish-output – without this, the read will hang. (There's also force-output.)
Python in interactive mode will work, too:
CL-USER> (defparameter *stream* (program-stream "python" '("-i")))
*STREAM*
CL-USER> (loop while (read-char-no-hang *stream*)) ; skip startup message
NIL
CL-USER> (format *stream* "1+2~%")
NIL
CL-USER> (finish-output *stream*)
NIL
CL-USER> (read-line *stream*)
"3"
NIL
CL-USER> (close *stream*)
T
But if you try this without the -i option (or similar options like -u), you'll probably be out of luck, because of the buffering going on. For example, on my system, reading from tr will hang:
CL-USER> (defparameter *stream* (program-stream "tr" '("a-z" "A-Z")))
*STREAM*
CL-USER> (format *stream* "foo bar baz~%")
NIL
CL-USER> (finish-output *stream*)
NIL
CL-USER> (read-line *stream*) ; hangs
; Evaluation aborted on NIL.
CL-USER> (read-char-no-hang *stream*)
NIL
CL-USER> (close *stream*)
T
Since tr doesn't provide a switch to turn off buffering, we'll wrap the call with a pty wrapper (in this case unbuffer from expect):
CL-USER> (defparameter *stream* (program-stream "unbuffer"
'("-p" "tr" "a-z" "A-Z")))
*STREAM*
CL-USER> (format *stream* "foo bar baz~%")
NIL
CL-USER> (finish-output *stream*)
NIL
CL-USER> (read-line *stream*)
"FOO BAR BAZ
"
NIL
CL-USER> (close *stream*)
T
So, long story short: Try using finish-output on the stream before reading. If that doesn't work, check for command line options preventing buffering. If it still doesn't work, you could try wrapping the programm in some kind of pty-wrapper.

How to download and save a file using drakma:http-request and flexistreams

I am trying to download and save a PDF but it fails while writing with an EOF-Error.
What would be the correct way of doing this?
(with-open-file (file "/home/*/test.pdf"
:direction :io
:if-does-not-exist :create
:if-exists :supersede
:element-type '(unsigned-byte 8))
(let ((input (drakma:http-request "http://www.fractalconcept.com/ex.pdf"
:want-stream t)))
(awhile (read-byte input)
(write-byte it file))
(close input)))
The solution was that I forgot to use the two optional parameters of read-byte.
The correct way would be to set eof-error-p and eof-value to nil:
(with-open-file (file "/home/*/test.pdf"
:direction :output
:if-does-not-exist :create
:if-exists :supersede
:element-type '(unsigned-byte 8))
(let ((input (drakma:http-request "http://www.fractalconcept.com/ex.pdf"
:want-stream t)))
(awhile (read-byte input nil nil)
(write-byte it file))
(close input)))
(ql:quickload "trivial-download")
(trivial-download:download URL FILE)
; Does pretty much exactly what your code does but in bigger chunks and can show a progress bar.
; QuickLisp can be found at http://QuickLisp.org.
AWHILE is found in package ARNESI.

How to get the shell function's output in slime?

In REPL, we can get its output normally:
(shell "head -3 ~/misc.lisp")
(asdf:make-build 'stumpwm :type :program :monolithic t
:move-here "."
:name-suffix ""
However, we get nil from (shell "head -3 ~/misc.lisp") in slime.
Sincerely!
I don't know which Lisp implementation you're using, but normally you should need a package to run shell command like:
trivial-shell
inferior-shell
clesh (build on top of trivial-shell)
use #'run-shell-command instead of shell:
(let ((str (ext:run-shell-command cmd :output :stream)))
(loop for line = (read-line str nil)
until (null line)
do (print line)))
it works now.

Resources