I want to sent some data from my program to a process executed via uiop:run-program.
The following works:
(require :asdf)
(asdf:load-system :uiop)
(uiop:with-temporary-file (:stream dot-program-stream
:pathname dot-program-file)
(format dot-program-stream "digraph g { n1 -> n2; }")
(finish-output dot-program-stream)
:close-stream
(uiop:with-temporary-file (:pathname png-data)
(uiop:run-program '("/usr/bin/dot" "-Tpng") :input dot-program-file
:output png-data)
(uiop:launch-program '("/usr/bin/display") :input png-data)))
It seems rather convoluted.
A simpler version, where I used only a stream did not finish-output and did not use the :close-stream label resulted in dot producung an empty 0 byte file.
How to execute a process and pass it data generated by my lisp program as standard input?
Take a closer look at the documentation of uiop:launch-program and uiop:run-program, especially the options for the :input and :output keys.
You can call launch-program with :input :stream. Launch-program returns a process info object that contains the stream connected to that program's standard input behind the accessor process-info-input, so you can print to that.
If you have a different program that produces output that should go into that input stream, you have several options:
create a temporary file, then read it and print it to the second program's input stream (that seems to be your current approach)
use run-program with :output :string for the first call, then use launch-program with :input :stream for the second and write the output of the first to that stream
use launch-program also for the first call, in this case with :output :stream, then read from that output and print it to the second program's input
You can either read everything first, then write everything, or you can do buffered reading and writing, which might be interesting for long running processes.
Instead of this in-process buffering, you could also use a fifo (named pipe) from your OS.
Related
I was hoping to experiment with cl-async to run a series of external programs with a large combinations of command line arguments. However, I can't figure out how to read the stdout of the processes launched with as:spawn.
I would typically use uiop which makes it easy to capture the process output:
(let ((p (uiop:launch-program ... :output :stream)))
(do-something-else-until-p-is-done)
(format t "~a~%" (read-line (uiop:process-info-output p))))
I've tried both :output :pipe and :output :stream options to as:spawn and executing (as:process-output process-object) in my exit-callback shows the appropriate pipe or async-stream objects but I can't figure out how to read from them.
Can anyone with experience with this library tell how to accomplish this?
So you go to your repl and type:
CL-USER> (documentation 'as:spawn 'function)
And you read whatever comes out (or put your point on the symbol and hit C-c C-d f). If you read it you’ll see that the format for the :input, etc arguments is either :pipe, (:pipe args...), :stream, or (:stream args...) (or some other options). And that :stream behaves similarly to :pipe but gives output of a different type and that for details of args one should look at PIPE-CONNECT so you go and look up the documentation for that. Well it tells you what the options are but it isn’t very useful. What’s the documentation/description of PIPE or STREAM? Well it turns out that pipe is a class and a subclass of STREAMISH. What about PROCESS that’s a class too and it has slots (and accessors) for things like PROCESS-OUTPUT. So what is a good plan for how to figure out what to do next? Here’s a suggestion:
Spawn a long running process (like cat foo.txt -) with :output :stream :input :pipe say
Inspect the result (C-c C-v TAB)
Hopefully it’s an instance of PROCESS. What is it’s output? Inspect that
Hopefully the output is a Gray stream (ASYNC-STREAM). Get it into your repl and see what happens if you try to read from it?
And what about the input? See what type that has and what you can do with it
The above is all speculation. I’ve not tried running any of this but you should. Alternatively go look at the source code for the library. It’s already on your computer and if you can’t find it it’s on GitHub. There are only about half a dozen source files and they’re all small. Just read them and see what you can learn. Or go to the symbol you want to know about and hit M-. to jump straight to its definition. Then read the code. Then see if you can figure out what to do.
I found the answer in the test suite. The output stream can only be processed asynchronously via a read call-back. The following is simple example for posterity
(as:start-event-loop
(lambda ()
(let ((bytes (make-array 0 :element-type '(unsigned-byte 8))))
(as:spawn "./test.sh" '()
:exit-cb (lambda (proc exit-status term-signal)
(declare (ignore proc exit-status term-signal))
(format t "proc output:~%~a"
(babel:octets-to-string bytes)))
:output (list :stream
:read-cb (lambda (pipe stream)
(declare (ignore pipe))
(let ((buf (make-array 128 :element-type '(unsigned-byte 8))))
(loop for n = (read-sequence buf stream)
while (plusp n) do
(setf bytes
(concatenate '(vector (unsigned-byte 8))
bytes
(subseq buf 0 n)))))))))))
with
$ cat test.sh
#!/bin/bash
sleep_time=$((1+$RANDOM%10))
echo "Process $$ will sleep for $sleep_time"
sleep $sleep_time
echo "Process $$ exiting"
yields the expected output
I am slowly getting closer to be able to read and write to/from named pipes of a background process through SBCL. What I do is kick off the program I am trying to read/write from/to:
todd#ubuntu:~/CoreNLP$ cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text > ./spout &
[1] 24616
So that all works out fine, so I kick off SBCL and do this:
(defparameter from-corenlp (open "./spout"))
Which also works out fine, but declaring the stream causes SBCL to spill the stream onto the screen (which is all the startup information from the background process). It does not wait until I read from the stream. Is that how things are supposed to work?
The solution, as I posted it to the stanford parser mailing list (stack overflow reformatted a lot of it to something weird, but you get the idea):
It took quite a while, but I finally figured out embedding (for the most part) the CoreNLP program (while in interactive mode) in SBCL Lisp.
First of all, forget using (sb-ext:run-program ...). This combination of spawning Java with a quoted argument (like the asterisk) no matter how well escaped, simply makes the spawned program crash.
Inferior shell seems to kick off the parser but it is only good for a one-off parse, even in the interactive mode. Perhaps I could have done better, but inferior shell needs to be installed and it is poorly documented.
The initial attempted solution of using Unix named pipes ends up being the final one, but it took a bit of work, first with buffering, then with the order of operations, and finally understanding some nuances about the parser program.
First, turning off buffering completely when running the program is important, so running it looks like this:
stdbuf --i=0 --o=0 --e=0 cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text > ./spout &
That is supposed to be running the parser in the background accepting input from spin and sending its output to spout. But if you look at the process table in Linux, you will not see it running. It is still waiting for something to pull from the output pipe before it can even run.
So, we run SBCL and start a stream pulling from the parser´s pipe:
(defparameter *from-corenlp* (open "./spout"))
NOW the parser starts running. Here, oddly, it also starts dumping output to the screen, not to the pipe! That is because all of this banner stuff when the parser starts and stops (and apparently even the NLP> prompt) is sent to stderr, not stdout. This is actually a good thing.
So then we declare the stream from Lisp to the parser:
(defparameter *to-corenlp* (open "./spin" :direction :output :if-exists :append))
Then we send some text for the parser to parse:
(write-line "This is the first test." *to-corenlp*)
I ran into a problem here a few times, even. Remember that Lisp has its own buffer so you have to clear out the stream every time:
(finish-output *to-corenlp*)
You then can run this line below a whole bunch of times to verify you obtain the exact same behavior you would have gotten from an interactive session of the parser:
(format t "~a~%" (read-line *from-corenlp*))
Which, if you are a good boy scout, should not only be true, but you can carry on with your interactive slave parser session for as long as you like:
(write-line "This is the second test." *to-corenlp*)
(finish-output *to-corenlp*)
Isn´t that great? And notice I pulled all of that off being terrible at Unix, terrible at Lisp and being a terrible boy scout!
Now so can you!
I understand that Scheme uses ports to perform Input and Output. While trying to learn how to get console input and output, I have come across MIT-Scheme's console-i/o-port variable.
But, the guile interpreter says it is an Unbound Variable. I would like to know how we can use ports to get input from and output to the console (Terminal in Unix) in a Guile Scheme Script. I am still a rookie in Scheme and Linux, a clear step-by-step is appreciated.
Also, how does (display <object>) work? Does it use ports inherently or is there another way.
P.S. If there is another way without using ports please let me know how to use that too.
If you want to read and write SExps, in guile you have (read), (write), (display) etc., if you want to read characters only use (read-char) and (write-char) -- they all use the input/output ports resp. you picked, by default they are stdin and stdout. Everything is rather straightforward (https://www.gnu.org/software/guile/manual/html_node/Input-and-Output.html#Input-and-Output).
You might also be interested in guile-ncurses (https://www.gnu.org/software/guile-ncurses/).
Of some more goodies check out pretty-print module from ice-9 (on very long sexps it's slow but outputs them really nicely formatted, great for e.g. code generation):
(use-modules (ice-9 pretty-print))
(pretty-print `(super cool stuff (+ 2 3) => ,(+ 2 3)))
And if you need your own parser, check out the lalr module (system base lalr).
edit a small example which reads a number, multiplies by itself and prints out the result:
#!/usr/bin/guile -s
!#
(let ((x (read)))
(display (* x x))
(newline))
(remember to chmod +x this script).
edit changed the expression to let form as Chris suggested, indeed the fewer parentheses the better
In guile you have 2 functions: current-input-port and current-output-port (the docs)
to read and put text into string (if you don't want to read s-expressions) you can use this function:
(define (read-port port)
(let iter ((result '()) (chr (read-char port)))
(if (eof-object? chr)
(list->string result)
(iter (append result (list chr)) (read-char port)))))
reading from stdin will be:
(read-port (current-input-port))
to write to stdout you can use display it also accept second argument which is port relevant docs
It might seem simple, but I can't get it to work. I simply need to read a file where the contents are just one big list
(a b c d)
. . . as is . . . into a list in my program. I have
(let ((ardplst nil))
...
(with-open-file (in ardpfile :direction :input :if-does-not-exist nil)
(when in
(read-sequence ardplst in))
(format t "~a" ardplst))
But it's not working. I get NIL. What am I doing wrong?
What does read-sequence do? It reads some elements from the stream, typically characters (but it depends on the element-type of the stream) and destructively insert them into the input sequence. So, you would collect characters #\(, then #\a, then #\Space, then #\b, etc. However, reading stops as soon as you reach the end of your sequence: with your empty list, that means immediately (you are supposed to pass a buffer, e.g. a vector). In you case, read-sequence returns 0.
The reason you get nil is because your last expression is format, which in the above code outputs to the standard output (because of t) and returns nil. You could have used print, which returns the printed object.
I don't understand why you are explicitely using :if-does-not-exist nil. Are you sure you want to silently skip the task if the file cannot be opened? What if the list you read is empty? You should probably let an error be signaled in case the file is not found.
I would use read while disabling read-time evaluation:
(with-open-file (in my-file)
(let* ((*read-eval* nil)
(list (read in)))
(prog1 list
(check-type list list))))
Note that the default :direction is :input. In my opinion it does not hurt to omit this argument here, though sometimes it can be more readable to write it explicitely.
I'm using a local emacs instance (aquamacs) to run R processes on a remote server, and I'd like to automate the process of connecting to my server. The process is as follows:
[in emacs]
M-x shell
[in the resulting console]
TERM=xterm
ssh -Y -C <my remote server>
screen -rd [and/or] R
[in emacs]
M-x ess-remote
r
I discovered this general approach here: http://blog.nguyenvq.com/2010/07/11/using-r-ess-remote-with-screen-in-emacs/. The -Y -C options allow you use xterm to view plots. I don't know lisp and tho I've googled around a bit, I can't seem to piece together how to actually define a function to automate this (e.g., in .emacs.el). Has anyone implemented anything like this?
Let's assume you just want to call shell in code. In Lisp, everything is prefix notation surrounded by parentheses. So we enter this into a buffer (say, the scratch buffer):
(shell)
Move your pointer to the end of the line after the close-paren, and type <C-x C-e> to execute the Lisp code. You should see that the shell function is called.
Now, let's make it a function, so we can add other things to it. The command to create a function is defun, and it takes the name of the function, the argument list (in parentheses), and then the body of the function:
(defun automate-connection ()
(shell))
Move your cursor to the end of the code, hit <C-x C-e>, and the function will be defined. You can call it from Lisp by executing
(automate-connection)
Ok, now we just need to put some text into the shell buffer.
(defun automate-connection ()
(shell)
(insert "TERM=xterm"))
Now, when we run that, we get "TERM=xterm" put into the shell buffer. But it doesn't actually send the command. Let's try putting a newline.
(defun automate-connection ()
(shell)
(insert "TERM=xterm\n"))
That puts in a newline, but doesn't actually make the command run. Why not? Let's see what the enter key does. Go to your *shell* buffer, and type <C-h c>, then hit the return key. (<C-h c> runs describe-key-briefly, which prints the name of the function invoked by hitting the given key). That says that when you hit RET, it's not putting a newline, but actually calling comint-send-input. So let's do that:
(defun automate-connection ()
(shell)
(insert "TERM=xterm")
(comint-send-input))
Now, when you run `(automate-connection) from any Lisp code, you should get the given thing sent. I leave it as an exercise to the reader to add your other commands.
But wait! We're not really done, are we? I assume you don't want to have to move to a Lisp scratch buffer, type in (automate-connection), then evaluate that code. You probably just want to type , and call it a day. You can't do that by default with the function we just created. Luckily, it's simple to allow that: just add a call to (interactive) in your function:
(defun automate-connection ()
(interactive)
(shell)
(insert "TERM=xterm")
(comint-send-input))
Now you can call it as you want, and it'll open the *shell* buffer, put in the text, and tell Emacs to tell the shell to run that text.