The following fragment of CL code does not work as I expected with CCL running SLIME. If I
first compile and load the file using C-c C-k, and then run
(rdirichlet #(1.0 2.0 3.0) 1.0)
in the SLIME/CCL REPL, I get the error
value 1.0 is not of the expected type DOUBLE-FLOAT.
[Condition of type TYPE-ERROR]
It works with SBCL. I expected the (setf *read-default-float-format* 'double-float)) to allow me to use values like 1.0. If I load this file into CCL using LOAD at the REPL it works. What am I missing?
(eval-when (:compile-toplevel :load-toplevel :execute)
(require :asdf) (require :cl-rmath) (setf *read-default-float-format* 'double-float))
(defun rdirichlet (alpha rownum)
;;(declare (fixnum rownum))
(let* ((alphalen (length alpha))
(dirichlet (make-array alphalen :element-type '(double-float 0.0 *) :adjustable nil :fill-pointer nil :displaced-to nil)))
(dotimes (i alphalen)
(setf (elt dirichlet i) (cl-rmath::rgamma (elt alpha i) rownum)))
;; Divide dirichlet vector by its sum
(map 'vector #'(lambda (x) (/ x (reduce #'+ dirichlet))) dirichlet)))
UPDATE: I forgot to mention my platform and versions. I'm using Debian squeeze x86. The version of SLIME is from Debian unstable, 1:20120525-2.
CCL is the 1.8 release. I tried it both with the upstream binaries from http://svn.clozure.com/publicsvn/openmcl/release/1.8/linuxx86/ccl, and a binary package created by me - see Package ccl at mentors.debian.net. The result was the same in each case.
It seems probable that this issue is SLIME specific. It would be helpful if people could comment whether they see this behavior or not. Also,
what is the equivalent of C-c C-k in SLIME, if one is running CCL in basic command line mode? (LOAD filename), or something else? Or, to ask a slightly different question, what CCL function is C-c C-k calling?
I notice that calling C-c C-k on the following code
(eval-when (:compile-toplevel :load-toplevel :execute)
(require :asdf) (require :cl-rmath) (setf *read-default-float-format* 'double-float))
(print *read-default-float-format*)
produces DOUBLE-FLOAT, though even though *read-default-float-format* at the REPL immediately afterwards gives SINGLE-FLOAT.
UPDATE 2: It looks like, as Rainer said, that the compilation occurs in a separate thread.
Per the function all-processes in Threads Dictionary
printing all-processes from the buffer using C-c C-k gives
(#<PROCESS worker(188) [Active] #x18BF99CE> #<PROCESS repl-thread(12) [Semaphore timed wait] #x187A186E> #<PROCESS auto-flush-thread(11) [Sleep] #x187A1C9E> #<PROCESS swank-indentation-cache-thread(6) [Semaphore timed wait] #x186C128E> #<PROCESS reader-thread(5) [Active] #x186C164E> #<PROCESS control-thread(4) [Semaphore timed wait] #x186BE3BE> #<PROCESS Swank Sentinel(2) [Semaphore timed wait] #x186BD0D6> #<TTY-LISTENER listener(1) [Active] #x183577B6> #<PROCESS Initial(0) [Sleep] #x1805FCCE>)
CL-USER> (all-processes)
and in the REPL gives
(#<PROCESS repl-thread(12) [Active] #x187A186E> #<PROCESS auto-flush-thread(11) [Sleep] #x187A1C9E> #<PROCESS swank-indentation-cache-thread(6) [Semaphore timed wait] #x186C128E> #<PROCESS reader-thread(5) [Active] #x186C164E> #<PROCESS control-thread(4) [Semaphore timed wait] #x186BE3BE> #<PROCESS Swank Sentinel(2) [Semaphore timed wait] #x186BD0D6> #<TTY-LISTENER listener(1) [Active] #x183577B6> #<PROCESS Initial(0) [Sleep] #x1805FCCE>)
so it seems #<PROCESS worker(188) [Active] #x18BF99CE> is the thread that is doing the compilation. Of course, there still remains the question of why these variables are local to a thread, and also why SBCL does not behave the same way.
I can see that with CCL and some older (which I use) SLIME, too. Haven't tried it with a newer SLIME.
It does not happen with SBCL or LispWorks.
*read-default-float-format* is one of the I/O variables of Common Lisp. Something like WITH-STANDARD-IO-SYNTAX binds them to the standard values and, on exit, restores the previous values. So I suspect that CCL's SLIME code has such a binding in effect. This is confirmed by setting other I/O variables like *read-base* - they are also bound.
CCL, Emacs and SLIME has some layers of code which makes it slightly complex to debug this.
Emacs runs the ELISP code of SLIME and talks to SWANK.
SWANK is the backend of SLIME and is Common Lisp code running in CCL.
SWANK has portable code and some CCL-specific code.
On the Emacs side the SLIME/ELISP function SLIME-COMPILE-AND-LOAD-FILE is used.
On the SWANK side the Common Lisp function swank:compile-file-for-emacs gets called.
Later SWANK:LOAD-FILE gets called.
I don't see where the I/O variables are bound - maybe it is in the CCL networking code?
This seems to be the answer:
If CCL has thread-local I/O variables, then the compilation happens in another thread and won't change the I/O bindings in the REPL thread and also not in any other thread.
This is a difference between various CL implementations. Since the CL standard does not specify threads or processes, it is also not specified if special bindings are global or have a thread-local binding by default...
If you think about it, protecting a thread's I/O variables against changes from other threads makes sense...
So, the correct way to deal with it should be to make sure in each thread independently that the right I/O variable values are in effect.
Let me expand a bit why things are like they are.
Typically a Common Lisp implementation can run more than one thread. Some implementations also allow concurrent threads running at the same time on different cores. These threads can do very different things: one could run a REPL, another one could answer an HTTP request, one could load data from the disk and another one could read the contents of an Email. Lisp in this case runs several different tasks inside one Lisp system.
Lisp has several variables which determine the behavior of I/O operations. For example which format floats are when read or which base integer numbers are in when read. The letter is done by `read-base.
Now imagine that the above disk reading thread has set its *read-base* to 16 for some purpose. Now you change the global in another thread to 8 and then suddenly all other threads have base 8. The consequence: the disk reading thread will suddenly see *read-base* 8 instead of 16 and work differently.
So it makes sense to prevent this in some way. The simplest is that in each thread the running code has their own bindings for the I/O values and then changing the *read-base* won't have effects on other threads. These bindings are usually introduced by LET or a function call. Typically the code would be responsible to bind the variables.
Another way to prevent it is to give each thread a number of initial bindings, which should for example include the I/O bindings. CCL does that. LispWorks for example does that, too. But not for the I/O variables.
Now each Lisp might give you a non-portable way to change the thread-local top binding (CCL has that, too - for example (setf ccl:symbol-value-in-process) ). Still it would not mean that it might change the binding in effect in the REPL. Since the REPL itself is a piece of Lisp code, running in a thread and it could have been setting up its own bindings.
In CCL you can also set the global static binding: (CCL::%SET-SYM-GLOBAL-VALUE sym value). But if you use such functionality you are probably doing something wrong or you have a good reason.
Some background for CCL: http://clozure.com/pipermail/openmcl-devel/2011-June/012882.html
Lore
Don't try to change global bindings from one thread visible for other threads.
Shield your code from changes in other threads by binding the crucial variables to the correct values.
Write a function which sets up the variable values to some values in a controllable fashion.
Related
I am trying to get live-coding to work in lisp. i have the file t.cl which contains only this line: (loop(write(- 2 1))). Now, when i run the file in bash with sbcl --load t.cl --eval '(quit)', it runs the line, but when I try to edit the file in another terminal and save it while it runs, nothing changes ..
Why your example fails
When running sbcl --load t.cl --eval '(quit)' in a shell, what this does is spin-up a SBCL Lisp image in a process, compile the file and run it. You then modify the file and save it to your disk. This last action is of no concern to the already running SBCL process, which has already compiled the previous file. SBCL read the file once when you asked it to, once it has the compiled instructions to run, it has no reason to look at the file again unless you explicitly ask it to.
A 'live' example with Emacs+SLIME
In order to perform 'live' changes to your running program, you must interact with the already running Lisp image. This is easily doable with Emacs+Slime. You can, for example, have a loop like so:
(defun foo (x) (+ x 3))
(dotimes (it 20)
(format t "~A~%" (foo it))
(sleep 1))
and then recompile foo during execution within the REPL with a new definition:
(defun foo (x) (+ x 100))
Another thread will be used to recompile the function. The new function will be used for future calls as soon as its compilation is finished.
The output in the REPL will look like:
3
4
5
CL-USER> (defun foo (x) (+ x 100))
WARNING: redefining COMMON-LISP-USER::FOO in DEFUN
FOO
103
104
105
...
This would also work with the new definition of foo being compiled from another file as opposed to being entered directly in the REPL.
Working from the system shell
While you can already use the example above for development purposes, you might want to interact with a running SBCL Lisp image from the shell. I am not aware of how to do that. For your exact example, you want to get SBCL to reload eventual files that you have modified. A brief look at the SBCL manual doesn't seem to provide ways to pipe lisp code to an already running SBCL process.
I am slowly getting closer to be able to read and write to/from named pipes of a background process through SBCL. What I do is kick off the program I am trying to read/write from/to:
todd#ubuntu:~/CoreNLP$ cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text > ./spout &
[1] 24616
So that all works out fine, so I kick off SBCL and do this:
(defparameter from-corenlp (open "./spout"))
Which also works out fine, but declaring the stream causes SBCL to spill the stream onto the screen (which is all the startup information from the background process). It does not wait until I read from the stream. Is that how things are supposed to work?
The solution, as I posted it to the stanford parser mailing list (stack overflow reformatted a lot of it to something weird, but you get the idea):
It took quite a while, but I finally figured out embedding (for the most part) the CoreNLP program (while in interactive mode) in SBCL Lisp.
First of all, forget using (sb-ext:run-program ...). This combination of spawning Java with a quoted argument (like the asterisk) no matter how well escaped, simply makes the spawned program crash.
Inferior shell seems to kick off the parser but it is only good for a one-off parse, even in the interactive mode. Perhaps I could have done better, but inferior shell needs to be installed and it is poorly documented.
The initial attempted solution of using Unix named pipes ends up being the final one, but it took a bit of work, first with buffering, then with the order of operations, and finally understanding some nuances about the parser program.
First, turning off buffering completely when running the program is important, so running it looks like this:
stdbuf --i=0 --o=0 --e=0 cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text > ./spout &
That is supposed to be running the parser in the background accepting input from spin and sending its output to spout. But if you look at the process table in Linux, you will not see it running. It is still waiting for something to pull from the output pipe before it can even run.
So, we run SBCL and start a stream pulling from the parser´s pipe:
(defparameter *from-corenlp* (open "./spout"))
NOW the parser starts running. Here, oddly, it also starts dumping output to the screen, not to the pipe! That is because all of this banner stuff when the parser starts and stops (and apparently even the NLP> prompt) is sent to stderr, not stdout. This is actually a good thing.
So then we declare the stream from Lisp to the parser:
(defparameter *to-corenlp* (open "./spin" :direction :output :if-exists :append))
Then we send some text for the parser to parse:
(write-line "This is the first test." *to-corenlp*)
I ran into a problem here a few times, even. Remember that Lisp has its own buffer so you have to clear out the stream every time:
(finish-output *to-corenlp*)
You then can run this line below a whole bunch of times to verify you obtain the exact same behavior you would have gotten from an interactive session of the parser:
(format t "~a~%" (read-line *from-corenlp*))
Which, if you are a good boy scout, should not only be true, but you can carry on with your interactive slave parser session for as long as you like:
(write-line "This is the second test." *to-corenlp*)
(finish-output *to-corenlp*)
Isn´t that great? And notice I pulled all of that off being terrible at Unix, terrible at Lisp and being a terrible boy scout!
Now so can you!
I'm trying to read a massive sexp from file into memory, and it seems to be working out fine for smaller inputs, but on more deeply nested ones sbcl conks out with stack exhaustion. There seems to be a hard recursion limit (at 1000 functions deep) that sbcl simply cannot surpass (strangely, even when its stack size is increased). Example (code is here): make check-c works, but make check-cpp exhausts the stack as below:
INFO: Control stack guard page unprotected
Control stack guard page temporarily disabled: proceed with caution
Unhandled SB-KERNEL::CONTROL-STACK-EXHAUSTED in thread #<SB-THREAD:THREAD
"main thread" RUNNING
{10034E6DE3}>:
Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.
PROCEED WITH CAUTION.
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10034E6DE3}>
0: ((LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX))
1: (SB-IMPL::CALL-WITH-SANE-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {100FC9006B}>)
2: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {100FC9003B}>)
...
Why am I using recursion, then? Actually, I'm not, but unfortunately the builtin (read) uses recursion, and that's where the stack overflow is occurring. The other option (which I've started working on) is to write an iterative version of read which relies upon the more limited syntax that I'm feeding into it from a separate program to avoid the complexity of re-implementing read (my (currently broken) attempts at that are in the lisp branch of the above repository).
However, I'd prefer a more canonical solution. Are there alternatives to the builtin read that can parse deeply nested structures by avoiding recursion?
EDIT: This appears to be an insurmountable issue with sbcl itself, not the input data. For a quick example, try running:
(for i in $(seq 1 2000); do
echo -n "("
done; echo -n "2"; for i in $(seq 1 2000); do
echo -n ")"
done; echo) > file
And then in sbcl:
(with-open-file (file "file" :direction :input) (read file))
The same failure occurs.
EDIT: Asked around on #sbcl, and apparently the control stack size really applies only to new threads, and that the stack size for the main thread is affected by a lot of other factors as well. So I tried putting the read in a separate thread. Still didn't work. Checkout this repo and run make check if you're interested.
I don't know what you did (because you didn't show it exactly), but when I start sbcl as follows your example works fine for me:
sbcl --control-stack-size 100
Of course I recommended GNU CLISP and Embedded Common Lisp as they also work A-OK for your example.
I'll add a reference to this answer for future readers: https://stackoverflow.com/a/9002973/816536
I'll also mention that compiling the code with appropriate optimization options may be necessary in many CL implementations to benefit from tail-call optimization.
Following this question: Strange symbols in filespec when calling load I tried my luck with pathnames, but, as you see, failed. Below is an example of the error, which I cannot explain:
This code does not work:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test #P"digit-recognition:digit-7.png")
(process-image-raw test)))
Neither does this:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test "digit-recognition:digit-7.png")
(process-image-raw test)))
But this code does:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test (translate-logical-pathname "digit-recognition:digit-7.png"))
(process-image-raw test)))
And so does this:
(defun test-process-imgae-raw ()
(cl-gd:with-image-from-file
(test (translate-logical-pathname #P"digit-recognition:digit-7.png"))
(process-image-raw test)))
Here's the "translator":
(setf (logical-pathname-translations "DIGIT-RECOGNITION")
`(("**;*.*" "/home/wvxvw/Projects/digit-recognition/**/*.*")))
And here's the error I'm getting:
Pathname components from SOURCE and FROM args to TRANSLATE-PATHNAME
did not match:
:NEWEST NIL
[Condition of type SIMPLE-ERROR]
Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT] Abort thread (#<THREAD "repl-thread" RUNNING {1003800113}>)
Backtrace:
0: (SB-IMPL::DIDNT-MATCH-ERROR :NEWEST NIL)
1: (SB-IMPL::TRANSLATE-COMPONENT :NEWEST NIL :NEWEST T)
2: (TRANSLATE-PATHNAME #P"DIGIT-RECOGNITION:DIGIT-7.PNG.NEWEST" #P"DIGIT-RECOGNITION:**;*.*" #P"/home/wvxvw/Projects/digit-recognition/**/*.*")
3: (TRANSLATE-LOGICAL-PATHNAME #P"DIGIT-RECOGNITION:DIGIT-7.PNG.NEWEST")
4: (SB-IMPL::QUERY-FILE-SYSTEM #P"DIGIT-RECOGNITION:DIGIT-7.PNG" :TRUENAME NIL)
5: (PROBE-FILE #P"DIGIT-RECOGNITION:DIGIT-7.PNG")
6: (CREATE-IMAGE-FROM-FILE #<unavailable argument> NIL)
7: (TEST-PROCESS-IMGAE-RAW)
I'm trying to read the Hyperspec section on translate-pathname, but I can make absolutely no sense of what it says, neither from the examples it shows. Let alone it, I can't even understand how there can possibly be an error if you transform a string by whatever rules you put in place, so far it's only one way transformation...
I'm trying to read SBCL sources for this function, but they are really lengthy, and trying to figure out the problem this way is taking huge amounts of time.
tl;dr How is it even possible that translate-logical-pathname called from user's code will produce something different to what is produced from that function if called from system code? This is not only non-portable, this is just outright broken.
EDIT:
Adding one more asterisk to the pattern on the left side, but not on the right solved this. But the purpose or logic of why is this necessary is beyond me.
I.e.
(setf (logical-pathname-translations "DIGIT-RECOGNITION")
`(("**;*.*.*" "/home/wvxvw/Projects/digit-recognition/**/*.*")))
This allows pathnames like digit-recognition:foo.bar.newest to succeed, just like digit-recognition:foo.bar but why is that asterisk a requirement flies beyond me. Also, why is the system function feels entitled to change the pathname to something else of what it was given?.. But just not to get you confused, with-image-from-file will only work with the path already expanded by translate-logical-pathname, it won't work otherwise.
EDIT2:
OK, it seems like this is the problem with cl-gd, instead of trying to expand the file name, it takes it literally. This code taken from create-image-from-file probably best answers my question:
(when (pathnamep file-name)
(setq file-name
#+:cmu (ext:unix-namestring file-name)
#-:cmu (namestring file-name)))
(with-foreign-object (err :int)
(with-cstring (c-file-name file-name)
(let ((image (ecase %type
((:jpg :jpeg)
(gd-image-create-from-jpeg-file c-file-name err))
I.e. instead of doing (namestring file-name) it has to do (namestring (trnaslate-logical-pathname file-name)). Duh...
Another way is to use TRUENAME, which returns the real file name. Normally this would not make a difference.
Image a file system with file versions (like the file systems of VMS, ...). If you have a logical pathname foo:bar;baz.png.newest, then it might translate to, say, /myfiles/images/baz.png~newest (again, just assume that it has version numbers). This still is not a real physical file. If such a Lisp system tries to open the file, it has to look into the file system to actually determine the newest file. That might be /myfiles/images/baz.png~42.
So, if you want to pass real physical filenames to external tools (like a C library), it might not be sufficient to expand the logical pathname, but it might be necessary to compute the truename - the real physical file.
The ability to deal with file versions comes from a time when file versions where quite common (see Versioning file system) with operating systems like ITS, VMS or the various Lisp Machine operating systems.
The main practical problem for this is that there is no common test suite for pathname operations for the various CL implementations and thus implementations differ in a lot of subtle details (especially when you need to deal with different file systems from different operating systems). Plus real file systems have complications - for example file names in Mac OS X use a special unicode encoding when dealing with Umlauts.
Quite often I find myself with bunch of R processes running in ESS buffers. There's a convenient Lisp function ess-request-a-process that asks for R process, and brings it to front. The only downside is that it somehow defaults to S, so each time I'm about to make a switch, I have to type R, ad nauseam.
I tried customising the ess-language variable, but even if I set value to "R", i.e. 4 for current session, or even if I save settings for future session, as soon as I type C-c C-k, automagically S appears once again. It's very annoying, and I really don't want to end up with C-x C-b and then C-s for desired R session! =)
I even tried setting (setq-default ess-language "R") in .emacs, but with no luck...
BTW, I'm running Emacs v. 23.1.1 on Linux Mint and Emacs v. 23.2 on Arch Linux, with ESS v. 5.12. If that's relevant, I run Emacs from terminal with -nw argument. Here's my .emacs:
;; start server
(server-start)
;; load ESS
(require 'ess-site)
(require 'ess-rutils)
;; set HTML help as default
(setq inferior-ess-r-help-command "help(\"%s\", help_type = \"html\")\n")
(custom-set-variables
;; custom-set-variables was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won't work right.
'(ess-help-kill-bogus-buffers t)
'(ess-rutils-keys nil)
'(show-paren-mode t))
(custom-set-faces
;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won't work right.
)
(put 'upcase-region 'disabled nil)
So... how to set R once and for all? (I don't use S/S+/SAS)
I did not know about this function so far. C-c C-k is bound to ess-force-buffer-current in ESS buffers.
[edit: C-c C-k is indeed bound to ess-request-a-process in iESS, in ESS it's ess-force-buffer-current]
In any case the variable you have to customize is ess-dialect
(setq-default ess-dialect "R")
It's buffer-local variable and some other stuff in ess-mode-hook might set it a different value.
Check it in each buffer with C-h v ess-dialect
Additionally, if you already running several processes then ess-switch-process (C-c C-s) might be the right way to go.
[edit: it will not jump to a process but just reset the associated process of the current ESS buffer]
[edit: After dwelling deeper on the issue it turned out that ess-request-a-process uses ess-language variable were the ess-dialect seems to be more appropriate. The problem is that each time an ess-inferior process starts it resets the global value of ess-language. This is why setting it in your case didn't work.
Here is a quick fix:
(defun ess-set-language ()
(setq-default ess-language "R")
(setq ess-language "R")
)
(add-hook 'ess-post-run-hook 'ess-set-language t)
]