When I run the following code:
(defun countdown (n)
(if (>= n 0)
(cons n (countdown (- n 1)))))
(countdown 100000)
I get the following message :
INFO: Control stack guard page unprotected
Control stack guard page temporarily disabled: proceed with caution
debugger invoked on a SB-KERNEL::CONTROL-STACK-EXHAUSTED in thread
#<THREAD "main thread" RUNNING {1002B03653}>:
Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.
PROCEED WITH CAUTION.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-KERNEL::CONTROL-STACK-EXHAUSTED-ERROR)
How do I increase the stack space? I do not want to change the code above; I supplied it purely to illustrate stack exhaustion.
Calling sbcl with the --help option on the command line gives some useful common options, including an option for how to increase the stack space. You want to to use the --control-stack-size argument.
$ sbcl --help
Usage: sbcl [runtime-options] [toplevel-options] [user-options]
Common runtime options:
--help Print this message and exit.
--version Print version information and exit.
--core <filename> Use the specified core file instead of the default.
--dynamic-space-size <MiB> Size of reserved dynamic space in megabytes.
--control-stack-size <MiB> Size of reserved control stack in megabytes.
...
By specifying a bigger value there, we can run your code:
$ sbcl --control-stack-size 100000
* (defun countdown (n)
(if (>= n 0)
(cons n (countdown (- n 1)))))
;=> (100000 … 6 5 4 3 2 1 0)
Not exactly in time for the question at hand, but for anyone also wondering about the --control-stack-size:
From the SBCL manual section "3.3.1 Runtime Options" the default value for the --control-stack-size is 2 megabytes.
So for a list containing 100k 64bit-integers 7 megabyte sbcl --control-stack-size 7 should be sufficient!
As a reference:
CL-USER> (type-of 42)
(INTEGER 0 4611686018427387903)
CL-USER> (/ (* 64 100e3) (expt 2 20))
6.1035156
Related
I was hoping to experiment with cl-async to run a series of external programs with a large combinations of command line arguments. However, I can't figure out how to read the stdout of the processes launched with as:spawn.
I would typically use uiop which makes it easy to capture the process output:
(let ((p (uiop:launch-program ... :output :stream)))
(do-something-else-until-p-is-done)
(format t "~a~%" (read-line (uiop:process-info-output p))))
I've tried both :output :pipe and :output :stream options to as:spawn and executing (as:process-output process-object) in my exit-callback shows the appropriate pipe or async-stream objects but I can't figure out how to read from them.
Can anyone with experience with this library tell how to accomplish this?
So you go to your repl and type:
CL-USER> (documentation 'as:spawn 'function)
And you read whatever comes out (or put your point on the symbol and hit C-c C-d f). If you read it you’ll see that the format for the :input, etc arguments is either :pipe, (:pipe args...), :stream, or (:stream args...) (or some other options). And that :stream behaves similarly to :pipe but gives output of a different type and that for details of args one should look at PIPE-CONNECT so you go and look up the documentation for that. Well it tells you what the options are but it isn’t very useful. What’s the documentation/description of PIPE or STREAM? Well it turns out that pipe is a class and a subclass of STREAMISH. What about PROCESS that’s a class too and it has slots (and accessors) for things like PROCESS-OUTPUT. So what is a good plan for how to figure out what to do next? Here’s a suggestion:
Spawn a long running process (like cat foo.txt -) with :output :stream :input :pipe say
Inspect the result (C-c C-v TAB)
Hopefully it’s an instance of PROCESS. What is it’s output? Inspect that
Hopefully the output is a Gray stream (ASYNC-STREAM). Get it into your repl and see what happens if you try to read from it?
And what about the input? See what type that has and what you can do with it
The above is all speculation. I’ve not tried running any of this but you should. Alternatively go look at the source code for the library. It’s already on your computer and if you can’t find it it’s on GitHub. There are only about half a dozen source files and they’re all small. Just read them and see what you can learn. Or go to the symbol you want to know about and hit M-. to jump straight to its definition. Then read the code. Then see if you can figure out what to do.
I found the answer in the test suite. The output stream can only be processed asynchronously via a read call-back. The following is simple example for posterity
(as:start-event-loop
(lambda ()
(let ((bytes (make-array 0 :element-type '(unsigned-byte 8))))
(as:spawn "./test.sh" '()
:exit-cb (lambda (proc exit-status term-signal)
(declare (ignore proc exit-status term-signal))
(format t "proc output:~%~a"
(babel:octets-to-string bytes)))
:output (list :stream
:read-cb (lambda (pipe stream)
(declare (ignore pipe))
(let ((buf (make-array 128 :element-type '(unsigned-byte 8))))
(loop for n = (read-sequence buf stream)
while (plusp n) do
(setf bytes
(concatenate '(vector (unsigned-byte 8))
bytes
(subseq buf 0 n)))))))))))
with
$ cat test.sh
#!/bin/bash
sleep_time=$((1+$RANDOM%10))
echo "Process $$ will sleep for $sleep_time"
sleep $sleep_time
echo "Process $$ exiting"
yields the expected output
Over a remote ssh connection, I'm trying to cross-compile sbcl using clisp. The steps I've followed thus far are such:
I downloaded most recent sbcl source, (at this time sbcl-1.3.7), uncompressed it, and entered its source directory.
Then to build it:
root#remotehost:/sbcl-1.3.7# screen
root#remotehost:/sbcl-1.3.7# sh make.sh --prefix=/usr --dynamic-space-size=2Gb --xc-host='clisp -q'
root#remotehost:/sbcl-1.3.7# Ctrl-A Ctrl-D
[detached from 4486.pts-1.remotehost]r/fun-info-funs.fas
root#remotehost:/sbcl-1.3.7#
Over a second remote ssh connection to the same box, top reports cpu usage at 6%
nproc says I have 16 cores (this is google compute engine--no way could I afford something with 16 cores :)
MAKEFLAGS are set in my environment to -j16, but I guess clisp isn't aware of that. How can I get this build to make use of all 16 cores?
I reccomend you to use a parallism library, I really like lparallel library
It has pretty utilities to parallelize your code ammong all processors in your machine. This is an example for my macbook pro (4 cores) using SBCL. There are a great series of common lisp concurrence and parallelism here
But let's create a sample example using lparallel cognates, note that this example is not a well exercise of parallelism is only to show the power of leparallel and how easy is to use.
Let's consider a fibonacci tail recursive function from cliki:
(defun fib (n) "Tail-recursive computation of the nth element of the
Fibonacci sequence" (check-type n (integer 0 *)) (labels ((fib-aux
(n f1 f2)
(if (zerop n) f1
(fib-aux (1- n) f2 (+ f1 f2)))))
(fib-aux n 0 1)))
This will be the sample high computation cost algorithm. let's use it:
CL-USER> (time (progn (fib 1000000) nil))
Evaluation took:
17.833 seconds of real time
18.261164 seconds of total run time (16.154088 user, 2.107076 system)
[ Run times consist of 3.827 seconds GC time, and 14.435 seconds non-GC time. ]
102.40% CPU
53,379,077,025 processor cycles
43,367,543,984 bytes consed
NIL
this is the calculation for the 1000000th term of the fibonacci series on my computer.
Let's for example calculate a list of fibonnaci numbers using mapcar:
CL-USER> (time (progn (mapcar #'fib '(1000000 1000001 1000002 1000003)) nil))
Evaluation took:
71.455 seconds of real time
73.196391 seconds of total run time (64.662685 user, 8.533706 system)
[ Run times consist of 15.573 seconds GC time, and 57.624 seconds non-GC time. ]
102.44% CPU
213,883,959,679 processor cycles
173,470,577,888 bytes consed
NIL
Lparallell has cognates:
They return the same results as their CL counterparts except in cases
where parallelism must play a role. For instance premove behaves
essentially like its CL version, but por is slightly different. or
returns the result of the first form that evaluates to something
non-nil, while por may return the result of any such
non-nil-evaluating form.
first load lparallel:
CL-USER> (ql:quickload :lparallel)
To load "lparallel":
Load 1 ASDF system:
lparallel
; Loading "lparallel"
(:LPARALLEL)
So in our case, the only thing that you have to do is initially a kernel with the number of cores you have available:
CL-USER> (setf lparallel:*kernel* (lparallel:make-kernel 4 :name "fibonacci-kernel"))
#<LPARALLEL.KERNEL:KERNEL :NAME "fibonacci-kernel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1004E1E693}>
and then launch the cognates from the pmap family:
CL-USER> (time (progn (lparallel:pmapcar #'fib '(1000000 1000001 1000002 1000003)) nil))
Evaluation took:
58.016 seconds of real time
141.968723 seconds of total run time (107.336060 user, 34.632663 system)
[ Run times consist of 14.880 seconds GC time, and 127.089 seconds non-GC time. ]
244.71% CPU
173,655,268,162 processor cycles
172,916,698,640 bytes consed
NIL
You can see how easy is to parallelize this task, lparallel has a lot of resources that you can explore:
I also add a capture of the cpu usage from the first mapcar and pmapcar in my Mac:
SBCL cross compiling (aka bootstrapping) is done in 100% vanilla CL and there are nothing in the standard about threading or multiple processes.
Using a supplied SBCL binary for your machine might not use threading either even though I know SBCL does have thread support in the language.
You only need to do this if you have altered the source of SBCL as the latest supported version is usally precompiled already. I haven't compiled SBCL myself but i doubt it takes longer than my compilations of linux in the 90s that used less time than my nightly sleep.
I wrote some core.async code in Clojure and when I ran it it consumed all available memory and failed with an error. It appears that using mapcat in a core.async pipeline breaks back pressure. (Which is unfortunate for reasons beyond the scope of this question.)
Here is some code that demonstrates the problem by counting :xs in and out of a mapcating transducer:
(ns mapcat.core
(:require [clojure.core.async :as async]))
(defn test-backpressure [n length]
(let [message (repeat length :x)
input (async/chan)
transform (async/chan 1 (mapcat seq))
output (async/chan)
sent (atom 0)]
(async/pipe input transform)
(async/pipe transform output)
(async/go
(dotimes [_ n]
(async/>! input message)
(swap! sent inc))
(async/close! input))
(async/go-loop [x 0]
(when (= 0 (mod x (/ (* n length) 10)))
(println "in:" (* #sent length) "out:" x))
(when-let [_ (async/<! output)]
(recur (inc x))))))
=> (test-backpressure 1000 10)
in: 10 out: 0
in: 2680 out: 1000
in: 7410 out: 2000
in: 10000 out: 3000 ; Where are the other 7000 characters?
in: 10000 out: 4000
in: 10000 out: 5000
in: 10000 out: 6000
in: 10000 out: 7000
in: 10000 out: 8000
in: 10000 out: 9000
in: 10000 out: 10000
The producer races far ahead of the consumer.
It appears that I'm not the first person to discover this.
But the explanation given here doesn't quite seem to cover it. (Although it does provide an adequate workaround.)
Conceptually, I would expect the producer to be ahead, but only by the length of the few messages that might be buffered in the channels.
My question is, where are all the other messages? By the fourth line of output 7000 :xs are unaccounted for.
UPDATE 2020-01-14: The memory leak is now fixed.
There are two possible interpretations of the question "Where is the memory leak?"
Firstly, where is the data held? The answer seems to be in the channel buffer immediately downstream of the expanding transform.
Channels by default use a FixedBuffer (clojure.core.async.impl.buffers/FixedBuffer) which can tell if it is full but does not object to being overfull.
Secondly, which passage of code causes the buffer to be overfull? This (correct me if I am wrong) appears to be in the take! method of ManyToManyChannel (clojure.core.async.impl.channels/ManyToManyChannel) where the first call to add! on the buffer occurs before any calls to full? have taken place.
It seems that take! assumes that it can add at least one item to the buffer for every item it removes. In the case of long running expanding transducers such as mapcat this is not always a safe assumption.
By changing this line to (when (and (.hasNext iter) (not (impl/full? buf))) in a local copy of core.async I can make the code in the question behave as expected. (N.B. My understanding of core.async is insufficient for me to guarantee that this is a robust solution for your use case.)
UPDATE 2016-09-17: there is now an issue for this: http://dev.clojure.org/jira/browse/ASYNC-178
UPDATE 2020-01-14: this is now fixed as of: https://clojure.atlassian.net/browse/ASYNC-210 (although the earlier ticket was closed as 'Declined')
I'm curious to learn the reason for this since set seems unique. For instance:
(set 'nm 3) ;; set evaluates its first argument, a symbol, has to be quoted
nm ;; ==> evaluates to 3
(set 'nm 'nn) ;; assigns nn to the value cell of nm
nm ;; ==> evaluates to nn
nn ;; ==> ERROR. no value
(set nm 3) ;; since nm evaluates to nn ...
nm ;; evaluates to nn
nn ;; evaluates to 3
To achieve similar behavior, I've only been able to use setf:
(setq tu 'ty) ;;
(symbol-value 'tu) ;; returns ty
(setq (symbol-value 'tu) 5) ;; ERROR. setq expects a symbol
(setf (symbol-value tu) 5) ;; has to be unquoted to access the value cell
tu ;; ==> evaluates to ty
ty ;; ==> evaluates to 3
In other programming languages the reason(s) for demotion are pretty clear: inefficient, bug prone, or insecure come to mind. I wonder what the criteria for deprecation for set was at the time. All I've been able to glean from the web is this, which is laughable. Thanks.
The main reason set is deprecated is that its use can lead to errors when it is used on bound variables (e.g., in functions):
(set 'a 10)
==> 10
a
==> 10
(let ((a 1)) ; new lexical binding
(set 'a 15) ; modification of the value slot of the global symbol, not the lexical variable
a) ; access the lexical variable
==> 1 ; nope, not 15!
a
==> 15
set is a legacy function from the times when Lisp was "the language of symbols and lists". Lisp has matured since then; direct operations on symbol slots are relatively rare, and there is no reason to use set instead of the more explicit (setf symbol-value).
In your example:
(set nm 'nn) ;; assigns nn to the value cell of nm
nm ;; ==> evaluates to nn
This is completely wrong and the main reason why it's deprecated. It's the symbol you get when evaluating nm that is bound to nn. Unfortunately in your example that is the number 3 and it will signal an error at run time since you cannot use numbers as variables. If you were to write (setq 3 'nn) the error can be seen at compile time.
With set there is an additional reason. It's very difficult to compile when you don't know what symbol is to be bound and the compiler cannot optimize it.
Scheme didn't have automatic quoting either in it's first version and they didn't even have a quoted macro like setq. Instead it stated that set' would suffice. It's obvious that it didn't since Scheme no longer have set but only define.
I personally disagree that it should be removed (deprecation results in removal eventually) but like eval it should be avoided and only used as a last resort.
I want to use a step function to see how it went to the expected output, but it's not working.
Like this simple example:
(STEP (IF (ODDP 3) 'YES 'NO))
but nothing happens.
Is there any optimization that makes me can't see the trace steps ???
How to turn it off?
Thanks!
It's because CL:STEP is not implemented on CCL that I implemented com.informatimago.common-lisp.lisp.stepper.
You can get it with quicklisp.
The documentation is at: https://gitorious.org/com-informatimago/com-informatimago/source/2b53ae44e8fa4d040fafcf4d93976500a8e464dc:common-lisp/lisp/stepper-packages.lisp#L146
STEP is not supported in CCL.
Solution for TRACE:
When a (globally named) function FOO is defined with DEFUN, the compiler is allowed to assume that functional references to that function name refer to the function being defined (unless they're lexically shadowed); it can therefore skip the implicit SYMBOL-FUNCTION ("call whatever is in the function cell of FOO") on a self-call (a call to FOO from within FOO.) This saves an instruction or two on those calls, but (since TRACE works by changing what SYMBOL-FUNCTION returns) those inlined self-calls can't be traced.
However, the compiler can't do this (can't even assume that something defined by DEFUN won't be redefined later) if the function name is declared NOTINLINE at the point of the self call:
example:
? (defun fact (x acc)
(declare (notinline fact))
(if (= x 0)
acc
(fact (- x 1) (* x acc))))
? (trace fact)
NIL
? (fact 3 1)
0> Calling (FACT 3 1)
1> Calling (FACT 2 3)
2> Calling (FACT 1 6)
3> Calling (FACT 0 6)
<3 FACT returned 6
<2 FACT returned 6
<1 FACT returned 6
<0 FACT returned 6
? (step (fact 3 1))
0> Calling (FACT 3 1)
1> Calling (FACT 2 3)
2> Calling (FACT 1 6)
3> Calling (FACT 0 6)
<3 FACT returned 6
<2 FACT returned 6
<1 FACT returned 6
<0 FACT returned 6
That's the way to say to the compiler, "as a matter of policy, I'd rather have the ability to trace functions which call themselves and are defined with DEFUN and don't care about saving a few cycles per self-call".
from: DebugWithOpenMCL
or Evaluate the following form:
(DECLAIM (OPTIMIZE (DEBUG 3)))
before defining any function to be traced.
I don't think Clozure CL supports stepping. IIRC nobody has funded this feature yet. It would need some work, since Clozure CL lacks an interpreter (where stepping could be supported relatively painless).
Other implementations support stepping.
This library is not distributed on quicklisp anymore however it builds successfully when installed as a local quicklisp project.
use cl-stepper:step instead of cl-user:step , 'cause Clozure CL don't support it.
if you have already installed quicklisp, try to install it : like this.
(ql:quickload "com.informatimago.common-lisp.lisp.stepper")
(defpackage :test (:use :cl-stepper))
(in-package :test)
(def bar (hoge)
;; define some function
)
(step (bar 3))