I'm calling a funny API that returns a byte array, but I want a text stream. Is there an easy way to get a text stream from a byte array? For now I just threw together:
(defun bytearray-to-string (bytes)
(let ((str (make-string (length bytes))))
(loop for byte across bytes
for i from 0
do (setf (aref str i) (code-char byte)))
str))
and then wrap the result in with-input-from-string, but that can't be the best way. (Plus, it's horribly inefficient.)
In this case, I know it's always ASCII, so interpreting it as either ASCII or UTF-8 would be fine. I'm using Unicode-aware SBCL, but I'd prefer a portable (even ASCII-only) solution to a SBCL-Unicode-specific one.
FLEXI-STREAMS (http://weitz.de/flexi-streams/) has portable conversion function
(flexi-streams:octets-to-string #(72 101 108 108 111) :external-format :utf-8)
=>
"Hello"
Or, if you want a stream:
(flexi-streams:make-flexi-stream
(flexi-streams:make-in-memory-input-stream
#(72 101 108 108 111))
:external-format :utf-8)
will return a stream that reads the text from byte-vector
There are two portable libraries for this conversion:
flexi-streams, already mentioned in another answer.
This library is older and has more features, in particular the extensible streams.
Babel, a library specificially for character encoding and decoding
The main advantage of Babel over flexi-streams is speed.
For best performance, use Babel if it has the features you need, and fall back to flexi-streams otherwise. Below a (slighly unscientific) microbenchmark illustrating the speed difference.
For this test case, Babel is 337 times faster and needs 200 times less memory.
(asdf:operate 'asdf:load-op :flexi-streams)
(asdf:operate 'asdf:load-op :babel)
(defun flexi-streams-test (bytes n)
(loop
repeat n
collect (flexi-streams:octets-to-string bytes :external-format :utf-8)))
(defun babel-test (bytes n)
(loop
repeat n
collect (babel:octets-to-string bytes :encoding :utf-8)))
(defun test (&optional (data #(72 101 108 108 111))
(n 10000))
(let* ((ub8-vector (coerce data '(simple-array (unsigned-byte 8) (*))))
(result1 (time (flexi-streams-test ub8-vector n)))
(result2 (time (babel-test ub8-vector n))))
(assert (equal result1 result2))))
#|
CL-USER> (test)
Evaluation took:
1.348 seconds of real time
1.328083 seconds of user run time
0.020002 seconds of system run time
[Run times include 0.12 seconds GC run time.]
0 calls to %EVAL
0 page faults and
126,402,160 bytes consed.
Evaluation took:
0.004 seconds of real time
0.004 seconds of user run time
0.0 seconds of system run time
0 calls to %EVAL
0 page faults and
635,232 bytes consed.
|#
If you don't have to worry about UTF-8 encoding (that, essentially, means "just plain ASCII"), you may be able to use MAP:
(map 'string #'code-char #(72 101 108 108 111))
I say go with the proposed flexistream or babel solutions.
But just for completeness and the benefit of future googlers arriving at this page I want to mention sbcl's own sb-ext:octets-to-string:
SB-EXT:OCTETS-TO-STRING is an external symbol in #<PACKAGE "SB-EXT">.
Function: #<FUNCTION SB-EXT:OCTETS-TO-STRING>
Its associated name (as in FUNCTION-LAMBDA-EXPRESSION) is
SB-EXT:OCTETS-TO-STRING.
The function's arguments are: (VECTOR &KEY (EXTERNAL-FORMAT DEFAULT) (START 0)
END)
Its defined argument types are:
((VECTOR (UNSIGNED-BYTE 8)) &KEY (:EXTERNAL-FORMAT T) (:START T) (:END T))
Its result type is:
*
SBCL supports the so-called Gray Streams. These are extensible streams based on CLOS classes and generic functions. You could create a text stream subclass that gets the characters from the byte array.
Try the FORMAT function. (FORMAT NIL ...) returns the results as a string.
Related
I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk).
This function takes a pointer to a foreign memory (where chunk is placed) as an argument.
So, I make buffer for a chunk: foreign-buffer.
I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:
(let ((foreign-buffer (foreign-alloc :uchar 1024)))
(stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
(loop
for cnt = (read-sequence stream-buffer MY-STREAM)
while (> cnt 0)
do
;; copy cnt bytes from stream-buffer into foreign-buffer
;; call db-write-chunk with foreign-buffer
L in BLOB is for Large and loop may iterate many times.
Besides that, all this code may be wrapped by the external loop (bulk-insert, for example).
So, I want to minimize the count of steps in the loop(s) body.
To have this done I need:
to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:
(read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
or to be able to interpret stream-buffer as foreign memory, like this:
(db-write-chunk (mem-aptr stream-buffer :uchar 0))
Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?
Like anything else ffi, this is implementation dependent, but cffi has cffi:make-shareable-byte-vector, which is a CL (unsigned-byte 8) array which you can then use with cffi:with-pointer-to-vector-data:
(cffi:defcfun memset :pointer
(ptr :pointer)
(val :int)
(size :int))
(let ((vec (cffi:make-shareable-byte-vector 256)))
(cffi:with-pointer-to-vector-data (ptr vec)
(memset ptr 0 (length vec))))
Depending on your use, this might be preferable to static-vectors, because you don't have to remember to free it manually. On SBCL this works by pinning the vector data during with-pointer-to-vector-data.
Over a remote ssh connection, I'm trying to cross-compile sbcl using clisp. The steps I've followed thus far are such:
I downloaded most recent sbcl source, (at this time sbcl-1.3.7), uncompressed it, and entered its source directory.
Then to build it:
root#remotehost:/sbcl-1.3.7# screen
root#remotehost:/sbcl-1.3.7# sh make.sh --prefix=/usr --dynamic-space-size=2Gb --xc-host='clisp -q'
root#remotehost:/sbcl-1.3.7# Ctrl-A Ctrl-D
[detached from 4486.pts-1.remotehost]r/fun-info-funs.fas
root#remotehost:/sbcl-1.3.7#
Over a second remote ssh connection to the same box, top reports cpu usage at 6%
nproc says I have 16 cores (this is google compute engine--no way could I afford something with 16 cores :)
MAKEFLAGS are set in my environment to -j16, but I guess clisp isn't aware of that. How can I get this build to make use of all 16 cores?
I reccomend you to use a parallism library, I really like lparallel library
It has pretty utilities to parallelize your code ammong all processors in your machine. This is an example for my macbook pro (4 cores) using SBCL. There are a great series of common lisp concurrence and parallelism here
But let's create a sample example using lparallel cognates, note that this example is not a well exercise of parallelism is only to show the power of leparallel and how easy is to use.
Let's consider a fibonacci tail recursive function from cliki:
(defun fib (n) "Tail-recursive computation of the nth element of the
Fibonacci sequence" (check-type n (integer 0 *)) (labels ((fib-aux
(n f1 f2)
(if (zerop n) f1
(fib-aux (1- n) f2 (+ f1 f2)))))
(fib-aux n 0 1)))
This will be the sample high computation cost algorithm. let's use it:
CL-USER> (time (progn (fib 1000000) nil))
Evaluation took:
17.833 seconds of real time
18.261164 seconds of total run time (16.154088 user, 2.107076 system)
[ Run times consist of 3.827 seconds GC time, and 14.435 seconds non-GC time. ]
102.40% CPU
53,379,077,025 processor cycles
43,367,543,984 bytes consed
NIL
this is the calculation for the 1000000th term of the fibonacci series on my computer.
Let's for example calculate a list of fibonnaci numbers using mapcar:
CL-USER> (time (progn (mapcar #'fib '(1000000 1000001 1000002 1000003)) nil))
Evaluation took:
71.455 seconds of real time
73.196391 seconds of total run time (64.662685 user, 8.533706 system)
[ Run times consist of 15.573 seconds GC time, and 57.624 seconds non-GC time. ]
102.44% CPU
213,883,959,679 processor cycles
173,470,577,888 bytes consed
NIL
Lparallell has cognates:
They return the same results as their CL counterparts except in cases
where parallelism must play a role. For instance premove behaves
essentially like its CL version, but por is slightly different. or
returns the result of the first form that evaluates to something
non-nil, while por may return the result of any such
non-nil-evaluating form.
first load lparallel:
CL-USER> (ql:quickload :lparallel)
To load "lparallel":
Load 1 ASDF system:
lparallel
; Loading "lparallel"
(:LPARALLEL)
So in our case, the only thing that you have to do is initially a kernel with the number of cores you have available:
CL-USER> (setf lparallel:*kernel* (lparallel:make-kernel 4 :name "fibonacci-kernel"))
#<LPARALLEL.KERNEL:KERNEL :NAME "fibonacci-kernel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1004E1E693}>
and then launch the cognates from the pmap family:
CL-USER> (time (progn (lparallel:pmapcar #'fib '(1000000 1000001 1000002 1000003)) nil))
Evaluation took:
58.016 seconds of real time
141.968723 seconds of total run time (107.336060 user, 34.632663 system)
[ Run times consist of 14.880 seconds GC time, and 127.089 seconds non-GC time. ]
244.71% CPU
173,655,268,162 processor cycles
172,916,698,640 bytes consed
NIL
You can see how easy is to parallelize this task, lparallel has a lot of resources that you can explore:
I also add a capture of the cpu usage from the first mapcar and pmapcar in my Mac:
SBCL cross compiling (aka bootstrapping) is done in 100% vanilla CL and there are nothing in the standard about threading or multiple processes.
Using a supplied SBCL binary for your machine might not use threading either even though I know SBCL does have thread support in the language.
You only need to do this if you have altered the source of SBCL as the latest supported version is usally precompiled already. I haven't compiled SBCL myself but i doubt it takes longer than my compilations of linux in the 90s that used less time than my nightly sleep.
I wrote some core.async code in Clojure and when I ran it it consumed all available memory and failed with an error. It appears that using mapcat in a core.async pipeline breaks back pressure. (Which is unfortunate for reasons beyond the scope of this question.)
Here is some code that demonstrates the problem by counting :xs in and out of a mapcating transducer:
(ns mapcat.core
(:require [clojure.core.async :as async]))
(defn test-backpressure [n length]
(let [message (repeat length :x)
input (async/chan)
transform (async/chan 1 (mapcat seq))
output (async/chan)
sent (atom 0)]
(async/pipe input transform)
(async/pipe transform output)
(async/go
(dotimes [_ n]
(async/>! input message)
(swap! sent inc))
(async/close! input))
(async/go-loop [x 0]
(when (= 0 (mod x (/ (* n length) 10)))
(println "in:" (* #sent length) "out:" x))
(when-let [_ (async/<! output)]
(recur (inc x))))))
=> (test-backpressure 1000 10)
in: 10 out: 0
in: 2680 out: 1000
in: 7410 out: 2000
in: 10000 out: 3000 ; Where are the other 7000 characters?
in: 10000 out: 4000
in: 10000 out: 5000
in: 10000 out: 6000
in: 10000 out: 7000
in: 10000 out: 8000
in: 10000 out: 9000
in: 10000 out: 10000
The producer races far ahead of the consumer.
It appears that I'm not the first person to discover this.
But the explanation given here doesn't quite seem to cover it. (Although it does provide an adequate workaround.)
Conceptually, I would expect the producer to be ahead, but only by the length of the few messages that might be buffered in the channels.
My question is, where are all the other messages? By the fourth line of output 7000 :xs are unaccounted for.
UPDATE 2020-01-14: The memory leak is now fixed.
There are two possible interpretations of the question "Where is the memory leak?"
Firstly, where is the data held? The answer seems to be in the channel buffer immediately downstream of the expanding transform.
Channels by default use a FixedBuffer (clojure.core.async.impl.buffers/FixedBuffer) which can tell if it is full but does not object to being overfull.
Secondly, which passage of code causes the buffer to be overfull? This (correct me if I am wrong) appears to be in the take! method of ManyToManyChannel (clojure.core.async.impl.channels/ManyToManyChannel) where the first call to add! on the buffer occurs before any calls to full? have taken place.
It seems that take! assumes that it can add at least one item to the buffer for every item it removes. In the case of long running expanding transducers such as mapcat this is not always a safe assumption.
By changing this line to (when (and (.hasNext iter) (not (impl/full? buf))) in a local copy of core.async I can make the code in the question behave as expected. (N.B. My understanding of core.async is insufficient for me to guarantee that this is a robust solution for your use case.)
UPDATE 2016-09-17: there is now an issue for this: http://dev.clojure.org/jira/browse/ASYNC-178
UPDATE 2020-01-14: this is now fixed as of: https://clojure.atlassian.net/browse/ASYNC-210 (although the earlier ticket was closed as 'Declined')
When I run the following code:
(defun countdown (n)
(if (>= n 0)
(cons n (countdown (- n 1)))))
(countdown 100000)
I get the following message :
INFO: Control stack guard page unprotected
Control stack guard page temporarily disabled: proceed with caution
debugger invoked on a SB-KERNEL::CONTROL-STACK-EXHAUSTED in thread
#<THREAD "main thread" RUNNING {1002B03653}>:
Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.
PROCEED WITH CAUTION.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-KERNEL::CONTROL-STACK-EXHAUSTED-ERROR)
How do I increase the stack space? I do not want to change the code above; I supplied it purely to illustrate stack exhaustion.
Calling sbcl with the --help option on the command line gives some useful common options, including an option for how to increase the stack space. You want to to use the --control-stack-size argument.
$ sbcl --help
Usage: sbcl [runtime-options] [toplevel-options] [user-options]
Common runtime options:
--help Print this message and exit.
--version Print version information and exit.
--core <filename> Use the specified core file instead of the default.
--dynamic-space-size <MiB> Size of reserved dynamic space in megabytes.
--control-stack-size <MiB> Size of reserved control stack in megabytes.
...
By specifying a bigger value there, we can run your code:
$ sbcl --control-stack-size 100000
* (defun countdown (n)
(if (>= n 0)
(cons n (countdown (- n 1)))))
;=> (100000 … 6 5 4 3 2 1 0)
Not exactly in time for the question at hand, but for anyone also wondering about the --control-stack-size:
From the SBCL manual section "3.3.1 Runtime Options" the default value for the --control-stack-size is 2 megabytes.
So for a list containing 100k 64bit-integers 7 megabyte sbcl --control-stack-size 7 should be sufficient!
As a reference:
CL-USER> (type-of 42)
(INTEGER 0 4611686018427387903)
CL-USER> (/ (* 64 100e3) (expt 2 20))
6.1035156
"with-open-file" will read from the beginning of a file. If the file is VERY big how to read the last 20 lines efficiently ?
Sincerely!
This opens a file, reads the final byte, and closes the file.
(defun read-final-byte (filename)
(with-open-file (s filename
:direction :input
:if-does-not-exist :error)
(let ((len (file-length s)))
(file-position s (1- len)) ; 0-based position.
(read-char s nil)))) ; don't error if reading the end of the file.
If you want to specifically read the last n lines, you will have to read back an indeterminate number of bytes until you get n+1 newlines. In order to do this, you will either have to do block reads backwards (faster but will wind up in reading unneeded bytes), or byte-reads (slower but allows precision and a slightly more obvious algorithm).
I suspect tail has a reasonable algorithm applied for this, so it would likely be worth reading tail's source for a guideline.