Julia has a number of internal variables such as WORD_SIZE (indicates whether target system is 32- or 64-bit) and CPU_CORES (number of cpu cores presently available).
In the Julia REPL, is there some way to list all of these internal variables?
AFAIK, whos and names(Main) does not show them...
EDIT: From this previous question on listing exported function names, I see that names(Base) does show these internal variables, along with every other exported item in the Base module.
as #DNF pointed in the comment above, whos(Base.Sys) won't print all the internal(constant) variables in Base. but we can search those variables directly from whos(Base) via:
julia> whos(Base, r"^\s*[A-Z_]+$")
ARGS 0 bytes 0-element Array{UTF8String,1}
BLAS 214 KB Module
CPU_CORES 8 bytes Int64
C_NULL 8 bytes Ptr{Void}
ENDIAN_BOM 4 bytes UInt32
ENV 0 bytes Base.EnvHash with 29 entries
FFTW 149 KB Module
HTML 168 bytes DataType
I 8 bytes UniformScaling{Int64}
IO 92 bytes DataType
JULIA_HOME 66 bytes ASCIIString
LAPACK 933 KB Module
LOAD_PATH 190 bytes 2-element Array{ByteString,1}
MIME 148 bytes DataType
OS_NAME 0 bytes Symbol
STDERR 217 bytes Base.TTY
STDIN 64 KB Base.TTY
STDOUT 217 bytes Base.TTY
VERSION 40 bytes VersionNumber
WORD_SIZE 8 bytes Int64
this lies in the fact that Julia's constants are UPPERCASE. you may find that some Modules are also in the list, but it's easy to identify. indeed, one can use a more complex regex to expel them.
note that, those variables which are not exported into Base will not shown out. e.g.
whos(Base.Libdl)
Related
I am trying to understand why this printf statement gives two different outputs; I think I have a decent understanding of one of the outputs.
Here is the code:
const char *ptr = "hello";
const char array[] = "hello";
//Question 2
printf("%zu %zu\n", sizeof(ptr),sizeof(array));
Now I understand why sizeof(array) returns six: this is because the length of "hello" is 6 plus an additional null terminator.
But I do not understand why sizeof(ptr) is 8; my guess is that all memory addresses in C occupy 8 bits of memory hence the size is 8. Is this correct?
The C language, itself, doesn't define what size a pointer is, or even if all pointers are the same size. But, yes, on your platform, the size of a char* pointer is 8 bytes.
This is typical on 64-bit platforms (the name implies 64-bit addressing which, with 8 bits per byte, is 64/8 = 8 bytes). For example, when compiling for a Windows 64-bit target architecture, sizeof(char*) is 8 bytes; however, when targeting the Windows 32-bit platform, sizeof(char*) will be 4 bytes.
Note also that the "hello" string literal/array is 6 bytes including the nul terminator.
sizeof(ptr) returns 8 because it's the size of a pointer on a typical 64-bit architecture. The pointer ptr is a pointer to a constant character, so its size is 8 bytes on a 64-bit system. The value stored in the pointer is the memory address of the first character in the string literal "hello", so ptr takes up 8 bytes of memory, not 6.
The size of a pointer is platform-dependent and can be different on different architectures. On a 32-bit system, the size of a pointer would typically be 4 bytes.
I have a NASM 64 dll called by ctypes. The program multiplies two 64-bit integers and returns a 128-bit integer, so I am using xmm SIMD instructions. It loops through 10,000 times and stores its results in a memory buffer created by malloc.
Here is the part of the NASM code where the SIMD calculations are performed:
cvtsi2sd xmm0,rax
mov rax,[pcalc_result_0]
cvtsi2sd xmm1,rax
PMULUDQ xmm0,xmm1
lea rdi,[rel s_ptr] ; Pointer
mov rbp,qword[rdi]
mov rcx,[s_ctr]
;movdqa [rbp + rcx],xmm0
movdqu [rbp + rcx],xmm0
add rcx,16
The movdqa instruction does not work (the program crashes, even though it's assembled with the align=16 directive). The movdqu instruction does work, but when I return the array to ctypes, I need to convert the return pointer to 128-bits, but there is no 128-bit ctypes datatype. Here's the relevant part of the ctypes code:
CallName.argtypes = [ctypes.POINTER(ctypes.c_double)]
CallName.restype = ctypes.POINTER(ctypes.c_int64)
n0 = ctypes.cast(a[0],ctypes.POINTER(ctypes.c_int64))
n0_size = int(a[0+1] / 8)
x0 = n0[:n0_size]
where x0 is the returned array converted to a usable form, but not to 128 bits.
There is a post at Handling 128-bit integers with ctypes that deals with passing 128-bit arrays in but not out.
My questions are:
-- Should I use an instruction other than movdqa or movdqu? Of the many SIMD instructions, these seem the most appropriate.
-- Python can handle integers up to any arbitrary size, but apparently ctypes can't. Is there any way to use 128-bit integers from ctypes when there is no ctypes size larger than 64 bits?
You can generate byte arrays containing 16 bytes representing a 128-bit integer and convert to and from byte format. This may not be aligned, so you should use movdqu. I would use an input/output parameter instead of a return value, so Python can manage the memory:
>>> import ctypes
>>> value = 0xaabbccddeeff
>>> int128 = ctypes.create_string_buffer(value.to_bytes(16,'little',signed=True))
>>> int128
<ctypes.c_char_Array_17 object at 0x000001ECCB1D41C8>
>>> int128.raw
b'\xff\xee\xdd\xcc\xbb\xaa\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
(NOTE: The buffer gets null-terminated, which is why it is 17 bytes)
Pass this writable buffer to your function, the function can write the result back to the same buffer. On return use the following to convert back to a Python integer:
>>> hex(int.from_bytes(int128.raw[:16],'little',signed=True))
'0xaabbccddeeff'
Since I’m interested in C++ as well as in Lisp, I tried to reproduce the benchmark presented here written by Guillaume Michel. The benchmark is basically a DAXPY BLAS Level 1 operation performed multiple times on big arrays. The full code was thankfully posted on github and is in both languages roughly one page.
Sadly, I discovered that it was not possible for me to reach the velocity of his calculations for lisp.
For Linux, he got similar results for C++ as well as Lisp:
Size | C++ | Common Lisp
100,000,000 | 181.73 | 183.9
The numbers differ (naturally) both on my PC:
Size | C++ | Common Lisp
100,000,000 | 195.41 | 3544.3
Since I wanted an additional measurement, I started both programs with the time command and got (shortened up):
$ time ./saxpy
real 0m7.381s
$ time ./saxpy_lisp
real 0m40.661s
I assumed different reasons for this blatant difference. I scanned through both code samples, but found no big algorithmic or numeric difference between the C++ and Lisp Version. Then I thought, the usage of buildapp created the delay, so I started the benchmark directly in the REPL. My last resort was to try another version of sbcl, so I downloaded the newest sbcl-1.3.11 and evaluated it there – still the (optimized) Lisp version needs much longer than it’s C++ counterpart.
What am I missing?
I cannot replicate your findings:
sylwester#pus:~/a/bench-saxpy:master$ sbcl --dynamic-space-size 14000
This is SBCL 1.3.1.debian, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (compile-file "saxpy.lisp")
; compiling file "/pussycat/natty-home/westerp/apps/bench-saxpy/saxpy.lisp" (written 06 NOV 2016 12:04:30 AM):
; compiling (DEFUN SAXPY ...)
; compiling (DEFUN DAXPY ...)
; compiling (DEFMACRO TIMING ...)
; compiling (DEFUN BENCH ...)
; compiling (DEFUN MAIN ...)
; /pussycat/natty-home/westerp/apps/bench-saxpy/saxpy.fasl written
; compilation finished in 0:00:00.038
#P"/pussycat/natty-home/westerp/apps/bench-saxpy/saxpy.fasl"
NIL
NIL
* (load "saxpy.fasl")
T
* (main t)
Size, Execution Time (ms)
10, 0.0
100, 0.0
1000, 0.0
10000, 0.1
100000, 0.49999997
1000000, 3.6
10000000, 39.2
100000000, 346.30002
(NIL NIL NIL NIL NIL NIL NIL NIL)
So on my machine it took 346ms (relatively old machine). The whole test took about 5 seconds but that is a series of many tests. Interesting that running from sbcl was faster than making an image and running it:
sylwester#pus:~/a/bench-saxpy:master$ make lisp
buildapp --output saxpy_lisp --entry main --load saxpy.lisp --dynamic-space-size 14000
;; loading file #P"/pussycat/natty-home/westerp/apps/bench-saxpy/saxpy.lisp"
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into saxpy_lisp:
writing 4944 bytes from the read-only space at 0x20000000
writing 3168 bytes from the static space at 0x20100000
writing 60522496 bytes from the dynamic space at 0x1000000000
done]
sylwester#pus:~/a/bench-saxpy:master$ ./saxpy_lisp
Size, Execution Time (ms)
10, 0.0
100, 0.0
1000, 0.0
10000, 0.0
100000, 0.4
1000000, 3.5
10000000, 40.2
100000000, 369.49997
In the C version I get:
100000000, 296.693634, 295.762695, 340.574860
It seems like the C version actually calculates the same in 3 different ways and reports the time it took. Using time wouldn't do it justice.
What does ACL2 exit code 137 mean? The output reads like this:
Form: ( INCLUDE-BOOK "centaur/ubdds/param" ...)
Rules: NIL
Time: 0.00 seconds (prove: 0.00, print: 0.00, other: 0.00)
:REDUNDANT
Note: not introducing any A4VEC field bindings for A, since none of
its fields appear to be used.
Note: not introducing any MODSCOPE field bindings for SCOPE, since
none of its fields appear to be used.
;;; Starting full GC, 10,736,500,736 bytes allocated.
Exit code from ACL2 is 137
top.cert seems to be missing
Looks like "Linux OOM killer" killed your program.
Exit status 137 means program was terminated with singal 9 (SIGKILL) (See here):
When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status.
128+9=137
This message from the log tells us that your ACL2 proof consumed 10Gb of memory:
;;; Starting full GC, 10,736,500,736 bytes allocated.
Linux has a feature where it kills an offending process when the system is very low on memory. It is called OOM Killer: https://www.kernel.org/doc/gorman/html/understand/understand016.html
Such events are logged by kernel. You can immediately see them just to make sure:
$ dmesg |grep -i "killed process"
Mar 7 02:43:11 myhost kernel: Killed process 3841 (acl2) total-vm:128024kB, anon-rss:0kB, file-rss:0kB
There are two ACL2 calls : set-max-mem and maybe-wash-memory which you can use to control memory consumtion.
(include-book "centaur/misc/memory-mgmt" :dir :system) ;; adds ttag
(value-triple (set-max-mem (* 4 (expt 2 30)))) ;; 4 GB
Unfortunately these two calls do not guarantee that memory will be freed. Consider using a more powerful computer for your proof.
An exit code of 137 suggests that it has been killed by bash with -9
Reference: http://www.tldp.org/LDP/abs/html/exitcodes.html
I'm calling a funny API that returns a byte array, but I want a text stream. Is there an easy way to get a text stream from a byte array? For now I just threw together:
(defun bytearray-to-string (bytes)
(let ((str (make-string (length bytes))))
(loop for byte across bytes
for i from 0
do (setf (aref str i) (code-char byte)))
str))
and then wrap the result in with-input-from-string, but that can't be the best way. (Plus, it's horribly inefficient.)
In this case, I know it's always ASCII, so interpreting it as either ASCII or UTF-8 would be fine. I'm using Unicode-aware SBCL, but I'd prefer a portable (even ASCII-only) solution to a SBCL-Unicode-specific one.
FLEXI-STREAMS (http://weitz.de/flexi-streams/) has portable conversion function
(flexi-streams:octets-to-string #(72 101 108 108 111) :external-format :utf-8)
=>
"Hello"
Or, if you want a stream:
(flexi-streams:make-flexi-stream
(flexi-streams:make-in-memory-input-stream
#(72 101 108 108 111))
:external-format :utf-8)
will return a stream that reads the text from byte-vector
There are two portable libraries for this conversion:
flexi-streams, already mentioned in another answer.
This library is older and has more features, in particular the extensible streams.
Babel, a library specificially for character encoding and decoding
The main advantage of Babel over flexi-streams is speed.
For best performance, use Babel if it has the features you need, and fall back to flexi-streams otherwise. Below a (slighly unscientific) microbenchmark illustrating the speed difference.
For this test case, Babel is 337 times faster and needs 200 times less memory.
(asdf:operate 'asdf:load-op :flexi-streams)
(asdf:operate 'asdf:load-op :babel)
(defun flexi-streams-test (bytes n)
(loop
repeat n
collect (flexi-streams:octets-to-string bytes :external-format :utf-8)))
(defun babel-test (bytes n)
(loop
repeat n
collect (babel:octets-to-string bytes :encoding :utf-8)))
(defun test (&optional (data #(72 101 108 108 111))
(n 10000))
(let* ((ub8-vector (coerce data '(simple-array (unsigned-byte 8) (*))))
(result1 (time (flexi-streams-test ub8-vector n)))
(result2 (time (babel-test ub8-vector n))))
(assert (equal result1 result2))))
#|
CL-USER> (test)
Evaluation took:
1.348 seconds of real time
1.328083 seconds of user run time
0.020002 seconds of system run time
[Run times include 0.12 seconds GC run time.]
0 calls to %EVAL
0 page faults and
126,402,160 bytes consed.
Evaluation took:
0.004 seconds of real time
0.004 seconds of user run time
0.0 seconds of system run time
0 calls to %EVAL
0 page faults and
635,232 bytes consed.
|#
If you don't have to worry about UTF-8 encoding (that, essentially, means "just plain ASCII"), you may be able to use MAP:
(map 'string #'code-char #(72 101 108 108 111))
I say go with the proposed flexistream or babel solutions.
But just for completeness and the benefit of future googlers arriving at this page I want to mention sbcl's own sb-ext:octets-to-string:
SB-EXT:OCTETS-TO-STRING is an external symbol in #<PACKAGE "SB-EXT">.
Function: #<FUNCTION SB-EXT:OCTETS-TO-STRING>
Its associated name (as in FUNCTION-LAMBDA-EXPRESSION) is
SB-EXT:OCTETS-TO-STRING.
The function's arguments are: (VECTOR &KEY (EXTERNAL-FORMAT DEFAULT) (START 0)
END)
Its defined argument types are:
((VECTOR (UNSIGNED-BYTE 8)) &KEY (:EXTERNAL-FORMAT T) (:START T) (:END T))
Its result type is:
*
SBCL supports the so-called Gray Streams. These are extensible streams based on CLOS classes and generic functions. You could create a text stream subclass that gets the characters from the byte array.
Try the FORMAT function. (FORMAT NIL ...) returns the results as a string.