This is mostly a stupid question, since UPX (a tool that wrings extra bytes out of your executable files) saves a tiny amount of space over the built in compression in the buildapp tool.
A very small demo application creates a 42 megabyte file. Understandable, since the SBCL environment isn't tiny.
Passing the --compress-core option to buildapp shrinks that down to 9.2MB.
I thought I'd try throwing UPX at the resulting binary, and the savings only amounts to a few more bytes: 9994288 -> 9871360
However, the resulting file no longer runs anymore - it just jumps into the SBCL REPL (with no errors, it's as if I just ran sbcl by hand), and some poking around there reveals that the functions making up my test program no longer exist.
What did UPX do to the binary that resulted in this breakage?
This may not be the answer, but it may serve as a clue: I've found that if you add anything, even a single byte, to the end of an SBCL executable created with sb-ext:save-lisp-and-die, then all the definitions disappear, just as you described.
Perhaps SBCL creates executables by appending the core (which contains your definitions) to a copy of the SBCL ELF (or PE on Windows) binary, plus some metadata at the end so that SBCL can still find the beginning of the core even though it's appended to an executable.
If you hex-edit an executable created with save-lisp-and-die, you'll find that it ends with the string "LCBS" (SBCL backwards), which seems to support my theory. "LCBS" probably serves as a magic number, letting SBCL know that yes, this executable contains its own core.
UPX compresses executables, probably including that magic number at the end. When SBCL opens its UPX-compressed self on disk, it won't find "LCBS" at the end, so it assumes that there is no core appended to the executable.
I can't explain why the standard library seems to still be there if this is the case. It could be that SBCL loads /usr/lib/sbcl/sbcl.core (or its equivalent on Windows) in that case. This could be tested by moving the executable to a machine where SBCL is not installed and seeing if it works at all, and if so, whether you still have car, cdr, list, etc.
Related
I'm new to working with Coq, and I'm progressing through the first volume of the Software Foundations book, but I can't for the life of me figure out how to compile the Basics.v file for the second chapter on induction.
I've seen things floating around about using the Coqc command on the command line, but I don't know how to access the terminal, or at least the windows terminal doesn't recognize the command. If anyone could walk me through this it'd be much appreciated!
In case you are using CoqIDE, there is a menu item "Compile/Compile buffer", which creates a .vo file for the currently loaded .v file. For a few files and early stages of learning, this might be the easiest way.
Then SF likely comes with a make file which you can just run with make. You didn't say what OS your are using - on Linux and Mac this should be trivial, on Windows it depends on how you did install Coq. The Windows installer doesn't come with make, but if you used the Coq Platform scripts to setup Coq on Windows, everything is there.
Otherwise it might get a bit complicated - you need to pass the right options to coqc (which CoqIDE and make do automatically for you).
I am new to Common Lisp. This is how I develop programs in other languages, and also how I now develop programs in Common Lisp:
Open a text editor (e.g. vim or emacs) to create/edit a text file.
Write source code into the text file. (If unsure about the behavior of a snippet of code, and an REPL is available, then evaluate the snippet in the REPL, verify that the snippet evaluates as expected, and then go back to writing more code.)
Save the text file.
Ask the compiler/interpreter to load and run the source code in the text file. (e.g. sbcl --script myprog.lisp)
Go to step 1 if needed.
This is the conventional write-compile-run development cycle for most programming languages. However, in the lisp world, I hear things like "interactive development" and "image-based development", and I feel that I am missing out on an important feature of Common Lisp. How do I do "image-based development" instead of "write-compile-run development"?
Can someone provide a step-by-step example of "image-based development" similar to how I described "write-compile-run development" above?
(Note: I am using SBCL)
In typical Common Lisp implementations the runtime, the compiler, parts of the development environment and the program you are developing reside in the same program and share the same object space. The compiler is always available while you develop the program and the program can be incrementally developed. The development tools have access to all objects and can inspect their state. One can also undefine/remove, replace, enhance functionality from the running program.
Thus:
don't restart the program you are developing. Stay connected and update it. Even days, weeks, or months - if possible.
write code in such a way that the program can be replicated and built from scratch if necessary. Build it from time to time and fix any build problems.
once you use our program and there is an error -> fix the error within the program, while being able to inspect the full error state
creating a running program is either loading all code into a plain Lisp all the time or saving an executable image with the loaded code/data
Fixes to program bugs can also shipped to the user as compiled Lisp files, which gets loaded into the delivered program and update the code then.
Let's say that you are using SBCL with Emacs and SLIME (e. g. through Portacle).
Open Emacs
Start SLIME (M-x slime) — this starts a “plain” Lisp process in the background and connects the editor functions provided by slime to it; then gives you a REPL that is also connected into this process (image)
Open a text file (e. g. foo.lisp)
Type some code
Press C-c C-k to compile the file and load it into the running Lisp process
Switch to the REPL, try it out
Switch to the Lisp file (step 4).
This is just very basic usage. Further things to do/learn
You can also compile and load just a single toplevel form (C-c C-c)
Learn about packages
Learn about systems (ASDF)
Learn how to use Quicklisp to get the libraries you want
Learn how to access inline documentation from the REPL
Note that you never need to unload your program, you just modify it, even when downloading and loading new libraries. This makes the feedback cycle instantaneous in most cases. You also never need to switch away from the IDE (Emacs).
I know the sticky bit on directories allows file deletion only by the owner of files contained inside the sticky dir.
But I can also chmod 1777 a File and the ls -l cmd shows that the file indeed has the sticky bit set.
Nothing.
From the docs: https://www.freebsd.org/cgi/man.cgi?query=sticky&sektion=7&apropos=0&manpath=FreeBSD+10.3-RELEASE+and+Ports
A special file mode, called the sticky bit (mode S_ISTXT), is used to
indicate special treatment for directories. It is ignored for regular
files. See chmod(2) or the file for an explanation of file
modes.
Erik Bennett linked the Wikipedia article on the history of the sticky bit - its original purpose was a hint to the kernel to try to keep files hot in swap, for performance. Modern OSs have abandoned this feature though.
When set, it instructed the operating system to retain the text segment of the program in swap space after the process exited. This speeds up subsequent executions by allowing the kernel to make a single operation of moving the program from swap to real memory.
One notable problem with "stickied" programs was replacing the executable (for instance, during patching); to do so required removing the sticky bit from the executable, executing the program and exiting to flush the cache, replacing the binary executable, and then restoring the sticky bit.
Currently, this behavior is only operative in HP-UX and UnixWare. Solaris appears to have abandoned this in 2005. The 4.4-Lite release of BSD retained the old sticky bit behavior, but it has been subsequently dropped from OpenBSD (as of release 3.7) and FreeBSD (as of release 2.2.1). No version of Linux has ever supported this traditional behavior.
we have a old AIX server and it has an executable file and we want to rewrite the same logic of the executable file on linux server, so we are trying to read it but could not find a way to do that.could you please let us know if there is a way to decipher this file
$ file execfile
execfile: executable (RISC System/6000) or object module
The IBM RS/6000 has a POWER architecture CPU, possibly a PowerPC 603 or PowerPC 604, or possibly one of the newer models like POWER1, POWER2, POWER3, etc. The most recent (current) systems use POWER7 or POWER8.
Anyway, if the system has the compiler and toolchain installed on it then there should be a decent symbolic debugger included, and you should be able to use that to disassemble any executable. Depending on exactly which version of the OS it was compiled on, and which compiler was used, you might even be able to use PowerPC tools on some other OS, such as MacOS, or even potentially a cross-compiler toolchain on any type of system, to disassemble the program. For example GDB built for PowerPC may be able to disassemble the program.
However if the executable has been stripped of symbols (as was typically the case on AIX systems, IIRC), and especially if it had been run through the most powerful optimizing stage of the compiler, then you'll be pretty much lost and what you are trying to do will be impractical and require many man hours to decipher -- indeed many thousands of man hours for any significantly sized program, even if you're able to hire someone to help who is familiar with the code generation patterns of the particular compiler which was used.
You might be better off trying to hire an archeologist to help you dig through the specific landfill where you might hope to find listings or backup tapes or CDs or disks containing the original source code, or specification documents, etc., for this program. Seriously.
Or try to find and (re-)hire the original author(s).
Julia language compiles the script every time, can't we compile binaries with julia instead?
I tried a small helloworld script with println function it took like 2,3 seconds for julia to show the output! It would be better if we can make binaries instead of compiling every time
Update: There have been some changes in Julia, since I asked this question. Though I'm not following the updates for julia anymore, since I've asked this question and if you're looking for something similar, look into the below answers and comments by people who are following julia.
Also, its good to know that now it takes around 150ms to load a script.
Keno's answer is spot on, but maybe I can give a little more detail on what's going on and what we're planning to do about it.
Currently there is only an LLVM JIT mode:
There's a very trivial interpreter for some simple top-level statements.
All other code is jitted into machine code before execution. The code is aggressively specialized using the run-time types of the values that the code is being applied to, propagated through the program using dynamic type inference.
This is how Julia gets good performance even when code is written without type annotations: if you call f(1) you get code specialized for Int64 — the type of 1 on 64-bit systems; if you call f(1.0) you get a newly jitted version that is specialized for Float64 — the type of 1.0 on all systems. Since each compiled version of the function knows what types it will be getting, it can run at C-like speed. You can sabotage this by writing and using "type-unstable" functions whose return type depends on run-time data, rather than just types, but we've taken great care not to do that in designing the core language and standard library.
Most of Julia is written in itself, then parsed, type-inferred and jitted, so bootstrapping the entire system from scratch takes some 15-20 seconds. To make it faster, we have a staged system where we parse, type-infer, and then cache a serialized version of the type-inferred AST in the file sys.ji. This file is then loaded and used to run the system when you run julia. No LLVM code or machine code is cached in sys.ji, however, so all the LLVM jitting still needs to be done every time julia starts up, which therefore takes about 2 seconds.
This 2-second startup delay is quite annoying and we have a plan for fixing it. The basic plan is to be able to compile whole Julia programs to binaries: either executables that can be run or .so/.dylib shared libraries that can be called from other programs as though they were simply shared C libraries. The startup time for a binary will be like any other C program, so the 2-second startup delay will vanish.
Addendum 1: Since November 2013, the development version of Julia no longer has a 2-second startup delay since it precompiles the standard library as binary code. The startup time is still 10x slower than Python and Ruby, so there's room for improvement, but it's pretty fast. The next step will be to allow precompilation of packages and scripts so that those can startup just as fast as Julia itself already does.
Addendum 2: Since June 2015, the development version of Julia precompiles many packages automatically, allowing them to load quickly. The next step is static compilation of entire Julia programs.
At the moment Julia JIT compiles its entire standard library on startup. We are aware of the situation and are currently working on caching the LLVM JIT output to remedy the situation, but until then, there's no way around it (except for using the REPL).