I am running a common lisp project that fetches market data every 5 seconds. I made some tweaks to the code and want to update it on the production environment. The event loop is really standard:
(loop
(fetch-data)
(sleep 5))
Because of the blocking nature of the loop I don't have the REPL at my disposal.
My question: Can I dynamically update the running code?
I know that I can recompile the project with (asdf:compile-system :system-name)
I also know that I can redefine classes at runtime. (Not that I am using classes in my implementation)
However now that I cannot use the REPL I must somehow load the environment in another REPL. Is there a way to do that? (I am using SBCL)
It seems to me that the cleanest approach would be to just implement asynchronous data fetching.
Because of the blocking nature of the loop I don't have the REPL at my disposal.
There are ways around it:
start the loop in its own thread.
or connect something like SLIME to a running SWANK in your Lisp
or your loop could regularly check for code updates
I would also not compile the code inside the program, unless you know that the code actually compiles or you can handle compilation errors. Compiling and loading is also usually slower than just loading.
Note that loading code into a running program is not standardized. With SBCL you could not only have threads, but the threads might run concurrently. There are changes, which might not be thread safe or which are causing problems in a running program. Thus you need to plan what kind of changes you want to do.
One thing which simplifies changing the code: load the update while your loop sleeps, if that is possible within the 5 second window.
Some programs might need some more architecture, like the ability to shut down parts of the program while it is updating these parts.
Not sure if that's going to work in your situation, but I have used quite successfully the swank::handle-request calling in the loop for REPLing with a running interactive application:
(defun handle-swank-requests ()
(let ((connection (or swank::*emacs-connection*
(swank::default-connection))))
(when connection
(swank::handle-requests connection t))))
(defun main-cycle ()
(loop
(restart-case
(progn
(handle-swank-requests)
(fetch-data)
(sleep 5))
(continue () :report "Continue" (print "Continued after error from SWANK")))))
The restart-case construction allows to not break the loop on errors, and continue it after errors get fixed.
I'm not sure if that approach is bulletproof, AFAIK swank uses threading for requests handling by default, so there may be some synchronization issues, so probably not for production usage, for development / debugging.
AFAIK CEPL uses something similar, may be more sophisticated and robust version. Also nice CEPL demo video.
Related
I am new to Common Lisp. This is how I develop programs in other languages, and also how I now develop programs in Common Lisp:
Open a text editor (e.g. vim or emacs) to create/edit a text file.
Write source code into the text file. (If unsure about the behavior of a snippet of code, and an REPL is available, then evaluate the snippet in the REPL, verify that the snippet evaluates as expected, and then go back to writing more code.)
Save the text file.
Ask the compiler/interpreter to load and run the source code in the text file. (e.g. sbcl --script myprog.lisp)
Go to step 1 if needed.
This is the conventional write-compile-run development cycle for most programming languages. However, in the lisp world, I hear things like "interactive development" and "image-based development", and I feel that I am missing out on an important feature of Common Lisp. How do I do "image-based development" instead of "write-compile-run development"?
Can someone provide a step-by-step example of "image-based development" similar to how I described "write-compile-run development" above?
(Note: I am using SBCL)
In typical Common Lisp implementations the runtime, the compiler, parts of the development environment and the program you are developing reside in the same program and share the same object space. The compiler is always available while you develop the program and the program can be incrementally developed. The development tools have access to all objects and can inspect their state. One can also undefine/remove, replace, enhance functionality from the running program.
Thus:
don't restart the program you are developing. Stay connected and update it. Even days, weeks, or months - if possible.
write code in such a way that the program can be replicated and built from scratch if necessary. Build it from time to time and fix any build problems.
once you use our program and there is an error -> fix the error within the program, while being able to inspect the full error state
creating a running program is either loading all code into a plain Lisp all the time or saving an executable image with the loaded code/data
Fixes to program bugs can also shipped to the user as compiled Lisp files, which gets loaded into the delivered program and update the code then.
Let's say that you are using SBCL with Emacs and SLIME (e. g. through Portacle).
Open Emacs
Start SLIME (M-x slime) — this starts a “plain” Lisp process in the background and connects the editor functions provided by slime to it; then gives you a REPL that is also connected into this process (image)
Open a text file (e. g. foo.lisp)
Type some code
Press C-c C-k to compile the file and load it into the running Lisp process
Switch to the REPL, try it out
Switch to the Lisp file (step 4).
This is just very basic usage. Further things to do/learn
You can also compile and load just a single toplevel form (C-c C-c)
Learn about packages
Learn about systems (ASDF)
Learn how to use Quicklisp to get the libraries you want
Learn how to access inline documentation from the REPL
Note that you never need to unload your program, you just modify it, even when downloading and loading new libraries. This makes the feedback cycle instantaneous in most cases. You also never need to switch away from the IDE (Emacs).
Julia language compiles the script every time, can't we compile binaries with julia instead?
I tried a small helloworld script with println function it took like 2,3 seconds for julia to show the output! It would be better if we can make binaries instead of compiling every time
Update: There have been some changes in Julia, since I asked this question. Though I'm not following the updates for julia anymore, since I've asked this question and if you're looking for something similar, look into the below answers and comments by people who are following julia.
Also, its good to know that now it takes around 150ms to load a script.
Keno's answer is spot on, but maybe I can give a little more detail on what's going on and what we're planning to do about it.
Currently there is only an LLVM JIT mode:
There's a very trivial interpreter for some simple top-level statements.
All other code is jitted into machine code before execution. The code is aggressively specialized using the run-time types of the values that the code is being applied to, propagated through the program using dynamic type inference.
This is how Julia gets good performance even when code is written without type annotations: if you call f(1) you get code specialized for Int64 — the type of 1 on 64-bit systems; if you call f(1.0) you get a newly jitted version that is specialized for Float64 — the type of 1.0 on all systems. Since each compiled version of the function knows what types it will be getting, it can run at C-like speed. You can sabotage this by writing and using "type-unstable" functions whose return type depends on run-time data, rather than just types, but we've taken great care not to do that in designing the core language and standard library.
Most of Julia is written in itself, then parsed, type-inferred and jitted, so bootstrapping the entire system from scratch takes some 15-20 seconds. To make it faster, we have a staged system where we parse, type-infer, and then cache a serialized version of the type-inferred AST in the file sys.ji. This file is then loaded and used to run the system when you run julia. No LLVM code or machine code is cached in sys.ji, however, so all the LLVM jitting still needs to be done every time julia starts up, which therefore takes about 2 seconds.
This 2-second startup delay is quite annoying and we have a plan for fixing it. The basic plan is to be able to compile whole Julia programs to binaries: either executables that can be run or .so/.dylib shared libraries that can be called from other programs as though they were simply shared C libraries. The startup time for a binary will be like any other C program, so the 2-second startup delay will vanish.
Addendum 1: Since November 2013, the development version of Julia no longer has a 2-second startup delay since it precompiles the standard library as binary code. The startup time is still 10x slower than Python and Ruby, so there's room for improvement, but it's pretty fast. The next step will be to allow precompilation of packages and scripts so that those can startup just as fast as Julia itself already does.
Addendum 2: Since June 2015, the development version of Julia precompiles many packages automatically, allowing them to load quickly. The next step is static compilation of entire Julia programs.
At the moment Julia JIT compiles its entire standard library on startup. We are aware of the situation and are currently working on caching the LLVM JIT output to remedy the situation, but until then, there's no way around it (except for using the REPL).
I have C source with MPI calls.
I wonder can I get sequential program from the source by linking with some MPI stub library? Where can I get this lib?
Most correctly-written MPI programs should not depend on the number of processes they use to get a correct answer -- eg, if you run them on one process (mpirun -np 1 ./a.out) they should still work. So you oughtn't need a stub library - just use MPI. (If for some reason you just don't want extraneous libraries kicking around, it's certainly possible to write stubs and link against them -- I did this back in the day when setting up MPI on my laptop was a huge PITA, you could use this as a starting point and add any functionality you need. But these days, fiddling with the stub library is probably going to be more work than just using an existing MPI implementation.)
If your MPI program does not currently work correctly on one processor, a stub library probably won't help; you'll need to find the special cases that it's not handling and fix them.
I don't think this is possible. Contrary to OpenMP, programs using MPI don't necessarily run or produce the same result when you simply take away the MPI part.
PETSc contains a stub MPI library that works for one process (ie serial) execution:
http://www.mcs.anl.gov/petsc/petsc-3.2/include/mpiuni/mpi.h
We are using RDI (IBM Rational Developer for System i) to do cobol development work, we are eager to write automation test cases for our program, to make the testing work easier. But we don't know how to use script to compile and run cobol, which on i-series server.
For now, our solution is that we use scripts prepare test data (insert data to database/files),and then run cobol on RDI manually, finally, run scripts to check the results. It makes our work easier, but still not real automation test.
So, I want to know if there are some methods to invoke the compile&run process according to scripts, such eclipse headless or telnet technologies.
We've already found the solution: use telnet to compile/run program. Because green screen is one kind of telnet, it reliable.
I use a third-party DLL (FTD2xx) to communicate with an external device. Using Qt4, in debug mode everything works fine, but the release crashes silently after successfully completing a called function. It seems to crash at return, but if I write something to the console (with qDebug) at the end of the function, sometimes it does not crash there, but a few, or few dozen lines later.
I suspect a not properly cleaned stack, what the debug build can survive, but the release chokes on it. Did someone encounter a similar problem? The DLL itself cannot be changed, as the source is not available.
It seems the reduction of the optimization level was the only way around. The DLL itself might have problems, as a program which does nothing but calls a single function from that DLL crashes the same way if optimization is turned on.
Fortunately, the size and speed lost by the change in optimization level is negligible.
Edit: for anyone with similar problems on Qt 5.0 or higher: If you change the optimization level (for example, to QMAKE_CXXFLAGS_RELEASE = -O0), it's usually not enough to just rebuild the application. A full "clean all" is required.
Be warned - the EPANET library is not thread safe, it contains a lot of global variables.
Are you calling two methods of that library from different threads?