I am looking for a JIT compiler or a small compiler library that can be embedded in my program. I indent to use it to compile dynamically generated code that perform complex number arithmetics. The generated code are very simple in structure: no loops, no conditionals, but they can be quite long (a few MB when compiled by GCC). The performance of the resulting machine code is important, while I don't really care about the speed of compilation itself. Which JIT compiler is best for my purpose? Thanks!
Detailed requirements
Support double precision complex number arithmetics
Support basic optimization
Support many CPUs (x86 and x86-64 at least)
Make use of SSE on supported CPUs
Support stack or a large set of registers for local variables
ANSI-C or C++ interface
Cross platform (mainly Linux, Unix)
You might want to take a look at LLVM.
Cint is a c++(ish) environment that offers the ability to mix compiled code and interpreted code. There is a set of optimization tools for the interpreter. ROOT extends this even further by supporting compile and link at run-time at run-time (see the last section of http://root.cern.ch/drupal/content/cint-prompt), though it appears to use the system compiler and thus may not help. All the code is open source.
I make regular use of all these features as part of my work.
I don't know if it makes active use of SIMD instructions, but it seems to meet all your other requirements.
As I see that you are currently using the compile to dynamic library at link on the fly methond, you might consider TCC, though I don't believe that it does much optimization and suspect that it does not support SIMD.
Sounds like you want to be able to compile on the fly and then dynamically load the compiled library (.DLL or .so). This would give you the best performance, with an ANSI-C or C++ interface. So, forget about JITing and consider spawning a C/C++ compiler to do the compilation.
This of course assumes that a compiler can be installed at the point where the dynamically generated code is actually generated.
Related
Goal
I would like to have my Common Lisp (SBCL + GNU Emacs + Slime) environment be sort of like a Smalltalk image in that I want to have a big ball of mud of all my code organized in packages and preferably projects. In other words I have messed about a bit with save-lisp-and-die and setting Lisp in Emacs to bring up the saved image. Where I get lost is the appropriate way to make it work with Swank.
Problem
I believe it is required to put swank hooks inside my Lisp image before save-lisp-and-die. But it seems a bit fragile as on change to either my SBCL version or Slime version it seems to throw a version mismatch.
Question
Am I missing something? Do people work this way or tend to be more separate project as a loadable set of packages under ASDF?
I really miss the Smalltalk way and feel like per project ASDF is a bit clunkier and more rooted in the file system. In comparison it reminds me too much of every other language and their app/project orientation. OTOH it seem a bit more stable-ish re-versions of depended upon packages. Well, the entire versioning hell across languages is another matter.
Any hints how to do what I want or why it isn't such a good idea would be much appreciated.
Images
Common Lisp implementations like SBCL support images. The idea of saved memory appeared early in Lisp in the 60s.
Smalltalk took that idea from Lisp. In many Smalltalk implementations images might be portable (OS, runtime, ...) - especially when using machine independent byte code. SBCL OTOH compiles to native machine code.
Managed source code
Smalltalk added the idea of managed source code. Smalltalk often uses a simple database plus a change log to store source code. One Lisp doing something similar was Xerox Interlisp - but with slightly different approaches.
Other Lisp implementations / IDEs don't support managed source code that way - only the Xerox Interlisp variants - AFAIK.
DEFSYSTEM
In Common Lisp the use of defsystem facilities like ASDF and IDEs like GNU Emacs + SLIME is much more file system based. Code resides in multiple systems, which are files in a directory with a system description.
It's not even clear that it's meaningful to load a newer version of a system into a Lisp system where an older version is loaded. One might be able to arrange that, but there is nothing preventing me from messing that up.
Updating Lisp
Updating a Lisp like SBCL from one version to another might
make the saved image incompatible to the runtime
make the compiled code in FASL files incompatible with the runtime
You might save an image with the runtime included/bundled. That way you have the right combination of image and runtime.
But when you update the runtime, you usually/often need to regenerate a new compatible images with your code loaded.
Since SBCL brings releases once a month, there is a temptation to update regularly. Other implementations might use different strategies: LispWorks is an example. LispWorks is released much less often and publishes patches between releases, which are loaded into the released version.
Updating SLIME
I have no idea if it would be possible to update a loaded SLIME (a SLIME which has been already loaded in an earlier version into a Lisp system) by loading a new version on top. Probably a good idea to check with the SLIME maintainers.
This question might sound a bit weird...
Imagine I have an MPI application, but I don't have a system with MPI installed.
So I want to compile the application with no MPI support (1-process, 1-thread) without modifying source code.
Is that possible?
I found somewhere a "mimic_mpi.h" wrapper which is supposed to do exactly what I want. But there were some MPI functions missing in there (e.g., MPI_Cart_create, MPI_Cart_get, etc.), so I didn't succeed.
mimic_mpi.h http://openmx.sourcearchive.com/documentation/3.2.4.dfsg-3/mimic__mpi_8h-source.html
mimic_mpi.c http://openmx.sourcearchive.com/documentation/3.2.4.dfsg-3/mimic__mpi_8c-source.html
Do you know any other approach I could use to compile MPI apps with no MPI support?
Thanks in advance!
You can run a "real" MPI application easily with a single process. In practice this even works without using mpiexec/mpirun although I'm not sure if that's officially supported. That said a full and confirming 1-process MPI "serial" implementation would probably become rather complex and its own library - so in that case, why not just use a real full MPI implementation?
I hope you see the circle I'm trying to draw:
If you want full MPI behavior, just use an MPI implementation - regardless if it's just limited to a single process.
In practice, applications that want to be able to function with or without MPI often seem to use their own MPI abstractions using domain specific communication wrappers, #ifdef HAVE_MPI or more complex macros.
I would want to compile existing software into presentation that can later be run on different architectures (and OS).
For that I need a (byte)code that can be easily run/emulated on another arch/OS (LLVM IR? Some RISC assemby?)
Some random ideas:
Compiling into JVM bytecode and running with java. Too restricting? C-compilers available?
MS CIL. C-Compilers available?
LLVM? Can Intermediate representation be run later?
Compiling into RISC arch such as MMIX. What about system calls?
Then there is the system call mapping thing, but e.g. BSD have system call translation layers.
Are there any already working systems that compile C/C++ into something that can later be run with an interpreter on another architecture?
Edit
Could I compile existing unix software into not-so-lowlevel binary, which could be "emulated" more easily than running full x86 emulator? Something more like JVM than XEN HVM.
There are several C to JVM compilers listed on Wikipedia's JVM page. I've never tried any of them, but they sound like an interesting exercise to build.
Because of its close association with the Java language, the JVM performs the strict runtime checks mandated by the Java specification. That requires C to bytecode compilers to provide their own "lax machine abstraction", for instance producing compiled code that uses a Java array to represent main memory (so pointers can be compiled to integers), and linking the C library to a centralized Java class that emulates system calls. Most or all of the compilers listed below use a similar approach.
C compiled to LLVM bit code is not platform independent. Have a look at Google portable native client, they are trying to address that.
Adobe has alchemy which will let you compile C to flash.
There are C to Java or even JavaScript compilers. However, due to differences in memory management, they aren't very usable.
Web Assembly is trying to address that now by creating a standard bytecode format for the web, but unlike the JVM bytecode, Web Assembly is more low level, working at the abstraction level of C/C++, and not Java, so it's more like what's typically called an "assembly language", which is what C/C++ code is normally compiled to.
LLVM is not a good solution for this problem. As beautiful as LLVM IR is, it is by no means machine independent, nor was it intended to be. It is very easy, and indeed necessary in some languages, to generate target dependent LLVM IR: sizeof(void*), for example, will be 4 or 8 or whatever when compiled into IR.
LLVM also does nothing to provide OS independence.
One interesting possibility might be QEMU. You could compile a program for a particular architecture and then use QEMU user space emulation to run it on different architectures. Unfortunately, this might solve the target machine problem, but doesn't solve the OS problem: QEMU Linux user mode emulation only works on Linux systems.
JVM is probably your best bet for both target and OS independence if you want to distribute binaries.
As Ankur mentions, C++/CLI may be a solution. You can use Mono to run it on Linux, as long as it has no native bits. But unless you already have a code base you are trying to port at minimal cost, maybe using it would be counter productive. If it makes sense in your situation, you should go with Java or C#.
Most people who go with C++ do it for performance reasons, but unless you play with very low level stuff, you'll be done coding earlier in a higher level language. This in turn gives you the time to optimize so that by the time you would have been done in C++, you'll have an even faster version in whatever higher level language you choose to use.
The real problem is that C and C++ are not architecture independent languages. You can write things that are reasonably portable in them, but the compiler also hardcodes aspects of the machine via your code. Think about, for example, sizeof(long). Also, as Richard mentions, there's no OS independence. So unless the libraries you use happen to have the same conventions and exist on multiple platforms then it you wouldn't be able to run the application.
Your best bet would be to write your code in a more portable language, or provide binaries for the platforms you care about.
I have seen a few blog entries on this and have had a discussion or two with my team mates but I would like to see what the stack overflow community thinks.
So why does the Adobe Alchemy Tool create so much faster running flash byte code than the flex compiler?
Also, when will the flex compiler be able to make similar performance gains?
Will it require programmer specific use of special Array's or something of that nature to get the same performance?
Alchemy is an implementation of LLVM in ActionScript. Simply put, it's an virtual machine that uses a ByteArray as it's memory store.
The C code compiled by Alchemy has direct access to "memory" (via some opcodes introduced in Flash 10), allowing it to chunk memory around at it's leisure (including pointers to objects). This results in some, but by no means all, code running faster. Some types of code will actually run slower in Alchemy due to it being a VM running on top of the AVM (another VM).
Additionally, Alchemy does not have native access to ActionScript classes and must access them through interop classes.
The alchemy tool creates code that uses instructions in the flash player that aren't available to the regular compiler (and the talk is that these instructions were exposed especially for alchemy).
Whether the regular compiler will eventually make similar gains, hopefully. It's been proven a few times that the compiler creates substandard code, and there are a couple of projects which optimise the generated code. These may shame Adobe into improving.
Chances are, no, there won't be anything special a programmer needs to do to get these performance gains (though check out the optimising blogs, writing loops in a particular way means they can be optimised better).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
A few years ago I looked into using some build system that isnt Make, and tools like CMake and SCons seemed pretty primitive. I'd like to find out if the situation has improved. So, under the following criteria, what is currently the best build tool:
platform agnostic: should work on windows, linux, mac
language agnostic: should have built-in support for common things like building C/C++ and other static langs. I guess it doesn't need to support the full autotools suite.
extensible: I need to be able to write rules to generate files, like from restructuredText, latex, custom formats, etc. I dont really care what language I have to write the rules in, but I would prefer a real language rather than a DSL.
I would prefer to avoid writing any XML by hand, which I think for example ant requires.
Freely available (preferably open source)
The term "best" is slightly subjective, but I think answers can be rated objectively by the criteria above.
I'd definitively put my vote up for premake. Although it is not as powerful as it's older brothers, it's main advantage is absurd simplicity and ease of use. Makes writing multi-compiler, multi-platform code a breeze, and natively generates Visual Studio solutions, XCode projects, Makefiles, and others, without any additional work needed.
So, judging purely by the criteria set forth in the question, the build system that seems like the best fit is probably waf - pure Python, provides support for C++ and other languages, general, powerful, not a DSL.
However, from my personal experience, I prefer CMake for C++ projects. (I tried CMake, SCons, and waf, and liked them in roughly that order). CMake is a general solution, but it has built-in support for C++ that makes it nicer than a more generic solution when you're actually doing C++.
CMake's build model for C++ is more declarative and less imperative, and thus, to me, easier to use. The CMake language syntax isn't great, but a declarative build with odd syntax beats an imperative build in Python. Of the three, CMake also seems to have the best support for "advanced" things like precompiled headers. Setting up precompiled headers reduced my rebuild time by about 70%.
Other pluses for CMake include decent documentation and a sizable community. Many open source libraries have CMake build files either in-tree or provided by the CMake community. There are major projects that already use CMake (OGRE comes to mind), and other major projects, like Boost and LLVM, are in the process of moving to CMake.
Part of the issue I found when experimenting with build systems is that I was trying to build a NPAPI plugin on OS X, and it turns out that very few build systems are set up to give XCode the exact combination of flags required to do so. CMake, recognizing that XCode is a complex and moving target, provides a hook for manually setting commands in generated XCode projects (and Visual Studio, I think). This is Very Smart as far as I'm concerned.
Whether you're building a library or an application may also determine which build system is best. Boost still uses a jam-based system, in part because it provides the most comprehensive support for managing build types that are more complex than "Debug" and "Release." Most boost libraries have five or six different versions, especially on Windows, anticipating people needing compatible libraries that link against different versions of the CRT.
I didn't have any problems with CMake on Windows, but of course your mileage may vary. There's a decent GUI for setting up build dependencies, though it's clunky to use for rebuilds. Luckily there's also a command-line client. What I've settled on so far is to have a thin wrapper Makefile that invokes CMake from an objdir; CMake then generates Makefiles in the objdir, and the original Makefile uses them to do the build. This ensures that people don't accidentally invoke CMake from the source directory and clutter up their repository. Combined with MinGW, this "CMake sandwich" provides a remarkably consistent cross-platform build experience!
Of course that depends on what your priorities are. If you are looking primarily for ease of use, there are at least two new build systems that hook into the filesystem to automatically track dependencies in a language agnostic fashion.
One is tup:
http://gittup.org/tup/
and the other is fabricate:
http://code.google.com/p/fabricate/
The one that seems to be the best performing, portable, and mature (and the one I have actually used) is tup. The guy who wrote it even maintains a toy linux distro where everything is a git submodule, and everything (including the kernel) is build with tup. From what I've read about the kernel's build system, this is quite an accomplishment.
Also, Tup cleans up old targets and other cruft, and can automatically maintain your .gitignore files. The result is that it becomes trivial to experiment with the layout and names of your targets, and you can confidently jump between git revisions without rebuilding everything. It's written in C.
If you know haskell and are looking for something for very advanced use cases, check out shake:
http://community.haskell.org/~ndm/shake/
Update: I haven't tried it, but this new "buildsome" tool also hooks into the filesystem, and was inspired by tup, so is relevant:
https://github.com/ElastiLotem/buildsome
CMake
CMake is an extensible, open-source
system that manages the build process
in an operating system and in a
compiler-independent manner.
Gradle seems to match all the criteria mentioned above.
It's a build system which took the best of Maven and Ant combined. To me, that's the best.
The Selenium project is moving over to Rake, not because its the best but because it handles multiple languages slightly better than all the other build tools and is cross platform (developed in Ruby).
All build tools have their issues and people learn to live with them. Something that runs on the JVM tends to be really good for building apps so Ant, Maven (i know its hideous), Ivy, Rake
Final Builder is well known in Windows world
smooth build matches most of your requirements.
platform agnostic: yes, it's written in java
language agnostic: it doesn't support c/c++t yet, only java but it is extensible via plugins written in java so adding more compilers support is not a problem
extensible: yes, you can implement smooth function via java plugin, you can also create smooth function via defining it as expression built of other smooth functions.
I would prefer to avoid writing any XML: you won't see a single line of it in smooth build
Freely available: yes, Apache 2 license
disclosure: I'm the author of smooth build.