I have an application that uses MPI_COMM_WORLD. I'm building the application with HPC Pack 2008 R2 MPI and everything works fine on my local and most PCs.
Occasionally when installing on a different PC I will run into issues with competing versions of MPI (e.g. Intel). This is usually solved by prepending my HPC version to the PATH.
I have recently hit an issue that I can't work around.
My MPI is first in the path, but I'm getting an error "link library mkl_intel_thread.dll" cannot be found. This tells me the app is looking at the Intel version.
where mpiexec
mkl_thread not found
So my general questions are:
Is there a version of MPI that is compatibly with all others?
Or is there a way to compile my application in a way that I can
ensure it looks for the right MPI library?
Or is there a way to compile to make the application MPI agnostic?
Thanks in advance
The advantage of MPI is the MPI standard. This means as long as you stick to a certain version of the MPI standard, your programm should be compatible with MPI standard compliant implementations.
The missing Math Kernel Library has nothing to do with MPI incompatibility.
Related
Can you send MPI messages across two libraries?
Example - If I have MS-MPI installed on a windows machine and then Open-MPI on a linux cluster (Same Network), can I get messages between code running on the two different OS.
Basically multiple program multiple data MPMD but using Windows and Linux resources.
Just need to know if this is possible, and if so any info would be nice, but not needed.
No, that is not possible. You will even get into great trouble if you try this with different versions or configurations of one implementation.
MPI is targeted at homogeneous installations on HPC systems. The communication protocol is not standardized, only the programming interface is.
In general: no. This is not specified within the MPI standard, and most implementations do not support such jobs. It is a rather uncommon use case, I guess.
However, Intel MPI does provide cross-os launching of jobs, see:
https://software.intel.com/en-us/mpi-developer-guide-linux-cross-os-launch-mode
Some detail:
Intel MPI, as well as other commercial MPI implementations are based on the MPICH open source project. I am not sure if cross-os launching can be achieved somehow via MPICH, though. A quick google research only provided negative, but possibly outdated results, e.g.
https://lists.mcs.anl.gov/pipermail/mpich2-dev/2005-July/000085.html
I am just starting to learn MPI. The implementation is MPICH2.
The first thing I encountered is SMPD i.e Simple Multi-Purpose Daemon.
I want to know what exactly it is and how it is related to MPI
Most of the links on internet talks about its usage, but not the internal details. At max I know that:
Multipurpose daemon* (MPD) is the Intel® MPI Library process
management system for starting parallel jobs.
Where can I learn more about it ?
SMPD is one implementation of parallel process manager, i.e. a system for launching/monitoring processes in a parallel job. SMPD was specifically useful for MPI jobs on Windows or a mix of Windows and Unix. See the MPICH FAQ: http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_What_are_process_managers.3F
Also note that the newest versions of MPICH use Hydra as the default process manager, and support for Windows has been dropped.
I know it is weird but I need to execute a Qt GUI application that supports X11 protocol on MS Windows platform.
It is obvious that if I build my Qt source code in Unix environment it automatically becomes an X11 application and this application can be run only on Unix Operating Systems, not MS Windows.
I tried to use MXE (MinGW cross environment) to compile my application in Unix for Windows, but it does not help for X11 issue. It creates an application for Windows but the application is not an XWindow application.
There is also a XPortMinGW project for minGW. But I do not know if it works for Qt applications.
Is there any way to build my Qt application with X11 support that can run in Windows OS?
I don't think that there's any need for cross-compilation. There are two problems:
Getting Xlib compiled for Windows. I don't know offhand of any maintained Xlib implementations that are ported to Windows. You will definitely need to find one, or make one. Hopefully Xlib depends only on a handful of posix APIs and they may be easy to translate to win32. Forget about Cygwin.
You may have luck porting Xlib yourself to Qt's network backend and making it cross-platform in the process, too. Xlib would simply use the core and network modules of Qt. No problem with that.
The good news here are that Qt 5 comes with a bundled Xlib implementation. It would be presumably much easier to get that to compile on Windows than some other implementation.
Configuring Qt to build using said Xlib on Windows. To do it right, you'll probably want to mess a bit with and rebuild the configure tool that comes with Qt, and add a makespec that will link with the Xlib.
This should be much easier to do with Qt 5.
It's not an entirely trivial thing to do, but should be possible. Give yourself a month and expect to get very intimate with some details of Qt's configuration and Xlib.
Notes About Cygwin
If your application is closed-source and distributed outside of your organization, you need to buy a commercial license in order to redistribute cygwin.dll. Otherwise, Cygwin is GPL and merely linking to cygwin.dll makes your application a derived work.
Cygwin becomes problematic when you mix posix- and non-posix code in the same application, especially if the non-posix code is a large framework like Qt. You can of course use the Posix platform code in Qt and make it all run on Cygwin, but that way you lose performance and expose yourself to shortcoming of Cygwin's emulation of POSIX APIs.
Using Cygwin for Xlib / windowing system but having Qt otherwise use Windows APIs may require a lot of tweaking for Qt code. So may using Qt with Xlib on Windows of course, but at least you don't have to debug/work around Cygwin's quirks.
I have several Windows box and Linux box interconnected with Infiniband, and I need to run MPI jobs in both environment, does anyone know what's the best way to interwork them ?
Currently, I am considering using the beta release of windows binary of open MPI. Maybe I need to add additional things into my hpc sw stack ? Or should I just forget about MPI and directly code in a lower layer for the getting the windows part to work, since there are few jobs that needs windows anyway.
Any idea is appreciated. Many thanks!
So I dig through some of forums and find that
openMPI currently does not support interworking of task spawning between windows and linux systems, however MpiCh2 seems to be capable of interworking task spawning. For sending and receiving messages using MPI send, I will need to investigate more
I would want to compile existing software into presentation that can later be run on different architectures (and OS).
For that I need a (byte)code that can be easily run/emulated on another arch/OS (LLVM IR? Some RISC assemby?)
Some random ideas:
Compiling into JVM bytecode and running with java. Too restricting? C-compilers available?
MS CIL. C-Compilers available?
LLVM? Can Intermediate representation be run later?
Compiling into RISC arch such as MMIX. What about system calls?
Then there is the system call mapping thing, but e.g. BSD have system call translation layers.
Are there any already working systems that compile C/C++ into something that can later be run with an interpreter on another architecture?
Edit
Could I compile existing unix software into not-so-lowlevel binary, which could be "emulated" more easily than running full x86 emulator? Something more like JVM than XEN HVM.
There are several C to JVM compilers listed on Wikipedia's JVM page. I've never tried any of them, but they sound like an interesting exercise to build.
Because of its close association with the Java language, the JVM performs the strict runtime checks mandated by the Java specification. That requires C to bytecode compilers to provide their own "lax machine abstraction", for instance producing compiled code that uses a Java array to represent main memory (so pointers can be compiled to integers), and linking the C library to a centralized Java class that emulates system calls. Most or all of the compilers listed below use a similar approach.
C compiled to LLVM bit code is not platform independent. Have a look at Google portable native client, they are trying to address that.
Adobe has alchemy which will let you compile C to flash.
There are C to Java or even JavaScript compilers. However, due to differences in memory management, they aren't very usable.
Web Assembly is trying to address that now by creating a standard bytecode format for the web, but unlike the JVM bytecode, Web Assembly is more low level, working at the abstraction level of C/C++, and not Java, so it's more like what's typically called an "assembly language", which is what C/C++ code is normally compiled to.
LLVM is not a good solution for this problem. As beautiful as LLVM IR is, it is by no means machine independent, nor was it intended to be. It is very easy, and indeed necessary in some languages, to generate target dependent LLVM IR: sizeof(void*), for example, will be 4 or 8 or whatever when compiled into IR.
LLVM also does nothing to provide OS independence.
One interesting possibility might be QEMU. You could compile a program for a particular architecture and then use QEMU user space emulation to run it on different architectures. Unfortunately, this might solve the target machine problem, but doesn't solve the OS problem: QEMU Linux user mode emulation only works on Linux systems.
JVM is probably your best bet for both target and OS independence if you want to distribute binaries.
As Ankur mentions, C++/CLI may be a solution. You can use Mono to run it on Linux, as long as it has no native bits. But unless you already have a code base you are trying to port at minimal cost, maybe using it would be counter productive. If it makes sense in your situation, you should go with Java or C#.
Most people who go with C++ do it for performance reasons, but unless you play with very low level stuff, you'll be done coding earlier in a higher level language. This in turn gives you the time to optimize so that by the time you would have been done in C++, you'll have an even faster version in whatever higher level language you choose to use.
The real problem is that C and C++ are not architecture independent languages. You can write things that are reasonably portable in them, but the compiler also hardcodes aspects of the machine via your code. Think about, for example, sizeof(long). Also, as Richard mentions, there's no OS independence. So unless the libraries you use happen to have the same conventions and exist on multiple platforms then it you wouldn't be able to run the application.
Your best bet would be to write your code in a more portable language, or provide binaries for the platforms you care about.