QtConcurrent is extremely convenient as a high-level interface for multithreaded processing in my data-heavy/CPU-heavy application. The Qt6 upgrades vaguely referenced "Revamped Concurrency APIs" but I didn't see much change, except a reference to being able to pass a custom QThreadPool.
That got me wondering... is it possible to extend QThreadPool into a class that manages threads/tasks on other machines, not just the host machine? Or is that too far from its original design? Or is there another Qt class that can manage distributed processing?
Don't bother linking me to non-Qt solutions. That's not the point of this question.
QtConcurrent doesn't deal with any of it, unfortunately.
In a most general approach, you only need some worker machines on the network, and a way to connect to them via ssh (if they are Unix), or via Windows credentials (on a Windows network). At that point you can send a binary to the worker and execute it. Doing this in Qt is of course possible, but you'd need to wrap some other libraries (e.g. Samba for RPC calls, or openssh) to do that.
No matter whether the software can "distribute itself" or is otherwise installed on the workers, you got it running on multiple machines. Now they have to communicate, with one being a master, and the other being slaves. Master selection could be done via command line arguments, or even by having two binaries: workers that include only the back-end functionality, and a front end that includes both (and has some sort of UI).
At that point you can leverage Qt Remote Objects, the idea being what you'd "distribute" is QObjects that do work in the slots, and return results either via return value of the slot, by sending a signal. It's not as convenient as using QtConcurrent directly, but in general there's no way to distribute work transparently without some introspection that C++ doesn't quite provide yet.
I know that OpenMPI is not a Qt-based solution, it certainly works and makes life easy, and for sure it can interoperate with Qt code - you can even distribute methods and lambdas that way (with some tricks).
If you manage worker objects encapsulated as QObjects, it's not too hard to distribute the work in e.g. round-robin fashion. You could then have a front-end QObject that acts as a proxy: you submit all the work to it, and it signals all the results, but internally it invokes the slots on the remote QObjects.
Would you be interested in a demo? I could write one up if there was enough demand :)
Related
I have a Go program which cannot be rewritten in Common Lisp for efficiency reasons. How can I run it via Common Lisp?
Options so far:
1. CFFI
Using the foreign function interface seems to me like the "correct" way to do this. However, the research I did lead directly to a dead end. If this is the winner, what resources are there to learn about how to interface with Go?
2. Sockets
Leaving the Go program running all the time while listening on a port would work. If this is the best way, I'll continue trying to make it work.
3. Execute System Command
This seems all kinds of wrong.
4. Unknown
Or is there an awesome way I haven't thought of yet?
It depends on what you want to do, but 1-3 are all viable options
1. CFFI
To get this to work you will need to use FFI on both the go and lisp side.
You will need to extern the appropriate function from Go as C functions, and then call them using cffi from lisp. See https://golang.org/cmd/cgo/#hdr-C_references_to_Go on how to extern function in go. In this case you would create a dynamically linkable library (dll or so file) rather than an executable file.
2. Sockets (IPC)
The second option is to run your go program as a daemon and use some form of IPC (such as sockets) to communicate between lisp and go. This works well if your program is long running, or if it makes sense to have a server and one or more clients (the server could just as easily be the lisp code as the go code). Sockets in particular are also more flexible, you could write components in other languages or change languges for one component without having to change the others as long as you maintain the same protocol. Also, you could potentially run the components on seperate hardware. However, using sockets may hurt performance.
There are other IPC methods available, such as FIFO files (named pipes), SHM, and message queues, but they are more system dependent than sockets.
3. System command (subprocess)
The third way is to start a sub-process. This is a viable option, but it has some caveats. First of all, the behavior of starting a sub process is dependent both on the lisp implementation and the operating system. UIOP smooths out a lot of the details for implementation differences, but some are too great to overcome. In particular, depending on the implementation you may or may not be able to run a subprocess in parallel. If not you will have to run a seperate command every time you want to communicate with go, which means waiting for the process to start up every time you need it. You also may, or may not be able to send input to the subprocess after starting it.
Another option is to run a command to start a go process, and then communicate with it using sockets or some other IPC, and then running a command to stop the process before closing the lisp program.
Personally, I think that using sockets is the most attractive option, but depending on your needs, on of the other options might be better suited.
CFFI is to use C from Common Lisp. It's an easy way to get new features without too much hassle as the libraries out there usually are written in C or have a C interface. If you can make a C library from your Go source then you can do this and use the foreign feature from CL.
Sockets (or other two way communication bus) are good if the Go program is a service that is supposed to provide something. Eg. an application server to serve http requests. Usually if you only need to use the go program once each run of the CL program this isn't the way to go.
Subprocess is best if you can run your application with arguments and get a result that is used in Common Lisp. It's not good if you are going to use the Go program many times as it will have overhead (in which the sockets thing would be best)
Awesome way to do this is to make the whole thing in Common Lisp. If you choose a implementation that has a good compiler and write it well you might get away with having the application as a CL image. If you need to speed up things you can focus on the slow parts and optimize them og you can use CFFI by writing the optimizations in C. There is even a Inline C for SBCL where you can just write C where you want to optimize in CL and you don't need to write the optimizations in their own file and compile and link separately.
Is there an MPI implementation that allows nodes to be dynamically added/removed at runtime? Do any recover from complete hardware failure of a node, allowing the node to be repaired and relaunched without restarting the program?
Is there an MPI implementation that allows nodes to be dynamically added/removed at runtime?
This is actually two questions. Nodes usually can be dynamically added at runtime using calls like MPI_Comm_spawn. As #Hristo pointed out in the comments, you should set the correct info key in Open MPI. It may also be possible in other implementations. As for removing nodes, that's a big area of research at the moment. Most MPI implementations currently have varying levels of success surviving a total node failure. In the current releases of Open MPI, I don't believe there is any support for that sort of failure [citation needed], though there is work to improve that ongoing. In the current version of MPICH, you can pass the flag -disable-auto-cleanup to mpiexec and it will not automatically clean up your application after a process/node failure. However, you'll still have to modify your MPI application to handle this situation. The various derivatives of MPICH (Intel MPI, Cray MPI, IBM MPI, MVAPICH, etc.) all don't support this feature AFAIK. There are other research implementations that are also available to extend the support of the MPI Standard. User Level Failure Mitigation is currently being considered by the standardization body as a way of letting the user handle process failures. There is a research implementation based on Open MPI available at the website linked, and an experimental prototype will also be in the next version of MPICH (3.2).
Do any recover from complete hardware failure of a node, allowing the node to be repaired and relaunched without restarting the program?
This is essentially the same process as above. You would need to use the APIs to remove a process and then somehow find out that it's available and add it back using spawn. These calls have to be made from inside the application though, not externally.
Suppose I have two or more different server applications developed in Clojure using ZeroMQ and BSON as protocols. How can I deploy them using a single JVM instance while also sharing common dependencies?
It seems a waste of memory to use a JVM instance for each standalone application. I plan to develop several Clojure applications in the future and VPS memory is not cheap.
Although not explicitly said, applications running in an application server (Jetty, Glassfish) seem to share the same JVM while isolating their state. However, they require a container and neither Servlets or Enterprise JavaBeans have an implementation that I could easily adapt to my custom protocol.
I've been thinking about using Servlets and implementing a dummy service() method though I don't like the idea of having a pointless HTTP server overhead. As for the EJB container, I cannot even figure out its implementation.
It would be nice to have a container requiring only init() and destroy() methods but I can't find an application server providing it.
Maybe there is a way around or I don't even need an application server. Could somebody point me in the right direction?
It sounds like you would be okay using an EJB container, but only if it were easier or simpler to work with. Have you looked at Immutant? It's basically a wrapper around JBossAS for Clojure, written by guys at Red Hat (who also own JBossAS).
In addition to being an application server, those guys have wrapped JMS and other Java-EE features around Clojure, such that sending messages between apps appears pretty simple:
Also, they have Daemons and Jobs, which may provide something similar to what you were describing as simple services with init() and destroy().
That being said, I haven't used it, so I'm can't vouch for it's awesomenss/awfulness.
So you have two applications that both share the same dependencies and both want to respond to and/or generate events on a message bus.
If I understand what you're saying, this should be as simple as starting the JVM with access to all code in the classpath and initializing your message bus and your code from a main method.
If you wanted to use a container, you could create some dummy message driven beans that sat between your clojure code and the message bus assuming there is a JMS adapter for your message bus. Using netbeans/glassfish, this may not be that bad. You might gain some in terms of monitoring, but I'm not sure what else you would gain.
I kept searching and found out that some application servers implementing the OSGi service platform have simpler lightweight containers than those offered by Java EE.
Apache Karaf for instance can load POJO applications directly from JAR files.
I am not sure what DDs are but any JAR is a valid bundle. Since clojure is not type safe you will need to bridge the clojure world and OSGi/Java world but the OSGi API is a wet dream for such bridges.
Not that in OSGi bundle do not automatically provide their content, in OSGi a bundle is by default private. However, the API allows you to punch holes where ever you want.
I heard that when a request goes from browser(client) to IIS ,after extension filtering
(aspnet_isapi.dll)several named pipe connections are established between the ISAPI DLL and the worker process(w3wp.exe).
What is the name of those pipes ? will those pipe acts as a communication channel like the one we have with WCF?
You will find here a superb explanation by Rick Strahl of how ASP.NET works, and yes, named pipes are used in IIS5 for communication between the ISAPI DLL in the inetinfo process and the worker process, but in IIS6 this is no longer necessary since the lowest level of the HTTP stack was transferred to the kernel driver HTTP.SYS, which passes the requests directly to the worker process.
Named pipes are objects managed by the operating system kernel, for which there is a specific Win32 API. WCF named pipe bindings are built on top of these, but involve a great deal more layered on top of the raw pipe transport. Even in IIS5 where named pipes are used for ASP.NET, these are not used in anything like the same way that WCF uses them, so there is no reason to think of them as connected or analogous in any way. The types in the System.IO.Pipes namespace are a nearer comparison, being much thinner wrappers over the OS pipe API.
This is just a binary pipe, one of standard ways to communicate between processes in windows (the others are shared memory and com+ iirc). You can have several worker processes, obviously, so i am not sure there's one single name for the pipe. And i highly doubt this is using any kind of .net serialized data - not sure about this.
I'm building a Qt application that needs to use libssh, a SSH client library. libssh (understandably) performs its own network connections, however Qt has its own infrastructure for network connections (QTcpSocket etc).
Should I worry about these differences? Should I be trying to make libssh make network connections via QTcpSocket... Or if it works fine on the platforms I'm targeting, is that good enough?
The only downside is that you have another library that your code depends on.
The primary rule though is if it works, go with it.
I think it depends on how the abstraction you get from libssh looks like. If it is a socket-like API, you could create an QAbstractSocket implementation for it. If it is just some structure or handle to read from and write to, you could create a QIODevice subclass. Most I/O can be implemented generically operating on QIODevices (instead of explicitely operating on QFile, sockets, etc.).