I have to connect an old but critical software to RabbitMQ. The software doesn't support AMQP, but it can do HTTP Requests.
Does RabbitMQ support plain HTTP? Or should I use a "proxy" or "app" that actively transforms the HTTP Requests to AMQP 1.0 and pushes it to the RabbitMQ server?
https://www.rabbitmq.com/management.html
The management plugin supports a simple HTTP API to send and receive messages. This is primarily intended for diagnostic purposes but can be used for low volume messaging without reliable delivery.
As mentioned, it's designed for very low loads, but it may be usable. If you need higher loads, then by all means cast around for a library that does the job and create a proxy. Most languages will have something. I've personally created a lightweight API using Lumen and https://github.com/bschmitt/laravel-amqp to tie a few disparate services together in the past, and it seems to work very well.
It is possible not but really recommended depending on load. You have three options really, two of which are web socket based and one that seems like what you're looking for. I'd suggest starting with the rabbitmq docs.
Related
I want to create a web app using React as the front end technology. A requirement for the app is that the server will be able to update all the clients with information about changes (not have to be an exact real time, but should update after no more than 10 seconds).
Solutions like clients requesting updates from the server every several seconds are out of the question.
Requirements:
1) The server's should be implemented with either .NET or with Node.js.
2) The connection MUST be secured via port 443 of the IIS.
I read a bit about Micorsoft's SignalR and about Pusher Channels which seems to provide exactly the kind of service I require.
Could you please elaborate about what exactly are the differences between them? When should I choose each? Which of them got more community support? Which is easier to implement? Stuff like that...
Both SignalR and Pusher Channels ultimately both use websockets to deliver messages to clients, so both should meet your requirements to deliver messages to clients in realtime.
1) Both offerings also meet your requirements for both library support:
SignalR supports .NET:
https://dotnet.microsoft.com/apps/aspnet/signalr
Pusher Channels has server support for both nodejs and .NET:
https://github.com/pusher/pusher-http-node
https://github.com/pusher/pusher-http-dotnet
2) Both offerings also meet your requirements for sending messages over TLS/WSS:
SignalR:
https://kimsereyblog.blogspot.com/2018/07/signalr-with-asp-net-core.html
Pusher Channels:
Securing Pusher's messages
In terms of the differences between them this depends on your implementation, if you just run SignalR on your own ISS server then it will be down to you to manage all of the websocket connections and all of the scaling challenges that come with this.
However similar to how Channels works, SignalR also has a managed websocket service, so you do not need to manage the connections or scaling. You just make an API request with the message you want to send to either Channels or SignalR and this message is then broadcast to the interested clients connected by websockets. In this scenario you do not manage the websocket connections yourself.
However in terms of pricing Channels appears to be far more competitive (especially the free offering), so if you are looking at the managed offering Channels looks to be a better value proposition:
https://azure.microsoft.com/en-gb/pricing/details/signalr-service/
https://pusher.com/channels/pricing
Both offerings look fairly similar in terms of implementation (assuming you are using the managed service). The complexity would increase if you implement SignalR on ISS:
https://learn.microsoft.com/en-us/aspnet/core/signalr/scale?view=aspnetcore-2.2
In terms of support Pusher has a free application support offering:
https://support.pusher.com/hc/en-us
Hope this helps!
This presentation has some answers A 10 Minute Guide to Choosing a Realtime Framework
Is it possible to have RabbitMQ Consumer listening to a queue for message via AMQP protocol. I am aware that nginx only supports HTTP/s protocol. Was wondering if this can be achieved by using tcp module extension.
I am using nginx as API Gateway and want to do a protocol translation from AMQP to HTTP since all the backend service's are exposed on HTTP.
It would definitely be possible writing your own C extension. nginx is suitable for TCP proxying, therefore I don't see any reason why you couldn't send your own TCP packets to RabbitMQ using nginx, and consequently use nginx as a RabbitMQ consumer. It's probably a lot of work to make it run, and even more work to make it stable and reliable, but doable. Do me a favor though, don't do this. There will always be better, more elegant and simpler solution.
HTTP is definitely not suitable for consuming from a queue (in the amqp sense) because you have to keep the socket open while you consume. However, you could write a C extension to publish/retrieve messages to/from RabbitMQ (and apparently, somebody has already done this). If you're not that much into C or don't want to maintain your own nginx package, you could also write a LUA extension for lua-nginx-module (once again, somebody seems to have worked in this direction). These are PoC for talking to MQ from nginx, but they are not consumers. Both extensions seems to act in the HTTP context, so you need to answer (and close the socket) pretty fast.
However, as far as I know, there isn't any community-driven and well maintained project that would serve this purpose directly or indirectly; you'd have to make and maintain your own extension/client. Moreover, nginx is your current API gateway. Do take the risk into account. Things could go really wrong. Only you can tell whether it is worth the hassle or not, but it's most likely not.
Since you don't gave that much information on what you're exactly looking for, I just answered you on the NGINX/AMQP part. But you might just be looking for an HTTP interface for RabbitMQ. In this case, the Management Plugin might be the way to go. It has a pretty cool HTTP API. Once again, you'd loose every stateful features (like basic consuming, ack/nack/rejects), but that's inherently due to the way HTTP is designed.
Eventually, if you really need a RabbitMQ "basic-"consumer, I would recommend you to write a proper consumer as a separate application and forget about doing this in nginx. That's definitely the best and most supported solution.
It's seems to me that SignalR is only temporary step towards global domination of System.Web.WebSocket, and it's lower level System.Net.WebSocket.
If I have IIS8, and my clients have IE10, do I have any reason to use signalR?
Does it have a future?
(Of course, the same goes for Socket.IO + Node.js)
Thanks
ref:
http://pieterderycke.wordpress.com/2012/07/20/websockets-vs-signalr-or-why-you-should-not-have-to-care/
A number of advantages:
SignalR abstracts the actual connection type away so that you only deal with a logical connection. The advantage is that you can switch to other connection types if you want to without having to change your code (SSE might actually provide better performance than WebSockets in some cases).
You get the fallback options (long-polling etc.) for free in case you need to connect from a client that doesn't support WebSockets.
Hubs provide a level of organization (you can do that yourself, of course, but it's a good starting point, and it's convenient)
SignalR provides a rich API for calling a specific client, a group of clients, all clients (including the ability to exclude certain clients). Again, you can implement it yourself, so this is mostly about convenience.
You can pass strongly typed parameters between client and server (both ways).
You'll (probably) have to deal with less boilerplate code with SignalR.
Scale-out support
Off the top of my head, the group membership/broadcast and the ability to scale out on a webfarm are features that you don't get built into the WebSockets classes - so if you're using those features, you'll probably continue to use SignalR for the near future.
Additional to Damien's points:
SignalR also supports long polling and some other techniques if the user has no browser with web socket support
It's very easy to call methods on the client from the server
Also, when WebSockets is updated, SignalR will also be updated, so you won't need to worry about coding for the new version of WebSockets.
A little background.
Very big monolithic Django application. All components use the same database. We need to separate services so we can independently upgrade some parts of the system without affecting the rest.
We use RabbitMQ as a broker to Celery.
Right now we have two options:
HTTP Services using a REST interface.
JSONRPC over AMQP to a event loop service
My team is leaning towards HTTP because that's what they are familiar with but I think the advantages of using RPC over AMQP far outweigh it.
AMQP provides us with the capabilities to easily add in load balancing, and high availability, with guaranteed message deliveries.
Whereas with HTTP we have to create client HTTP wrappers to work with the REST interfaces, we have to put in a load balancer and set up that infrastructure in order to have HA etc.
With AMQP I can just spawn another instance of the service, it will connect to the same queue as the other instances and bam, HA and load balancing.
Am I missing something with my thoughts on AMQP?
At first,
REST, RPC - architecture patterns, AMQP - wire-level and HTTP - application protocol which run on top of TCP/IP
AMQP is a specific protocol when HTTP - general-purpose protocol, thus, HTTP has damn high overhead comparing to AMQP
AMQP nature is asynchronous where HTTP nature is synchronous
both REST and RPC use data serialization, which format is up to you and it depends of infrastructure. If you are using python everywhere I think you can use python native serialization - pickle which should be faster than JSON or any other formats.
both HTTP+REST and AMQP+RPC can run in heterogeneous and/or distributed environment
So if you are choosing what to use: HTTP+REST or AMQP+RPC, the answer is really subject of infrastructure complexity and resource usage. Without any specific requirements both solution will work fine, but i would rather make some abstraction to be able switch between them transparently.
You told that your team familiar with HTTP but not with AMQP. If development time is an important time you got an answer.
If you want to build HA infrastructure with minimal complexity I guess AMQP protocol is what you want.
I had an experience with both of them and advantages of RESTful services are:
they well-mapped on web interface
people are familiar with them
easy to debug (due to general purpose of HTTP)
easy provide API to third-party services.
Advantages of AMQP-based solution:
damn fast
flexible
cost-effective (in resources usage meaning)
Note, that you can provide RESTful API to third-party services on top of your AMQP-based API while REST is not a protocol but rather paradigm, but you should think about it building your AQMP RPC api. I have done it in this way to provide API to external third-party services and provide access to API on those part of infrastructure which run on old codebase or where it is not possible to add AMQP support.
If I am right your question is about how to better organize communication between different parts of your software, not how to provide an API to end-users.
If you have a high-load project RabbitMQ is damn good piece of software and you can easily add any number of workers which run on different machines. Also it has mirroring and clustering out of the box. And one more thing, RabbitMQ is build on top of Erlang OTP, which is high-reliable,stable platform ... (bla-bla-bla), it is good not only for marketing but for engineers too. I had an issue with RabbitMQ only once when nginx logs took all disc space on the same partition where RabbitMQ run.
UPD (May 2018):
Saurabh Bhoomkar posted a link to the MQ vs. HTTP article written by Arnold Shoon on June 7th, 2012, here's a copy of it:
I was going through my old files and came across my notes on MQ and thought I’d share some reasons to use MQ vs. HTTP:
If your consumer processes at a fixed rate (i.e. can’t handle floods to the HTTP server [bursts]) then using MQ provides the flexibility for the service to buffer the other requests vs. bogging it down.
Time independent processing and messaging exchange patterns — if the thread is performing a fire-and-forget, then MQ is better suited for that pattern vs. HTTP.
Long-lived processes are better suited for MQ as you can send a request and have a seperate thread listening for responses (note WS-Addressing allows HTTP to process in this manner but requires both endpoints to support that capability).
Loose coupling where one process can continue to do work even if the other process is not available vs. HTTP having to retry.
Request prioritization where more important messages can jump to the front of the queue.
XA transactions – MQ is fully XA compliant – HTTP is not.
Fault tolerance – MQ messages survive server or network failures – HTTP does not.
MQ provides for ‘assured’ delivery of messages once and only once, http does not.
MQ provides the ability to do message segmentation and message grouping for large messages – HTTP does not have that ability as it treats each transaction seperately.
MQ provides a pub/sub interface where-as HTTP is point-to-point.
UPD (Dec 2018):
As noticed by #Kevin in comments below, it's questionable that RabbitMQ scales better then RESTful servies. My original answer was based on simply adding more workers, which is just a part of scaling and as long as single AMQP broker capacity not exceeded, it is true, though after that it requires more advanced techniques like Highly Available (Mirrored) Queues which makes both HTTP and AMQP-based services have some non-trivial complexity to scale at infrastructure level.
After careful thinking I also removed that maintaining AMQP broker (RabbitMQ) is simpler than any HTTP server: original answer was written in Jun 2013 and a lot of changed since that time, but the main change was that I get more insight in both of approaches, so the best I can say now that "your mileage may vary".
Also note, that comparing both HTTP and AMQP is apple to oranges to some extent, so please, do not interpret this answer as the ultimate guidance to base your decision on but rather take it as one of sources or as a reference for your further researches to find out what exact solution will match your particular case.
The irony of the solution OP had to accept is, AMQP or other MQ solutions are often used to insulate callers from the inherent unreliability of HTTP-only services -- to provide some level of timeout & retry logic and message persistence so the caller doesn't have to implement its own HTTP insulation code. A very thin HTTP gateway or adapter layer over a reliable AMQP core, with option to go straight to AMQP using a more reliable client protocol like JSONRPC would often be the best solution for this scenario.
Your thoughts on AMQP are spot on!
Furthermore, since you are transitioning from a monolithic to a more distributed architecture, then adopting AMQP for communication between the services is more ideal for your use case. Here is why…
Communication via a REST interface and by extension HTTP is synchronous in nature — this synchronous nature of HTTP makes it a not-so-great option as the pattern of communication in a distributed architecture like the one you talk about. Why?
Imagine you have two services, service A and service B in that your Django application that communicate via REST API calls. This API calls usually play out this way: service A makes an http request to service B, waits idly for the response, and only proceeds to the next task after getting a response from service B. In essence, service A is blocked until it receives a response from service B.
This is problematic because one of the goals with microservices is to build small autonomous services that would always be available even if one or more services are down– No single point of failure. The fact that service A connects directly to service B and in fact, waits for some response, introduces a level of coupling that detracts from the intended autonomy of each service.
AMQP on the other hand is asynchronous in nature — this asynchronous nature of AMQP makes it great for use in your scenario and other like it.
If you go down the AMQP route, instead of service A making requests to service B directly, you can introduce an AMQP based MQ between these two services. Service A will add requests to the Message Queue. Service B then picks up the request and processes it at its own pace.
This approach decouples the two services and, by extension, makes them autonomous. This is true because:
If service B fails unexpectedly, service A will keep accepting requests and adding them to the queue as though nothing happened. The requests would always be in the queue for service B to process them when it’s back online.
If service A experiences a spike in traffic, service B won’t even notice because it only picks up requests from the Message Queues at its own pace
This approach also has the added benefit of being easy to scale— you can add more queues or create copies of service B to process more requests.
Lastly, service A does not have to wait for a response from service B, the end users don’t also have to wait for long— this leads to improved performance and, by extension, a better user experience.
Just in case you are considering moving from HTTP to AMQP in your distributed architecture and you are just not sure how to go about it, you can checkout this 7 parts beginner guide on message queues and microservices. It shows you how to use a message queue in a distributed architecture by walking you through a demo project.
I am searching for a good method to transfer data over internet, and I work in C++/windows environment. The data is binary, a compressed blob of an extracted image. Input and requirements are as follows:
6kB/packet # 10 packets/sec (60kBytes per second)
Reliable data transfer
I am new to network programming and so far I could figure out that one of the following methods will be suitable.
Sockets
MSMQ (MS Message Queuing)
The Client runs on a browser (Shows realtime images on browser). While server runs native C++ code. Please let me know if there are any other methods for achieving the same? Which one should I go for and why?
If the server determines the pace at which images are sent, which is what it looks like, a server push style solution would make sense. What most browsers (and even non-browsers) are settling for these days are WebSockets.
The main advantage WebSockets have over most proprietary protocols, apart from becoming a widely adopted standard, is that they run on top of HTTP and can thus permeate (most) proxies and firewalls etc.
On the server side, you could potentially integrate node.js, which allows you to easily implement WebSockets, and comes with a lot of other libraries. It's written in C++, and extensible via C++ and JavaScript, which node.js hosts a VM for. node.js's main feature is being asynchronous at every level, making that style of programming the default.
But of course there are other ways to implement WebSockets on the server side, maybe node.js is more than you need. I have implemented a C++ extension for node.js on Windows and use socket.io to do WebSockets and non-WebSocket transports for older browsers, and that has worked out fine for me.
But that was textual data. In your binary data case, socket.io wouldn't do it, so you could check out other libraries that do binary over WebSockets.
Is there any specific reason why you cannot run a server on your windows machine? 60kb/seconds, looks like some kind of an embedded device?
Based on our description, you ned to show image information, in realtime on a browser. You can possibly use HTTP. but its stateless, meaning once the information is transferred, you lose the connection. You client needs to poll the C++/Windows machine. If you are prety confident the information generated is periodic, you can use this approach. This requires a server, so only if a yes to my first question
A chat protocol. Something like a Jabber client running on your client, and a Jabber server on your C++/Windows machine. Chat protocols allow almost realtime
While it may seem to make sense, I wouldn't use MSMQ in this scenario. You may not run into a problem now, but MSMQ messages are limited in size and you may eventually hit a wall because of this.
I would use TCP for this application, TCP is built with reliability in mind and you can simply feed data through a socket. You may have to figure out a very simple protocol yourself but it should be the best choice.
Unless you are using an embedded device that understands MSMQ out of the box, your best bet to use MSMQ would be to use a proxy and you are then still forced to play with TCP and possibly HTTP.
I do home automation that includes security cameras on my personal time and I use the .net micro framework and even if it did have MSMQ capabilities I still wouldn't use it.
I recommend that you look into MJPEG (Motion JPEG) which sounds exactly like what you would like to do.
http://www.codeproject.com/Articles/371955/Motion-JPEG-Streaming-Server