Is it possible to have RabbitMQ Consumer listening to a queue for message via AMQP protocol. I am aware that nginx only supports HTTP/s protocol. Was wondering if this can be achieved by using tcp module extension.
I am using nginx as API Gateway and want to do a protocol translation from AMQP to HTTP since all the backend service's are exposed on HTTP.
It would definitely be possible writing your own C extension. nginx is suitable for TCP proxying, therefore I don't see any reason why you couldn't send your own TCP packets to RabbitMQ using nginx, and consequently use nginx as a RabbitMQ consumer. It's probably a lot of work to make it run, and even more work to make it stable and reliable, but doable. Do me a favor though, don't do this. There will always be better, more elegant and simpler solution.
HTTP is definitely not suitable for consuming from a queue (in the amqp sense) because you have to keep the socket open while you consume. However, you could write a C extension to publish/retrieve messages to/from RabbitMQ (and apparently, somebody has already done this). If you're not that much into C or don't want to maintain your own nginx package, you could also write a LUA extension for lua-nginx-module (once again, somebody seems to have worked in this direction). These are PoC for talking to MQ from nginx, but they are not consumers. Both extensions seems to act in the HTTP context, so you need to answer (and close the socket) pretty fast.
However, as far as I know, there isn't any community-driven and well maintained project that would serve this purpose directly or indirectly; you'd have to make and maintain your own extension/client. Moreover, nginx is your current API gateway. Do take the risk into account. Things could go really wrong. Only you can tell whether it is worth the hassle or not, but it's most likely not.
Since you don't gave that much information on what you're exactly looking for, I just answered you on the NGINX/AMQP part. But you might just be looking for an HTTP interface for RabbitMQ. In this case, the Management Plugin might be the way to go. It has a pretty cool HTTP API. Once again, you'd loose every stateful features (like basic consuming, ack/nack/rejects), but that's inherently due to the way HTTP is designed.
Eventually, if you really need a RabbitMQ "basic-"consumer, I would recommend you to write a proper consumer as a separate application and forget about doing this in nginx. That's definitely the best and most supported solution.
Related
I have to connect an old but critical software to RabbitMQ. The software doesn't support AMQP, but it can do HTTP Requests.
Does RabbitMQ support plain HTTP? Or should I use a "proxy" or "app" that actively transforms the HTTP Requests to AMQP 1.0 and pushes it to the RabbitMQ server?
https://www.rabbitmq.com/management.html
The management plugin supports a simple HTTP API to send and receive messages. This is primarily intended for diagnostic purposes but can be used for low volume messaging without reliable delivery.
As mentioned, it's designed for very low loads, but it may be usable. If you need higher loads, then by all means cast around for a library that does the job and create a proxy. Most languages will have something. I've personally created a lightweight API using Lumen and https://github.com/bschmitt/laravel-amqp to tie a few disparate services together in the past, and it seems to work very well.
It is possible not but really recommended depending on load. You have three options really, two of which are web socket based and one that seems like what you're looking for. I'd suggest starting with the rabbitmq docs.
I saw some performance of GWAN and interested in testing it as a reverse proxy of static content in front of Apache with APC for optimizing PHP opcode, to run a Wordpress multisite. I can get GWAN up and running but I have no idea how to configure it for reverse proxy, as there seems to be almost no information on it. Anyone use GWAN as a reverse proxy?
It's still a not documented feature ... maybe in a next release ? #gil ?
Right now there's no easy way for you to do that. That will change with the next release.
We first hardcoded the reverse-proxy feature in G-WAN along with the load-balancer. Then, as we needed to personalize reverse-proxying, we implemented it as a protocol handler script.
Protocol handler scripts allow users to implement any protocol (like SMTP, LDAP, etc.) without haveing to deal with multi-threading nor socket events.
But finally, to reduce complexity for users, we might revert to the hard-coded implementation with connection handlers scripts to let people personalize the reverse proxy.
It's maturing under different use cases, hence the delay in publicly releasing this feature and a few others.
Rushing to implement features and interfaces is not always optimal, if the goal is to stay flexible and easy to use.
I am developing a group chat using the python Twisted framework. The technique I am using is Long polling with Ajax. I am returning SERVER_NOT_DONE_YET to keep the connection open. The code is non-blocking and allows other requests. How much scalable is it ??
However, I want to move ahead of this streaming over open connections. I want to implement a pure server push. How to do it ? Do I need to go in the direction of XMPP ? If I open a socket on the server for each unique client, which web server would best suit the bridging ? How much scalable would it be ?
I want it to be as much scalable as the C10K problem.I would like to stick to Twisted because it has a lot of protocol implementations in easy steps. Please point me in the right direction. Thanx
Long-polling works, but isn't necessarily your best option. It starts getting really nasty in terms of integration with firewalls and flaky internet connections. For example, at work, a lot of our customers' firewalls kill off any HTTP connection that isn't active for 10-20 seconds.
We've solved a lot of problems by switching over to WebSocket over SSL. WebSocket gives you a full-duplex channel, which is perfect for server push. By using SSL, firewalls are often less aggressive in their garbage collecting, and transparent proxies are often fooled by the TLS encryption. You will still need to manage the occasional disconnection on an application-level, even if you're using WebSockets instead of long-polling, but even that can be handled gracefully by having a decent recovery protocol, regardless of whatever transport protocol you use.
This being said, instead of going directly for WebSockets, we've decided to use SockJS. The main reason for this choice was that SockJS can use WebSockets when available (rfc6455, hixie-76, hybi-10), but also fall back to xhr-streaming, xdr-streaming, etc, if the client's browser does not support it (or if the connection fails). When I say that it can "fall back", I mean that the code you use on the client side remains exactly the same, SockJS takes care of the dirty work.
On the server side, the same is true. We currently use Cyclone's SockJS implementation for Twisted (in production), but we're also aware of DesertBus' implementation, which we still have to check out. There's also some other stuff that we're hoping to check out, for example WAMP, and the accompanying Autobahn|Python.
With regards to performance, we use HAProxy for SSL termination and load-balancing. HAProxy's performance is pretty amazing, on a multitude of levels.
We have migrated to WebSockets now. It works perfectly fine !!
I am searching for a good method to transfer data over internet, and I work in C++/windows environment. The data is binary, a compressed blob of an extracted image. Input and requirements are as follows:
6kB/packet # 10 packets/sec (60kBytes per second)
Reliable data transfer
I am new to network programming and so far I could figure out that one of the following methods will be suitable.
Sockets
MSMQ (MS Message Queuing)
The Client runs on a browser (Shows realtime images on browser). While server runs native C++ code. Please let me know if there are any other methods for achieving the same? Which one should I go for and why?
If the server determines the pace at which images are sent, which is what it looks like, a server push style solution would make sense. What most browsers (and even non-browsers) are settling for these days are WebSockets.
The main advantage WebSockets have over most proprietary protocols, apart from becoming a widely adopted standard, is that they run on top of HTTP and can thus permeate (most) proxies and firewalls etc.
On the server side, you could potentially integrate node.js, which allows you to easily implement WebSockets, and comes with a lot of other libraries. It's written in C++, and extensible via C++ and JavaScript, which node.js hosts a VM for. node.js's main feature is being asynchronous at every level, making that style of programming the default.
But of course there are other ways to implement WebSockets on the server side, maybe node.js is more than you need. I have implemented a C++ extension for node.js on Windows and use socket.io to do WebSockets and non-WebSocket transports for older browsers, and that has worked out fine for me.
But that was textual data. In your binary data case, socket.io wouldn't do it, so you could check out other libraries that do binary over WebSockets.
Is there any specific reason why you cannot run a server on your windows machine? 60kb/seconds, looks like some kind of an embedded device?
Based on our description, you ned to show image information, in realtime on a browser. You can possibly use HTTP. but its stateless, meaning once the information is transferred, you lose the connection. You client needs to poll the C++/Windows machine. If you are prety confident the information generated is periodic, you can use this approach. This requires a server, so only if a yes to my first question
A chat protocol. Something like a Jabber client running on your client, and a Jabber server on your C++/Windows machine. Chat protocols allow almost realtime
While it may seem to make sense, I wouldn't use MSMQ in this scenario. You may not run into a problem now, but MSMQ messages are limited in size and you may eventually hit a wall because of this.
I would use TCP for this application, TCP is built with reliability in mind and you can simply feed data through a socket. You may have to figure out a very simple protocol yourself but it should be the best choice.
Unless you are using an embedded device that understands MSMQ out of the box, your best bet to use MSMQ would be to use a proxy and you are then still forced to play with TCP and possibly HTTP.
I do home automation that includes security cameras on my personal time and I use the .net micro framework and even if it did have MSMQ capabilities I still wouldn't use it.
I recommend that you look into MJPEG (Motion JPEG) which sounds exactly like what you would like to do.
http://www.codeproject.com/Articles/371955/Motion-JPEG-Streaming-Server
I want to make a solution where I can use XMPP chat clients connected to gtalk, resolving SOAP web services and REST calls meant for a LAN (behind firewall proxy etc) and transferring over to the other chat client on another different LAN (behind other firewall proxy etc).
I have narrowed down on Smack API, but being a newbie in XMPP and smack, I don't know if is it possible or not with Smack?
Does smack provide for such a functionality? I tried Smack documentation but couldn't understand it in enough depth.
Any other possible alternatives, suggestions too are most welcome.
Start with XEP-72 for a hint at the style of protocol you should use. There are some good reasons to do this, including:
Firewall traversal (as noted)
Don't have to do a TCP connection as often, saving latency
Don't have to authenticate for each request, but can authenticate the stream
No need for a separate XML parser for each request, which means processing can be more efficient
To make things easier start with http://xmpp.org/extensions/xep-0050.html and use command's node attribute in a restfull way