so our use case is to have a microservice which is very expensive (takes time to run). this service is consumed by any client.
i've read about some patterns for best practices for consuming, for example:
use WebSockets - which will make the server able to send the result back to the client.
Constant polling (i don't like it much)
Leases - it's some kind of polling - the client will acquire a lease for X minutes, and will renew it every few minutes until a response comes - but if the response comes back, the client will still have to wait for it (although the result is ready already) - this will make cleanup easier (clients which abandoned the requests)
i'd love to hear about your best practices in this event-driven microservices architecture
thanks!
Sounds like you need a pubsub framework/product, e.g. ActiveMQ, RabbitMQ, ZeroMQ (NetMQ) or maybe even Redis pubsub/queues
Related
Say I have a webserivce used internally by other webservices with an average response time of 1 minute.
What are the pros and cons of such a service with "synchronous" responses versus making the service return id of the request, process it in the background and make the clients poll for results?
Is there any cons with HTTP connections which stay active for more than one minute? Does the default keep alive of TCP matters here?
Depending on your application it may matter. Couple of things worth mentioning are !
HTTP protocol is sync
There is very wide misconception that HTTP is async. Http is synchronous protocol but your client could deal it async. E.g. when you call any service using http, your http client may schedule is on the background thread (async). However The http call will be waiting until either it's timeout or response is back , during all this time the http call chain is awaiting synchronously.
Sockets
Since HTTP uses socket and there is hard limit on sockets. Every HTTP connection (if created new every time) opens up new socket . if you have hundreds of requests at a time you can image how many http calls are scheduled synchronously and you may run of sockets. Not sure for other operation system but on windows even if you are done with request sockets they are not disposed straight away and stay for couple of mins.
Network Connectivity
Keeping http connection alive for long is not recommended. What if you loose network partially or completely ? your http request would timeout and you won't know the status at all.
Keeping all these things in mind it's better to schedule long running tasks on background process.
If you keep the user waiting while your long job is running on server, you are tying up a valuable HTTP connection while waiting.
Best practice from RestFul point of view is to reply an HTTP 202 (Accepted) and return a response with the link to poll.
If you want to hang the client while waiting, you should set a request timeout at the client end.
If you've some Firewalls in between, that might drop connections if they are inactive for some time.
Higher Response Throughput
Typically, you would want your OLTP (Web Server) to respond quickly as possible, Since your queuing the task on the background, your web server can handle more requests which results to higher response throughput and processing capabilities.
More Memory Friendly
Queuing long running task on background jobs via messaging queues, prevents abusive usage of web server memory. This is good because it will increase the Out of memory threshold of your application.
More Resilient to Server Crash
If you queue task on the background and something goes wrong, the job can be queued to a dead-letter queue which helps you to ultimately fix problems and re-process the request that caused your unhandled exceptions.
So I came across the idea of blocking and non-blocking I/O. But what I understood from the concept and some of the sample implementations is that we implement code on the server side to achieve this nature of the code.
But now my question is, if (for example postman sending HTTP request to the server) the request has to wait for the server to respond, then what's the point of non-blocking I/O? (Please correct me if I am wrong) Or the whole concept is just for the increase of throughput of the server instead of actual async nature w.r.t. to client.
For example, in one of my project what I did was created a post request to create a request in the system for processing which will return the transaction ID, now using this transaction id, I can query the server to know the outcome.
I may sound too naive, but the concept has confused me a lot. I do not understand this concept clearly. Please help.
Thanks
the request has to wait for the server to respond, then what's the point of non-blocking I/O?
There's a confusion. Waiting for a response and (non)blocking i/o are very loosely related. You always have to wait for response. That's why youve made the request to begin with. But the question is: how?
Non-blocking HTTP: "Dear server, here's my request, please process it and send me a response, I'm going to do something else in the meantime, like calculating n-th digit of Pi (I'm a weirdo)".
Blocking HTTP: "Dear server, here's my request, please process it and send me a response, I'm going to patiently wait for it doing nothing".
Or the whole concept is just for the increase of throughput of the server instead of actual async nature w.r.t. to client.
The whole concept is to be able to do other things while waiting for i/o at the same time. And to do that while minimizing the usage of threads which don't scale well.
Asynchronous systems, i.e. systems without "I'm going to wait idly" part tend to perform better at the cost of complexity.
Side note: nonblocking i/o can be used both on the server side and client side. For example almost all JS engines in browsers are built on top of some asynchronous engine. JS is often single-threaded, meaning nonblocking i/o is necessary to achieve any concurrency.
But what I understood from the concept and some of the sample implementations is that we implement code on the server side to achieve this nature of the code.
You implement code in whereever you are doing the non-blocking UI. What a server does has no bearing on whether a client uses blocking or non-blocking UI, and what a client does has no bearing on whether a server uses blocking or non-blocking UI.
if (for example postman sending HTTP request to the server) the request has to wait for the server to respond, then what's the point of non-blocking I/O?
So that you're not wasting resources.
Let's consider first a simple console application that hits the web and then does something with the results. In this case there's very little to gain with non-blocking I/O as the application is just going to be sitting around waiting for something to do anyway.
Now let's consider a simple console application that hits 50 different web resources and collates the responses. Now non-blocking I/O is more useful, because with blocking I/O it would have to either get one resource after the other, or spin up 50 threads. With non-blocking I/O one, a small number of threads is all that is needed to hit 50 resources and respond promptly to each returning a response.
Now let's consider a GUI version of this application that wants to remain responsive to user input, while also running on low-power low-memory devices in which blocked threads are all the more expensive. The advantages of the above are increased.
Finally, consider a web application that is doing I/O both with the client and also as a client to a database, file system and maybe other web applications. It may have multiple requests at the same time, and blocking on either the I/O it does with the client or any of the I/O it does with db, file or other applications would cost a thread, which would put a scalability limit on how many requests it can handle simultaneously. Not blocking on I/O allows threads to be used for other requests while the I/O is pending.
While looking for zookeeper, the accepted answer says that concurrent writes are not allowed.
Explaining Apache ZooKeeper
Now my question is as Zookeeper has linear writes, that does not stop me to use Asynchronous APIs to create nodes and take the response in a callback ? Though internally it may not allow concurrent writes , or am I missing something ?
Even though zookeeper operates in an ensemble, writes are always served through the leader. Therefore, leader is capable of queuing write requests and completing them sequentially.
Using the asynchronous API will not do any harm to the above mentioned approach. Even though the write requests are asynchronous (from the client side), leader will always make sure that they are served sequentially. Once a asynchronous write request is served, client will be notified through the callback. It is simple as that. Remember, the requests are asynchronous as viewed by the client. But from the leader's point of view, they are served sequentially.
We have a requirement to to support 10k+ users, where every user initiate a request and waits for a response from the server (the response can take as long as 20-30 seconds to arrive). it is only one request from the client, and after a long processing by the server, a response will be transmitted and then the connection will disconnect.
in the background, the server will do some DB search and wait for other background processes to notify on completion before responding to the client.
after doing some research i figured out we will need to use something like the atmosphere framework to support websockets/sse event/long polling along with an asynchronous server like netty (=> nettosphere) or jetty.
As for my experience - mostly Java EE world and Tomcat server.
my questions are:
what will be easier to implement in regard to my experience and our requirement: atmosphere + netty or atmoshphere+jetty? which one can scale better, has an easier learning curve and easier to implement other java technologies?
how do u implement in atmosphere a response that is sent only to the originating client and not broadcast to the rest of the clients? (all the examples i found are broadcast).
how can i implement in netty (or jetty) when using the atmosphere framework our response? i.e., the client send a request, after it is received in the server some background processes are run, and when they finish i need to locate the connection and transmit the response. is that achievable?
Some thoughts:
At 10k+ users, with 20-30 second response latency, you likely hit file descriptor limits if using just 1 network interface. Consider a solution that uses multiple network interfaces.
Your description of your request/response can be handled entirely with standard Servlet 3.0, standard HTTP/1.1, Async request handling, and large timeouts.
If your clients are web browsers, and you don't start sending a response from the server until the 20-30 second window, you might hit browser idle timeouts.
Atmosphere and Cometd do the same things, supporting long duration connections, with connection technique fallbacks, and with logical channel APIs.
I believe the AKKA framework will handle this sort of need. I am looking at using it to handle scaling issues possibly with a RabbitMQ to help off load work to potentially other servers that may be added later to scale as needed.
What's a good way to connect the synchronous http request/response model with an asynchronous queue based model?
When the user's HTTP request comes it generates a work request that goes onto a queue (beanstalkd in this case). One of the workers picks up the request, does the work, and prepares a response.
The queue model is not request/response - there are only requests, not responses. So the question is, how best do we get the response back into the world of HTTP and back to the user?
Ideas:
Beanstalkd supports light weight topics or queues (they call them tubes). We could create a tube for each request, have the worker create a message on that tube, and have the http process sit and wait on the tube for the response. Don't particularly like this one since it has apache processes sitting around taking memory.
Have the http client poll for the response. The user's initial HTTP request kicks off the job on the queue and returns immediately. The client (the user's browser) polls periodically for a response. On the backend the worker puts its response into memcached, and we connect nginx to memcached so the polling is light weight.
Use Comet. Similar to the second option, but with fancier http communication to avoid polling.
I'm leaning towards 2 since it's easy and well know (I haven't used comet yet). I'm guessing there's probably also a much better obvious model I haven't thought of. What do you think?
Here's how to implement request-response efficiently on JMS which might be helpful (though Java/JMS centric). The general idea is to create a temporary queue per client/thread then use correlationIDs to correlate requests to replies etc.
Polling is the simple solution; comet is the more efficient solution. You've got it nailed :)
I personally love comet (although I'm biased, since I helped write WebSync), it nicely lets your clients subscribe to a channel and get the message when your server process is ready. Works like a champ.
I'm looking to implement a Beanstalkd and memcached system to run a number of processes following a request - in this case, looking up information when a user logs in (the number of messages a user has waiting for example). The info is stored in Memcached and then read back on the next page load.
Without knowing a little more about what tasks you are doing though, it's not so easy to say what needs to be done, or how. Option #2 is however the simplest, and that may be all you need - depending on what you are pushing back into the workers.