In servlet 3.0 one can use startAsync to put long work in another thread, so that you can free up servlet thread.
Seems that I'm missing something, because I don't see, why not just to use servlet thread for working? Is the thread created by startAsync somehow cheaper?
In most situations when handling requests you are blocking or waiting on some external resource/condition. In this case you are occupying the thread (hence a lot of memory) without doing any work.
With servlet 3.0 you can serve thousands of concurrent connections, much more than available threads. Think about an application that provides downloading of files with limited throughput. Most of the time your threads are idle because they are waiting to send next chunk of data. In ordinary servlets you cannot serve more clients than the number of your HTTP threads, even though most of the time these threads are idle/sleeping.
In servlet 3.0 you can have thousands of connected clients with few HTTP threads. You can find a real world example in my article: Tenfold increase in server throughput with Servlet 3.0 asynchronous processing inspired by this question: Restrict download file bandwidth/speed in Servlet
Is the thread created by startAsync somehow cheaper?
There is no thread created by startAsync! It just tells the servlet container: hey, although the doGet/doPost method finished, I am not done with this request, please do not close. That's the whole point - you probably won't create new thread per each async request. Here is another example - you have thousands of browsers waiting for a stock price change using comet. In standard servlets this would mean: thousands of idle threads waiting for some event.
With servlet 3.0 you can just keep all asynchronous requests waiting in an ArrayList or a some queue. When the stock price change arrives, send it to all clients one after another. Not more than one thread is needed in this scenario - and all HTTP threads are free to process remaining resources.
With servlet 3.0 you can just keep all asynchronous requests waiting in an ArrayList or a some queue
Problem is this. You still need a new thread to process the request and pick up the request to finally send the response.
So we free up http threads but have to create some thread to process the request
Related
Say I have a webserivce used internally by other webservices with an average response time of 1 minute.
What are the pros and cons of such a service with "synchronous" responses versus making the service return id of the request, process it in the background and make the clients poll for results?
Is there any cons with HTTP connections which stay active for more than one minute? Does the default keep alive of TCP matters here?
Depending on your application it may matter. Couple of things worth mentioning are !
HTTP protocol is sync
There is very wide misconception that HTTP is async. Http is synchronous protocol but your client could deal it async. E.g. when you call any service using http, your http client may schedule is on the background thread (async). However The http call will be waiting until either it's timeout or response is back , during all this time the http call chain is awaiting synchronously.
Sockets
Since HTTP uses socket and there is hard limit on sockets. Every HTTP connection (if created new every time) opens up new socket . if you have hundreds of requests at a time you can image how many http calls are scheduled synchronously and you may run of sockets. Not sure for other operation system but on windows even if you are done with request sockets they are not disposed straight away and stay for couple of mins.
Network Connectivity
Keeping http connection alive for long is not recommended. What if you loose network partially or completely ? your http request would timeout and you won't know the status at all.
Keeping all these things in mind it's better to schedule long running tasks on background process.
If you keep the user waiting while your long job is running on server, you are tying up a valuable HTTP connection while waiting.
Best practice from RestFul point of view is to reply an HTTP 202 (Accepted) and return a response with the link to poll.
If you want to hang the client while waiting, you should set a request timeout at the client end.
If you've some Firewalls in between, that might drop connections if they are inactive for some time.
Higher Response Throughput
Typically, you would want your OLTP (Web Server) to respond quickly as possible, Since your queuing the task on the background, your web server can handle more requests which results to higher response throughput and processing capabilities.
More Memory Friendly
Queuing long running task on background jobs via messaging queues, prevents abusive usage of web server memory. This is good because it will increase the Out of memory threshold of your application.
More Resilient to Server Crash
If you queue task on the background and something goes wrong, the job can be queued to a dead-letter queue which helps you to ultimately fix problems and re-process the request that caused your unhandled exceptions.
I have a Nifi processor that is calling an external service that can take days before a result is returned. During this time the processor can call Thread.sleep() periodically to relinquish CPU.
The issue is that even if Thread.sleep() is called in an onTrigger() method, the NiFi processor will not read in and handle new FlowFiles since it is waiting for onTrigger() to finish. From NiFi's perspective the cpu is still blocking for the asynchronous call to finish.
Is there a way to maintain concurrency when asynchronous calls are being made in the onTrigger() method of a NiFi processor?
Val Bonn's suggestion of pushing asynchronous FlowFiles back to a WAIT queue works well. As asynchronous requests come in, java Process objects are created and held in memory. The FlowFile is then routed to a WAIT relationship which is connected back into the processor. Periodically FlowFiles from the WAIT queue are checked against the corresponding Process to see if it completed and are then routed to a SUCCESS relationship, otherwise they are penalized. This allows many long running asynchronous processes to be kicked off without allocating precious cpu resources for each incoming request. One source of complexity was handling processor shutdowns invoked from the UI. In these situations an onStopped method is invoked that waits for all in memory processes to complete and archives the stderr and stdout to disk. When the processor is started again, the archive is read back in and paired against any FlowFiles in the WAIT queue.
I'm new to GRPC. I want to know that if the server start a new thread to process when a GRPC client start one request.
There may be up to one Runnable enqueued to the Server's executor for application processing. Each request may generate more than one Runnable over time, but only one at a given time. The default executor is an unbounded cached thread pool, so worst-case each request gets its own thread initially, but later requests will generally reuse previous threads.
It is good practice for high QPS services to specify a fixed-sized executor, to avoid excessive number of threads and reduced context switch thrashing.
In order to overcome the (apparent) 4 minute idle connection timeout on the Azure load balancer, it seems necessary to send some data down the pipe to the client every now and again to keep the connection from being regarded as idle.
Our controller is set up as an AsyncController, and it fires several different asynchronous methods on other objects, all of which are set up to use IO Completion Ports. Thus, we return from our method immediately, and when the completion packet is processed, IIS hooks back up to the original request so that we can render our View.
Is there any way to periodically send a few bytes down the wire in this case? In a "classic" situation, we could have executed the method and then just spun while we waited, sending data every few seconds until the asynchronous method was complete. But, in this situation, the IIS thread is freed to go do other business, and we hook back up to it in our completion callback. What to do? Is this possible?
While your particular case concerns Windows Azure specific (the 4 minute timeout of LBs), the question is pure IIS / ASP.NET workwise. Anyway, I don't think it is possible to send "ping-backs" to the client while in AsyncController/AsyncPage. This is the whole idea of the AsyncPages/Controllers. The IIS leaves the socket aside having the thread serving other requests. And gets back only when you got the OutstandingOperations to zero with AsyncManager.OutstandingOperations.Decrement(); Only then the control is given back to send final response to the client. And once you are the point of sending response, there is no turning back.
I would rather argue for the architectural approach of why you thing someone would wait 4 minutes to get a response (even with a good animated "please wait")? A lot of things may happen during this time. From browser crash, through internet disruption to total power loss/disruption at client. If you are doing real Azure, why not just send tasks for a Worker Role via a Queue (Azure Storage Queues or Service Bus Queues). The other option that stays in front of you for so long running tasks is to use SingalR and fully AJAXed solution. Where you communicate via SignalR the status of the long running operation.
UPDATE 1 due to comments
In addition to the approach suggested by #knightpfhor this can be also achieved with a Queues. Requestor creates a task with some Unique ID and sends it to "Task submission queue". Then "listens" (or polls at regular/irregular intervals) a "Task completion" queue for a message with given Task ID.
In any way I don't see a reason for keeping client connected for the whole duration of the long running task. There are number of ways to decouple such communication.
What are the different approaches in developing a web-server?
So I guess there are (1) multi-thread (2) event-loop, is there anything else? What would be the pros/cons of each approach? when would you use each? can you list specific impl' for each approache
Different approach can be:
Single threaded: All connections are handled by a single thread that
"listens" for and awaits for connections and processes requests.It
is simple to implement but it is the most useless server as it can
only serve request at a time
Multithreaded:The server listens for requests and each incoming
request is allocated to a new thread to handle it.So each client
connection is handled by its dedicated thread. This approach(unlike
1) supports concurrent processing of client requests but does not
scale well since each new request creates a new thread at the server
and this takes a lot of resources.Eventually the server will hit a
limit
Multithreaded-Pools:Same idea as (2) but instead of creating a new
thread to handle each incoming request a thread from a thread-pool
is used.I.e. threads are created and placed on a pool for later
reuse.This scales very well supporting multiple client requests and
it is the standard approach.E.g. Tomcat works like this.
Event-Queue:Each incoming request is placed into a queue and is
processed by a background thread taking requests of the queue. It is
non-blocking and this type of asynchronous processing also scales
well.To be honest I am not sure if it is better than (3) in
performance.I think that tomcat can be configured for this using the
NIO architecture
You should add non-blocking I/O. Have a look at Netty.
Some servers like G-WAN mix Multithreaded-Pools and Event-Queues, letting the server saturate CPU Cores with each thread processing many connections.
Disclamer: I am involved in the development of this project.