I am working on a .net client which consumes a REST api (JSON). The client is a web application with high traffic and considerable user interaction
When writing the wrappers around the external REST api i am trying to decide
If i should make all calls made for the API async? This will be all the way from UI to the API as explained here http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html. This will help me in achieving the desired performance but i will have to figure out a way to handle UI when Tasks are waiting to be completed
Or Is it an overkill? And do i just use sync/sequential code? I could still get (some) performance by Horizontally scaling the application?
I am keen to understand what is the preferred way of calling an external REST api from a client (if there is any) and how is the UI handled for those case where people do use async?
So you have about 10 requests per second in a busy period. This by itself does not require asynchronous IO. Assuming 1sec per request that's 10 threads. 10 threads are nothing.
There is one special case, though: What if the backend service you are calling sometimes takes a long time to respond (a bug, overload, index rebuild, ...)? If it takes 30 seconds to respond or timeout that means that 300 requests are in flight. This is too much for the default thread-pool settings. This will effectively shut down the entire app by pool exhaustion until the requests are cleared.
You can do two things:
Use async IO for all high-volume actions.
Reduce timeouts and have a load-breaker for the number of in-flight requests. Example:
.
SemaphoreSlim sem = new SS(50); //max 50 in-flight
//...
if (!sem.WaitOne(TimeSpan.Zero))
throw new Exception("Load limit exceeded");
Both are equally safe and well-performing. Do not make the mistake to think that async IO causes your IOs to become faster.
The semaphore solution requires less architectural changes but it requires permission to drop requests. These requests would not have completed anyway with high likelihood.
Related
I have an AngularJS app that calls WebAPI. If I log the time I initiatiate a request (in my angluar controller) and log the time OnActionExecuting runs (in an action filter in my WebAPI controller), I notice at times a ~2 second gap. I'm assuming nothing else is running before this filter and this is due to requests being blocked/queued. The reason I assume this is because if I remove all my other data calls, I do not see this gap.
What is the number of parallel requests that WebAPI can handle at once? I tried looking at the ASP.NET performance monitors but couldn't find where I could see this data. Can someone shed some insight into this?
There's no straight answer for this but the shortest one is ...
There is no limit to this for WebApi the limits come from what your server can handle and how efficient the code you have it run is.
...
But since you asked, lets consider some basic things that we can assume about our server and our application ...
concurrent connections
A typical server is known for issues like "c10k" ... https://en.wikipedia.org/wiki/C10k_problem ...so that puts a hard limit on the number of concurrent connections.
Assuming each WebApi call is made from say, some AJAX call on a web page, that gives us a limit of around 10k connections before things get evil.
2.Dependency related overheads
If we then consider the complexity of the code in question you may then have a bottleneck in doing things like SQL queries, I have often written WebApi controllers that have business logic that runs 10+ db queries, the overhead here may be your problem?
Feed in Overhead
What about network bandwidth to the server?
Lets assume we are streaming 1MB of data for each call, it wont take long to choke a 1Gb/s ethernet line with messages that size.
Processing Overhead
Assuming you wrote an Api that does complex calculations (e.g mesh generation for complex 3D data) you could easily choke your CPU for some time on each request.
Timeouts
Assuming the server could accept your request and the request was made asynchronously the biggest issue then is, how long are you prepared to wait for your response? Assuming this is quite short you would reduce the number of problems you have time to solve before each request then needed a response.
...
So as you can see, this is by no means an exhaustive list but it outlines the complexity of the question you asked. That said, I would argue that WebApi (the framework) has no limits, it's really down to the infrastructure around it that has limitations in order to determine what can be possible.
With regards to asynchronous ASP.NET web pages article on MSDN.
The advantages are obvious with long-running pages or high server load. So, given projects where you think demand may be high somewhere down the track, when usage grows, is there any reason NOT to implement async ASP.NET in every web application as a standard? Are there any disadvantages to the approach?
Secondary question: are there any real-world studies/examples of where the advantages start to appear, in different web app situations? Or is it just a matter of suck it and see?
From your own link:
Only I/O-bound operations are good candidates for becoming async action methods on an asynchronous controller class. An I/O-bound operation is an operation that doesn’t depend on the local CPU for completion. When an I/O-bound operation is active, the CPU just waits for data to be processed (that is, downloaded) from external storage (a database or a remote service). I/O-bound operations are in contrast to CPU-bound operations, where the completion of a task depends on the activity of the CPU.
Async pages are not free, they do come at a price. They are generally good when your page is making an external call to a service or performing some long-running, non-CPU bound, operation. Otherwise, you are likely to thrash the CPU, leaving you with a worse situation that you had before going async.
The idea is to use async when you would be eating up a thread from your application's thread pool doing non-CPU intensive work (waiting for a response from a long-running service). That way, your application can continue processing requests and doesn't start queuing new ones, slowly draining the responsiveness from your app.
Here is another link with information when/when not to use async pages.
Edit
As for what is considered "long running," you're faced with the crummy answer of "It depends." To figure this out, you would need to profile your application, see how many of your "long running" requests cause subsequent requests to be queued, instead of processed, by IIS. The decision comes down to being in a situation in which paying the costly toll of context switching is less than the return you're going to get for doing so. If your bottleneck is a certain page or service that causes incoming requests to be held off, it is probably a good idea to start thinking about async work. But, you might also be doing too much work in the request and it could be a "code smell" that you need to refactor your code.
In the end, It depends.
Here is an exerpt from MSDN.
In general, use asynchronous pipelines when the following conditions are true:
The operations are network-bound or I/O-bound instead of CPU-bound.
Testing shows that the blocking operations are a bottleneck in site performance and that IIS can service more requests by using asynchronous action methods for these blocking calls.
Parallelism is more important than simplicity of code.
You want to provide a mechanism that lets users cancel a long-running request.
While the link is about MVC, the idea holds true for other flavors of ASP.NET, too.
Task:
I'm using static classes, so everyone shares data that's been already loaded. If someone makes a change, that request will put an item in a list with an incremental ID, and my idea would be that every client has it's version on client side and requests if there's any change.
My solution: For this I use a $.post with a timeout of 5000ms and sending the client version, on server side I have a 500 cycle for loop which checks if there's something newer and breaks the loop, returns the changes and have a 10ms Thread.Sleep in every cycle so it wouldn't hog the cpu. Either if on the client it times out, has an error, I call the post again, if it succeeds I process the return data, than call the post again. This way I always should get the changes almost instantly without an overwhelming number of requests, and if something fails, I only need to wait 5secs for it to resume.
My problem is that when this loop runs, other requests aren't handled. With asp.net development server, that's okay, because it's single threaded. But that's also the case with win7hp iis7.5.
What I tried: Set it in the registry (HKLM\SOFTWARE\Microsoft\ASP.NET\4.0.30319.0\MaxConcurrentRequestsPerCPU), increasing the worker threads for the application pool, updating the aspnet.config file with maxConcurrentRequestsPerCPU="12" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000" settings, and I also read that my win7hp should be able to use 3 threads. I also thought it's an optimization, that I use same variables in one request so it queues the others, so I commented those lines, left the for loop with the sleep only, but same result.
Don't use Thread.Sleep on threads handling requests in ASP.Net. This will essentially consume thread and prevent more requests to be started. There is restriction on number of threads ASP.Net will create to handle requests that you've tried to change, but high number of threads will make process less responsive and can easily cause OutOfMemeoryException for 32bit proceses - so it is not a good route.
There are several other threads discussing implemeting long poll requests with ASP.Net like this one - Can ASP.NET MVC's AsyncController be used to service large number of concurrent hanging requests (long poll)? and obviously Comet questions like this - Comet implementation for ASP.NET?.
Here's the scenario:
typical web push/comet application where short messages must be pushed at the same time to 3000-4000 connected users;
moving from 1 thread/connection model to a new implementation using Servlet 3.0 (AsyncContext) on Tomcat 7;
One thing I need help to understand is what's the best/safest aproach in implementing the 'notifier' mechanism.
The simplest way is the standard example: 1 thread waiting for a new message and then looping the AsyncContexts list and calling aCtx.getResponse().getWriter().print(message) for each.
Now, my concern is what happens when there are slow clients connected (and considering we may have 1000s this will always be the case). Could the notifier thread be blocked or waste too much time waiting on these slow clients and affecting everybody? This may be trivial but it's not clear for me if the 'write' is asynch or blocking or if the output buffers would compensate for this at least up to a point.
Or is it better to create for each connected client a Runnable (that would do the aCtx.getResponse().getWriter().print(message) ) and submit it to an Executor using a fixed thread pool? I'm not sure however if creating let's say 4000 Runnable at a time (for each message) makes sense or what the performance would be.
And one last thing, does anyone have real/production experience using Tomcat 7 for this sort of app (asynch servlet)? Should i consider it 'production ready in this area? Thanks.
I'm somewhat vague about when to use asynchronous operations in ASP.NET. I understand, whenever I make a call to external web services such as calling Twitter APIs and what not, I should be using asynchronous operations so the CLR threads could be freed and service other requests, make sense.
I read an excellent blog once mentioned that if your operation is using CPU efficiently, then you shouldn't do asynchronous operation because it has a context switch penalty, however, if you are doing a long operation and waits a lot, then it's worth to do the context switch.
What about a page that use ajax call to a local web service which in turn makes a database operations (takes around 3 seconds), returns JSON and then the page itself using JQuery, renders it for another second for a total of 4 seconds?
What about a traditional webform, from page_load makes the same database operation call (3 seconds) and then take another 3 seconds to render? For instance, a big forum post with 1000 comments?
My general impression is that shouldn't IIS be designed such that EVERY operation is asynchronous by default in the background such that all operation is non-blocking without the context switch penalty? Is that the idea of node.js? and If you do have static pages, no wait operations, only then should you specifically write a synchronous page? Basically the reverse of what is happening now?
Thanks a lot.
The only benefit of asynchronous requests is that it frees up worker threads. So you only notice it if you run out of worker threads. By default, there are 100 threads per CPU. So you won't notice the performance improvement unless you have more than 100 requests per 4 seconds per CPU.
The drawback of asynchronous requests is that it makes your code harder to understand and maintain.