I'm trying to figure out why my webservice is so slow and find ways to get it to respond faster. Current average response time without custom processing involved (i.e. apicontroller action returning a very simple object) is about 75ms.
The setup
Machine:
32GB RAM, SSD disk, 4 x 2.7Ghz CPU's, 8 logical processors, x64 Windows 10
Software:
1 asp.net mvc website running .net 4.0 on IISEXPRESS (System.Web.Mvc v5.2.7.0)
1 asp.net web api website running .net 4.0 on IISEXPRESS (System.Net.Http v4.2.0.0)
1 RabbitMQ messagebus
Asp.net Web API Code (Api Controller Action)
[Route("Send")]
[HttpPost]
[AllowAnonymous)
public PrimitiveTypeWrapper<long> Send(WebsiteNotificationMessageDTO notification)
{
_messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification });
return new PrimitiveTypeWrapper<long>(1);
}
The body of this method takes 2ms. Stackify tells me there's a lot of overhead on the AuthenticationFilterResult.ExecuteAsync method but since it's an asp.net thing I don't think it can be optimized much.
Asp.net MVC Code (MVC Controller Action)
The RestClient implementation is shown below. The HttpClientFactory returns a new HttpClient instance with the necessary headers and basepath.
public async Task<long> Send(WebsiteNotificationMessageDTO notification)
{
var result = await _httpClientFactory.Default.PostAndReturnAsync<WebsiteNotificationMessageDTO, PrimitiveTypeWrapper<long>>("/api/WebsiteNotification/Send", notification);
if (result.Succeeded)
return result.Data.Value;
return 0;
}
Executing 100 requests as fast as possible on the backend rest service:
[HttpPost]
public async Task SendHundredNotificationsToMqtt()
{
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 100; i++)
{
await _notificationsRestClient.Send(new WebsiteNotificationMessageDTO()
{
Severity = WebsiteNotificationSeverity.Informational,
Message = "Test notification " + i,
Title = "Test notification " + i,
UserId = 1
});
}
sw.Stop();
Debug.WriteLine("100 messages sent, took {0} ms", sw.ElapsedMilliseconds);
}
This takes on average 7.5 seconds.
Things I've tried
Checked the number of available threads on both the REST service and the MVC website:
int workers;
int completions;
System.Threading.ThreadPool.GetMaxThreads(out workers, out completions);
which returned for both:
Workers: 8191
Completions: 1000
Removed all RabbitMQ messagebus connectivity to ensure it's not the culprit. I've also removed the messagebus publish method from the rest method _messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification }); So all it does is return 1 inside a wrapping object.
The backend rest is using identity framework with bearer token authentication and to eliminate most of it I've also tried marking the controller action on the rest service as AllowAnonymous.
Ran the project in Release mode: No change
Ran the sample 100 requests twice to exclude service initialization cost: No change
After all these attempts, the problem remains, it will still take about +- 75ms per request. Is this as low as it goes?
Here's a stackify log for the backend with the above changes applied.
The web service remains slow, is this as fast as it can get without an expensive hardware upgrade or is there something else I can look into to figure out what's making my web service this slow?
Related
I have developed a .net core Web API and there is a scenario where I have a list of around 1000 records where each record will be looped through and calls a third party api. There is a restriction with the third party API where concurrently only 200 requests can be sent. So, I have used SemaphoreSlim and restricted the number of threads that uses this code block to 200 and it works fine.
When multiple users or multiple requests come in for this endpoint, then the third party api is throwing an error.
How can I restrict the SemaphoreSlim to use only 200 threads across all the requests (when multiple users or requests come in at the same time)?
SemaphoreSlim _concurrencySemaphoreForDescartesCall = new SemaphoreSlim(1,200);
List<Task<searchDetailsViewModel>> searchList = new List<Task<searchDetailsViewModel>>(searchCriteriaList.Count);
foreach (var criteria in searchCriteriaList)
{
await _concurrencySemaphore.WaitAsync();
searchList.Add(Task.Run<searchDetailsViewModel>(async () =>
{
searchDetailsViewModel searchResults = new searchDetailsViewModel();
try
{
searchResults.searchResults = await AsncCall(criteria);
}
catch (Exception)
{
searchResults.ErrorMessage = "There was a problem performing the s search.";
}
finally
{
// here we release the throttler immediately
_concurrencySemaphore.Release();
}
return searchResults;
}, cancellationToken));
}
searchDetailsViewModel[] searchResultsList = await Task.WhenAll(searchList);
How can I restrict the SemaphoreSlim to use only 200 threads across all the requests (when multiple users or requests come in at the same time)?
Change the scope of your SemaphoreSlim instance.
Currently, the code creates a SemaphoreSlim for each request, so each request is limited to 200 simultaneous requests. To have a SemaphoreSlim work across multiple requests, you should define it as shared between those requests. Either encapsulate the SemaphoreSlim within a type that is injected with singleton lifetime, or declare the SemaphoreSlim as static.
I'm facing problem with kestrel server's performance. I have following scenario :
TestClient(JMeter) -> DemoAPI-1(Kestrel) -> DemoAPI-2(IIS)
I'm trying to create a sample application that could get the file content as and when requested.
TestClient(100 Threads) requests to DemoAPI-1 which in turn request to DemoAPI-2. DemoAPI-2 reads a fixed XML file(1 MB max) and returns it's content as a response(In production DemoAPI-2 is not going to be exposed to outside world).
When I tested direct access from TestClient -> DemoAPI-2 I got expected result(good) which is following :
Average : 368ms
Minimum : 40ms
Maximum : 1056ms
Throughput : 40.1/sec
But when I tried to access it through DemoAPI-1 I got following result :
Average : 48232ms
Minimum : 21095ms
Maximum : 49377ms
Throughput : 2.0/sec
As you can see there is a huge difference.I'm not getting even the 10% throughput of DemoAPI-2. I was told has kestrel is more efficient and fast compared to traditional IIS. Also because there is no problem in direct access, I think we can eliminate the possible of problem on DemoAPI-2.
※Code of DemoAPI-1 :
string base64Encoded = null;
var request = new HttpRequestMessage(HttpMethod.Get, url);
var response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseContentRead).ConfigureAwait(false);
if (response.StatusCode.Equals(HttpStatusCode.OK))
{
var content = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
base64Encoded = Convert.ToBase64String(content);
}
return base64Encoded;
※Code of DemoAPI-2 :
[HttpGet("Demo2")]
public async Task<IActionResult> Demo2Async(int wait)
{
try
{
if (wait > 0)
{
await Task.Delay(wait);
}
var path = Path.Combine(Directory.GetCurrentDirectory(), "test.xml");
var file = System.IO.File.ReadAllText(path);
return Content(file);
}
catch (System.Exception ex)
{
return StatusCode(500, ex.Message);
}
}
Some additional information :
Both APIs are async.
Both APIs are hosted on different EC2 instances(C5.xlarge Windows Server 2016).
DemoAPI-1(kestrel) is a self-contained API(without reverse proxy)
TestClient(jMeter) is set to 100 thread for this testing.
No other configuration is done for kestrel server as of now.
There are no action filter, middleware or logging that could effect the performance as of now.
Communication is done using SSL on 5001 port.
Wait parameter for DemoAPI2 is set to 0 as of now.
The CPU usage of DEMOAPI-1 is not over 40%.
The problem was due to HttpClient's port exhaustion issue.
I was able to solve this problem by using IHttpClientFactory.
Following article might help someone who faces similar problem.
https://www.stevejgordon.co.uk/httpclient-creation-and-disposal-internals-should-i-dispose-of-httpclient
DEMOAPI-1 performs a non-asynchronous read of the streams:
var bytes = stream.Read(read, 0, DataChunkSize);
while (bytes > 0)
{
buffer += System.Text.Encoding.UTF8.GetString(read, 0, bytes);
// Replace with ReadAsync
bytes = stream.Read(read, 0, DataChunkSize);
}
That can be an issue with throughput on a lot of requests.
Also, I'm not fully aware of why are you not testing the same code with IIS and Kestrel, I would assume you need to make only environmental changes and not the code.
We have a simple application in ASP.NET Core which calls a website and returns the content. The Controller method looks like this:
[HttpGet("test/get")]
public ActionResult<string> TestGet()
{
var client = new WebClient
{
BaseAddress = "http://v-dev-a"
};
return client.DownloadString("/");
}
The URL which we call is just the default page of an IIS. I am using Apache JMeter to test 1000 requests in 10 seconds. I have always the same issue, after about 300-400 requests it gets stuck for a few minutes and nothing works. The appplication which holds the controller is completely frozen.
In the performance monitor (MMC) I see that the connection are at 100%.
I tried the same code with ASP.NET 4.7.2 and it runs without any issues.
I also read about that the dispose of the WebClient does not work and I should make it static. See here
Also the deactivation of the KeepAlive did not help:
public class QPWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
((HttpWebRequest)request).KeepAlive = false;
}
return request;
}
}
The HttpClient hast the same issue, it does not change anything
With dependency injection like recommended here there is an exception throw that the web client can't handle more request at the same time.
Another unsuccessful try was to change ConnectionLimit and SetTcpKeepAlive in ServicePoint
So I am out of ideas how to solve this issue. Last idea is to move the application to ASP.NET. Does anyone of you have an idea or faced already the same issue?
Please considerer the following scenario :
I have created a full-web application by using the ASP .NET MVC 3 framework. Now my application is managed by a web server.
An HTTP request is received on the server-side of my application.
A class implementing the singleton design pattern is instanciated on server-side.
A response is sent to the browser.
Another HTTP request is received on the server-side of my application. Is the singleton instance used at step 2 still available on server-side ?
I read some information about the life cycle of an ASP .NET application on this page : http://msdn.microsoft.com/en-us/library/ms178473.aspx
But I am still not able to answer my question.
Thanks in advance for your future help
I have just made some tests under VS2010.
Here is the list of the main components of my project :
The Home controller containing an Index HttpGet action method.
The view resulting from the Index action method.
The SingletonTest class which implements the singleton design pattern.
Here is the code of the SingletonTest class :
public class SingletonTest
{
private int counter;
private static SingletonTest instance = null;
public int Counter
{
get
{
return counter;
}
}
public static SingletonTest Instance
{
get
{
if (instance == null)
instance = new SingletonTest();
return instance;
}
}
private SingletonTest()
{
counter = 0;
}
public void IncrementCounter()
{
counter++;
}
}
Here is the code of the Index action method :
public ActionResult Index()
{
SingletonTest st = SingletonTest.Instance;
st.IncrementCounter();
return View();
}
Here is the code of the view :
#SingletonTest.Instance.Counter
Here the test scenario I have followed :
The IIS server has been automatically launched by VS2010.
I have requested the /Home/Index/ URL then the value 1 has been displayed.
I have requested the /Home/Index/ URL then the value 2 has been displayed.
...
This test shows that the SingletonTest instance made at Step 1 is available when processing the next requests.
I guess that a memory space is allocated to my web application on the server.
Then I have stopped the IIS server and I have followed my test scenario again.
I have got the same results as before : 1, 2, ....
Even though the singleton may persist across multiple requests you need to be careful for exactly the reasons of your second test - when IIS is restarted or the app pool is recycled everything will be lost.
Are you sure that you need a singleton instance?
If you're looking to persist some state across all requests it would be better to use an external storage such as a database.
What about multi-instances of the same application that IIS Will Create to process the concurrent requests?
I thinks singleton Object will not be the same if IIS Create multiple instances of the same application in high traffic situation
I am just about to launch my ASP.NET MVC3 web app to production, however, as a complex app, it takes a LONG time to start up. Obviously, I don't want my users waiting over a minute for their first request to go through after the AppPool has timed out.
From my research, i've found that there are two ways to combat this:
Run a worker role or other process - which poll's the website every 19 minutes preventing the warm up.
Change the timeout from the default 20 minutes - To something much larger.
As Solution 2 seems like the better idea, i just wondered what the disadvantages would be of this, will I run out of memory etc.?
Thanks.
Could you use the auto-start feature of IIS? There is a post here that presents this idea.
You'd have IIS 7.5 and Win2k8 R2 with Azure OS family 2. You'd just need to be able to script/automate any setup steps and configuration.
I do this with a background thread that requests a keepalive URL every 15 minutes. Not only does this keep the app from going idle, but it also warms up the app right away anytime the web role or virtual machine restarts or is rebuilt.
This is all possible because Web Roles really are just Worker Roles that also do IIS stuff. So you can still use all the standard Worker Role startup hooks in a Web Role.
I got the idea from this blog post but tweaked the code to do a few extra warmup tasks.
First, I have a class that inherits from RoleEntryPoint (it does some other things besides this warm up task and I removed them for simplicity):
public class WebRole : RoleEntryPoint
{
// other unrelated member variables appear here...
private WarmUp _warmUp;
public override bool OnStart()
{
// other startup stuff appears here...
_warmUp = new WarmUp();
_warmUp.Start();
return base.OnStart();
}
}
All the actual warm up logic is in this WarmUp class. When it first runs it hits a handful of URLs on the local instance IP address (vs the public, load balanced hostname) to get things in memory so that the first people to use it get the fastest possible response time. Then, it loops and hits a single keepalive URL (again on the local role instance) that doesn't do any work and just serves to make sure that IIS doesn't shut down the application pool as idle.
public class WarmUp
{
private Thread worker;
public void Start()
{
worker = new Thread(new ThreadStart(Run));
worker.IsBackground = true;
worker.Start();
}
private void Run()
{
var endpoint = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["http"]; // "http" has to match the endpointName in your ServiceDefinition.csdef file.
var pages = new string[]
{
"/",
"/help",
"/signin",
"/register",
"/faqs"
};
foreach (var page in pages)
{
try
{
var address = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
page);
var webClient = new WebClient();
webClient.DownloadString(address);
Debug.WriteLine(string.Format("Warmed {0}", address));
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
}
}
var keepalive = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
"/keepalive");
while (true)
{
try
{
var webClient = new WebClient();
webClient.DownloadString(keepalive);
Debug.WriteLine(string.Format("Pinged {0}", keepalive));
}
catch (Exception ex)
{
//absorb
}
Thread.Sleep(900000); // 15 minutes
}
}
}
Personally I'd change the timeout, but both should work: effectively they would both have the same effect of preventing the worker processes from shutting down.
I believe the timeout is there to avoid IIS retaining resources that aren't needed for servers with lots of Web sites that are lightly used. Given that heavily used sites (like this one!) don't shut down their worker processes I don't think you'll see any memory issues.