I have developed a .net core Web API and there is a scenario where I have a list of around 1000 records where each record will be looped through and calls a third party api. There is a restriction with the third party API where concurrently only 200 requests can be sent. So, I have used SemaphoreSlim and restricted the number of threads that uses this code block to 200 and it works fine.
When multiple users or multiple requests come in for this endpoint, then the third party api is throwing an error.
How can I restrict the SemaphoreSlim to use only 200 threads across all the requests (when multiple users or requests come in at the same time)?
SemaphoreSlim _concurrencySemaphoreForDescartesCall = new SemaphoreSlim(1,200);
List<Task<searchDetailsViewModel>> searchList = new List<Task<searchDetailsViewModel>>(searchCriteriaList.Count);
foreach (var criteria in searchCriteriaList)
{
await _concurrencySemaphore.WaitAsync();
searchList.Add(Task.Run<searchDetailsViewModel>(async () =>
{
searchDetailsViewModel searchResults = new searchDetailsViewModel();
try
{
searchResults.searchResults = await AsncCall(criteria);
}
catch (Exception)
{
searchResults.ErrorMessage = "There was a problem performing the s search.";
}
finally
{
// here we release the throttler immediately
_concurrencySemaphore.Release();
}
return searchResults;
}, cancellationToken));
}
searchDetailsViewModel[] searchResultsList = await Task.WhenAll(searchList);
How can I restrict the SemaphoreSlim to use only 200 threads across all the requests (when multiple users or requests come in at the same time)?
Change the scope of your SemaphoreSlim instance.
Currently, the code creates a SemaphoreSlim for each request, so each request is limited to 200 simultaneous requests. To have a SemaphoreSlim work across multiple requests, you should define it as shared between those requests. Either encapsulate the SemaphoreSlim within a type that is injected with singleton lifetime, or declare the SemaphoreSlim as static.
Related
I have a handler to initiate a password reset. It always returns a successful 200 status code, so that an attacker cannot use it to find out which email addresses are stored in the database. The problem is, if an email is in the database, it'll take a while for the request to be fulfilled (blocking user lookup and sending the actual email with a reset token). If the user is not in the db, the request returns very quickly, so an attacked would know the email is not there.
How would I go about returning the HTTP response right away while processing the request in the background?
pub async fn forgot_password_handler(
email_from_path: web::Path<String>,
pool: web::Data<Pool>,
redis_client: web::Data<redis::Client>,
) -> HttpResponse {
let conn: &PgConnection = &pool.get().unwrap();
let email_address = &email_from_path.into_inner();
// search for user with email address in users table
match users.filter(email.eq(email_address)).first::<User>(conn) {
Ok(user) => {
// some stuff omitted.. this is what happens:
// create random token for user and store a hash of it in redis (it'll expire after some time)
// send email with password reset link and token (not hashed) to client
// then return with
HttpResponse::Ok().finish(),
}
_ => HttpResponse::Ok().finish(),
}
}
You can use an Actix Arbiter to schedule an asynchronous task:
use actix::Arbiter;
async fn do_the_database_stuff(
email: String,
pool: web::Data<Pool>,
redis_client: web::Data<redis::Client>)
{
// async database code here
}
pub async fn forgot_password_handler(
email_from_path: web::Path<String>,
pool: web::Data<Pool>,
redis_client: web::Data<redis::Client>,
) -> HttpResponse {
let email = email_from_path.clone();
Arbiter::spawn(async {
do_the_database_stuff(
email,
pool,
redis_client
);
});
HttpResponse::Ok().finish()
}
If your database code is blocking, to prevent hogging the long-lived Actix worker threads, you could instead create a new Arbiter, with its own thread:
fn do_the_database_stuff(email: String) {
// blocking database code here
}
pub async fn forgot_password_handler(email_from_path: String) -> HttpResponse {
let email = email_from_path.clone();
Arbiter::new().exec_fn(move || {
async move {
do_the_database_stuff(email).await;
};
});
HttpResponse::Ok().finish()
}
This may be a bit more work because Pool and redis::Client are unlikely to be safe to share between threads, so you will have to solve that too. That's why I didn't include them in the example code.
It's better to use Arbiters than be tempted to spawn a new native thread with std::thread. If you mix the two, you can end up accidentally including code that messes up the worker. For example using std::thread::sleep in an async context would pause unrelated tasks that just happen to be scheduled on the same worker, and may not even have any effect on the task you intended.
Finally, you might also consider an architectural change. If you factor database-heavy tasks into their own microservices, you would solve this problem automatically. The web handler can then just send a message (Kafka, RabbitMQ, ZMQ, HTTP, or whatever you choose) and immediately return. This will let you scale the microservices independently of the webserver - 10x web server instances doesn't have to mean 10x database connections, if you only need one instance for the password reset service.
I'm trying to figure out why my webservice is so slow and find ways to get it to respond faster. Current average response time without custom processing involved (i.e. apicontroller action returning a very simple object) is about 75ms.
The setup
Machine:
32GB RAM, SSD disk, 4 x 2.7Ghz CPU's, 8 logical processors, x64 Windows 10
Software:
1 asp.net mvc website running .net 4.0 on IISEXPRESS (System.Web.Mvc v5.2.7.0)
1 asp.net web api website running .net 4.0 on IISEXPRESS (System.Net.Http v4.2.0.0)
1 RabbitMQ messagebus
Asp.net Web API Code (Api Controller Action)
[Route("Send")]
[HttpPost]
[AllowAnonymous)
public PrimitiveTypeWrapper<long> Send(WebsiteNotificationMessageDTO notification)
{
_messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification });
return new PrimitiveTypeWrapper<long>(1);
}
The body of this method takes 2ms. Stackify tells me there's a lot of overhead on the AuthenticationFilterResult.ExecuteAsync method but since it's an asp.net thing I don't think it can be optimized much.
Asp.net MVC Code (MVC Controller Action)
The RestClient implementation is shown below. The HttpClientFactory returns a new HttpClient instance with the necessary headers and basepath.
public async Task<long> Send(WebsiteNotificationMessageDTO notification)
{
var result = await _httpClientFactory.Default.PostAndReturnAsync<WebsiteNotificationMessageDTO, PrimitiveTypeWrapper<long>>("/api/WebsiteNotification/Send", notification);
if (result.Succeeded)
return result.Data.Value;
return 0;
}
Executing 100 requests as fast as possible on the backend rest service:
[HttpPost]
public async Task SendHundredNotificationsToMqtt()
{
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 100; i++)
{
await _notificationsRestClient.Send(new WebsiteNotificationMessageDTO()
{
Severity = WebsiteNotificationSeverity.Informational,
Message = "Test notification " + i,
Title = "Test notification " + i,
UserId = 1
});
}
sw.Stop();
Debug.WriteLine("100 messages sent, took {0} ms", sw.ElapsedMilliseconds);
}
This takes on average 7.5 seconds.
Things I've tried
Checked the number of available threads on both the REST service and the MVC website:
int workers;
int completions;
System.Threading.ThreadPool.GetMaxThreads(out workers, out completions);
which returned for both:
Workers: 8191
Completions: 1000
Removed all RabbitMQ messagebus connectivity to ensure it's not the culprit. I've also removed the messagebus publish method from the rest method _messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification }); So all it does is return 1 inside a wrapping object.
The backend rest is using identity framework with bearer token authentication and to eliminate most of it I've also tried marking the controller action on the rest service as AllowAnonymous.
Ran the project in Release mode: No change
Ran the sample 100 requests twice to exclude service initialization cost: No change
After all these attempts, the problem remains, it will still take about +- 75ms per request. Is this as low as it goes?
Here's a stackify log for the backend with the above changes applied.
The web service remains slow, is this as fast as it can get without an expensive hardware upgrade or is there something else I can look into to figure out what's making my web service this slow?
I have been having a look to a digest authentication example at:
http://hc.apache.org/httpcomponents-client-4.3.x/examples.html
In my scenario the there are several threads issuing HTTP requests and each of them has to be authenticated with their own set of credentials. Additionally, please consider this question is probably very specific for the Apache HTTP client 4.3 onwards, 4.2 handles authentication probably in a different way, although I didn't check it myself. That said, there goes the actual question.
I want to use just one client instance (static member of the class, that is threadsafe) and give it a connection manager to support several concurrent requests. The point is that each request will provide different credentials and I am not seeing the way to assign credentials per request as the credentials provider is set when building the http client. From the link above:
[...]
HttpHost targetHost = new HttpHost("localhost", 80, "http");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(targetHost.getHostName(), targetHost.getPort()),
new UsernamePasswordCredentials("username", "password"));
CloseableHttpClient httpclient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider).build();
[...]
Checking:
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html#d5e600
The code sample in point 4.4 (seek 4.4. HTTP authentication and execution context), seems to say that the HttpClientContext is given the auth cache and the credentials provider and then is passed to the HTTP request. Next to it the request is executed and it seems that the client will get credentials filtering by the host in the HTTP request. In other words: if the context (or the cache) has valid credentials for the target host of the current HTTP request, he will use them. The problem for me is that different threads will perform different requests to the same host.
Is there any way to provide custom credentials per HTTP request?
Thanks in advance for your time! :)
The problem for me is that different threads will perform different requests to the same host.
Why should this be a problem? As long as you use a different HttpContext instance per thread, execution contexts of those threads are going to be completely indepenent
CloseableHttpClient httpclient = HttpClients.createDefault();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("user:pass"));
HttpClientContext localContext = HttpClientContext.create();
localContext.setCredentialsProvider(credentialsProvider);
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget, localContext);
try {
EntityUtils.consume(response.getEntity());
} finally {
response.close();
}
I have a similar issue.
I must call n-times a service with a single system user, authenticated with NTLM. I want to do this using multiple threads.
What I came up with is creating a single HTTPClient with no default credential provider. When a request needs to be performed I use an injected CredentialProviderFactory into the method performing the request (in a specific thread). Using this I get a brand new CredentialsProvider and I put this into a Context (created in the thread).
Then I call the execute method on the client using the overload execute(method, context).
class MilestoneBarClient implements IMilestoneBarClient {
private static final Logger log = LoggerFactory.getLogger(MilestoneBarClient.class);
private MilestoneBarBuilder builder;
private CloseableHttpClient httpclient;
private MilestoneBarUriBuilder uriBuilder;
private ICredentialsProviderFactory credsProviderFactory;
MilestoneBarClient(CloseableHttpClient client, ICredentialsProviderFactory credsProviderFactory, MilestoneBarUriBuilder uriBuilder) {
this(client, credsProviderFactory, uriBuilder, new MilestoneBarBuilder());
}
MilestoneBarClient(CloseableHttpClient client, ICredentialsProviderFactory credsProviderFactory, MilestoneBarUriBuilder uriBuilder, MilestoneBarBuilder milestoneBarBuilder) {
this.credsProviderFactory = credsProviderFactory;
this.uriBuilder = uriBuilder;
this.builder = milestoneBarBuilder;
this.httpclient = client;
}
// This method is called by multiple threads
#Override
public MilestoneBar get(String npdNumber) {
log.debug("Asking milestone bar info for {}", npdNumber);
try {
String url = uriBuilder.getPathFor(npdNumber);
log.debug("Building request for URL {}", url);
HttpClientContext localContext = HttpClientContext.create();
localContext.setCredentialsProvider(credsProviderFactory.create());
HttpGet httpGet = new HttpGet(url);
long start = System.currentTimeMillis();
try(CloseableHttpResponse resp = httpclient.execute(httpGet, localContext)){
[...]
For some reasons I sometimes get an error, but I guess it's an NTLMCredentials issue (not being thread-safe...).
In your case, you could probably pass the factory to the get methods instead of passing in creation.
I have built a bulk email sending website for a client that is required to send out 80,000 emails in a single send. It basically creates a new thread for the send so that control can be handed back to the UI (so that a feedback page can load) and then a new thread is created for each company in order to send emails to their recipients. The emails are all queued up using this code:
// Loop through the companies and send their mail to the specified recipients
// while creating a new thread for each company
// A new thread is started so that the feedback page can load
SendingThread = Task.Factory.StartNew(() =>
{
// This is a thread safe for loop
Parallel.ForEach<CompanyEntity>(companies, company =>
{
// Start a new thread for each company send
Task.Factory.StartNew(() =>
{
// Get the recipients for this company
var companyRecipients = GetSubscribersForCompany(company.Id, recipients);
// Send the newsletter to the company recipients
var success = SendNewsletterForCompany(newsletter, company, companyRecipients, language,
version, company.AdvertCollectionViaNewsletterCompanyAdvertLink, newsletter.NewsletterType, email);
// Add the status update so the front end can view a list of updated conpany statuses
if (success)
AddStatusUpdate(company.CompanyTitle + " has completed processing.");
// Starts sending the emails if the engine hasn't already been started
SendEngine.Start(CurrentSmtpClient, this);
}).ContinueWith(antecendent => EndCompaniesSendUpdate(companiesToProcess, companiesProcessed), TaskContinuationOptions.OnlyOnRanToCompletion);
});
}, new CancellationToken(), TaskCreationOptions.LongRunning, TaskScheduler.Default);
While the emails are queued, the send engine takes over and pulls emails from the queue and then sends them using the new Parallel class:
Action action = () =>
{
MailMessage message;
while (queue.TryDequeue(out message))
{
SendMessage(sendingServer, message, factory);
}
};
// Start 5 concurrent actions to send the messages in parallel.
Parallel.Invoke(action, action, action, action, action);
All of this works great and can send 40,000 newsletters out in about 10 minutes. The only problem is that the RAM and CPU on the server are 100% consumed for those 10 minutes. This affects other sites on the server as they can't be accessed.
Is there any way to restrict the resource usage of the sending application either in IIS 7.5 or by changing the code above?
Problems:
You are generating a thread inside of a Parallel ForEach, The "Parallel" part means it's already spawning a thread for the body. You are nesting Parallel Invoke instead of an Action inside of a Parallel ForEach inside of another Action.
You are running a while loop inside of a thread with no rest for the CPU. That is being Parallel Invoked 5x.
Answers:
For CPU usage, you need to give your processing a breather. In your While TryDequeue loop put a short Sleep.
MailMessage message;
while (queue.TryDequeue(out message))
{
SendMessage(sendingServer, message, factory);
Thread.Sleep(16);
}
For RAM and CPU usage, you need to process LESS at once.
SendingThread = Task.Factory.StartNew(() =>
{
foreach(var company in companies)
{
// Get the recipients for this company
var companyRecipients = GetSubscribersForCompany(company.Id, recipients);
// Send the newsletter to the company recipients
var success = SendNewsletterForCompany(newsletter, company, companyRecipients, language,
version, company.AdvertCollectionViaNewsletterCompanyAdvertLink, newsletter.NewsletterType, email);
// Add the status update so the front end can view a list of updated conpany statuses
if (success)
AddStatusUpdate(company.CompanyTitle + " has completed processing.");
// Starts sending the emails if the engine hasn't already been started
SendEngine.Start(CurrentSmtpClient, this);
}
}, new CancellationToken(), TaskCreationOptions.LongRunning, TaskScheduler.Default);
I am just about to launch my ASP.NET MVC3 web app to production, however, as a complex app, it takes a LONG time to start up. Obviously, I don't want my users waiting over a minute for their first request to go through after the AppPool has timed out.
From my research, i've found that there are two ways to combat this:
Run a worker role or other process - which poll's the website every 19 minutes preventing the warm up.
Change the timeout from the default 20 minutes - To something much larger.
As Solution 2 seems like the better idea, i just wondered what the disadvantages would be of this, will I run out of memory etc.?
Thanks.
Could you use the auto-start feature of IIS? There is a post here that presents this idea.
You'd have IIS 7.5 and Win2k8 R2 with Azure OS family 2. You'd just need to be able to script/automate any setup steps and configuration.
I do this with a background thread that requests a keepalive URL every 15 minutes. Not only does this keep the app from going idle, but it also warms up the app right away anytime the web role or virtual machine restarts or is rebuilt.
This is all possible because Web Roles really are just Worker Roles that also do IIS stuff. So you can still use all the standard Worker Role startup hooks in a Web Role.
I got the idea from this blog post but tweaked the code to do a few extra warmup tasks.
First, I have a class that inherits from RoleEntryPoint (it does some other things besides this warm up task and I removed them for simplicity):
public class WebRole : RoleEntryPoint
{
// other unrelated member variables appear here...
private WarmUp _warmUp;
public override bool OnStart()
{
// other startup stuff appears here...
_warmUp = new WarmUp();
_warmUp.Start();
return base.OnStart();
}
}
All the actual warm up logic is in this WarmUp class. When it first runs it hits a handful of URLs on the local instance IP address (vs the public, load balanced hostname) to get things in memory so that the first people to use it get the fastest possible response time. Then, it loops and hits a single keepalive URL (again on the local role instance) that doesn't do any work and just serves to make sure that IIS doesn't shut down the application pool as idle.
public class WarmUp
{
private Thread worker;
public void Start()
{
worker = new Thread(new ThreadStart(Run));
worker.IsBackground = true;
worker.Start();
}
private void Run()
{
var endpoint = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["http"]; // "http" has to match the endpointName in your ServiceDefinition.csdef file.
var pages = new string[]
{
"/",
"/help",
"/signin",
"/register",
"/faqs"
};
foreach (var page in pages)
{
try
{
var address = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
page);
var webClient = new WebClient();
webClient.DownloadString(address);
Debug.WriteLine(string.Format("Warmed {0}", address));
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
}
}
var keepalive = String.Format("{0}://{1}:{2}{3}",
endpoint.Protocol,
endpoint.IPEndpoint.Address,
endpoint.IPEndpoint.Port,
"/keepalive");
while (true)
{
try
{
var webClient = new WebClient();
webClient.DownloadString(keepalive);
Debug.WriteLine(string.Format("Pinged {0}", keepalive));
}
catch (Exception ex)
{
//absorb
}
Thread.Sleep(900000); // 15 minutes
}
}
}
Personally I'd change the timeout, but both should work: effectively they would both have the same effect of preventing the worker processes from shutting down.
I believe the timeout is there to avoid IIS retaining resources that aren't needed for servers with lots of Web sites that are lightly used. Given that heavily used sites (like this one!) don't shut down their worker processes I don't think you'll see any memory issues.