I am trying to read thousands of record using c# windows service. I am reading data from RabbitMQ (service broker) topic. My service is able to read thousands of record per sec but issue occurred with processing. I am using Task.Run to process data. Here is the sample code.
Task.Run(() => {
//db transactions
});
Based on my logging, everything works well. Only issue is with concurrent thread, when i check system log, my service has only 255 (concurrent) thread. Where it supposed to be around 1000+.
What could be blocking for my thread?
Related
I have an S1 AppService Plan at Azure with a SQL Database connected. I'm using EF Core.
Every now and then, not only after restarts of the app, database commands are extremely slow. The Profiler says only "waiting". But waiting for what?
Profiler picture
How can I find out what's blocking here?
It can be observed in the shared Profiler snip that your App has been stuck on DbContext.SaveChangesAsync method which is taking time to complete and therefore triggering AWAIT_TIME, but your parallel threads keep executing. Please visit double counting in parallel threads to know more.
The SaveChangesAsync method Asynchronously saves all changes made in this context to the underlying database. In order to cancel this process if it is waiting for the task to complete, you need to use cancellationToken parameter.
Check the sample code below for your reference:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA1801:ReviewUnusedParameters", MessageId="cancellationToken")]
public virtual System.Threading.Tasks.Task<int> SaveChangesAsync (System.Threading.CancellationToken cancellationToken);
Helpful link: https://learn.microsoft.com/en-us/dotnet/api/system.data.entity.dbcontext.savechangesasync?view=entity-framework-6.2.0#System_Data_Entity_DbContext_SaveChangesAsync_System_Threading_CancellationToken_
Is it worthwhile to use Task based Asynch action methods in an ASP.NET MVC project to retrieve a bunch of EntityFramework queries which use LINQ-to-EF. The website is hosted in Azure as is the database. There are quite a few queries that load up a bunch of arrays to send via JSON to an ajax call from the client.
It is worthwhile as long as you are using truly async api (such as EntityFramework async methods) and not creating an async wrappers for example by wrapping your synchronous core in Task.Run or Task.FromResult.
Asynchronous method doesn't suppose to block any thread while it executes (Great article about this).
By the way, because both your application and database are cloud based, it actually makes sense to look into async approach since both your application and you DB can scale.
Technically, it's all three. There's network latency in sending the query/receiving the results. There's CPU-bound work when SQL Server or whatever is actually running the query against your database, and there's I/O as the database itself is a file on the filesystem.
However, from the perspective of your application, interacting with an external database via Entity Framework is network-bound, and is eligible for async as a result.
However, async is not a magical pill that will make your app run faster. If anything using async can actually slow your application down, as there's a lot of overhead to make async work. It's only purpose is to allow the thread doing work to return to the pool while it waits on some task outside of its control to finish. In the case of a database query, the query is sent off to the database which takes some period of time depending on network latency. The database must run the query and construct a result which takes some period of time depending on the resources available on that server. Then, the result is sent back to your application, which again incurs network latency. In a synchronous scenario, your application would just sit there and wait, holding onto the thread. In an async scenario, the thread is released to do other work (server other requests for the web server, etc.) and then when the database response is received the thread is requested back to finish whatever was going on in the application.
In case it's not obvious, the purpose of this is to allow the web server or whatever else you're working with to handle additional load by utilizing periods of what otherwise would be downtime doing other work. In that regard, async is important for using resources efficiently, but it's not necessarily quicker or more performant.
I have an application that is working well in production, but I wonder if I could have implemented the concurrency better....
ASP.NET .NET 4, C#
Basically, it generates n number of sql statements on the fly (approx 50 at the moment) and then runs them concurrently and writes the data to .csv files.
EDIT: First I create a thread to do all the work on so the page request can return. Then on that thread...
For each of the SQL statements I create a new Task using the TPL and execute it using a datareader and write the data to disk. When the last file is created I write some summary data to a summary file and zip it all up and give it to the user.
Should I have used Threads or Asynchronous Delegates instead?
I haven't posted code as I am really just wondering if my overall approach (i.e. TPL) is the best option in this situation.
Please don't lecture me about creating dynamic sql, it is totally necessary due to the technicalities of the database I am reading from and not relevant to the question. (Its the back end of a proprietary system. Got 7 thousand+ tables).
Should I have used Threads or Asynchronous Delegates instead?
Apparently, your background thread operation spans across the boundaries of a single HTTP request. In this case, it doesn't really matter what API you use to run such operation: Task.Run, Delegate.BeginInvoke, ThreadPool.QueueUserWorkItem, new Thread or anything else.
You shouldn't be running a lengthy background thread operation, which lifetime spans multiple HTTP requests, inside ASP.NET address space. While it's relatively easy to implement, this approach may have issues with IIS maintainability, scalability and security. Create a WCF service for that and call it from your ASP.NET page:
How to: Host a WCF Service in a Managed Windows Service.
If we start a new thread in ASP.Net from the thread which is serving the http request, and new thread has an unhandled exception, the worker process will crash immediately. Even if we use WCF service and call that from ASP.Net the ASP.Net thread is going to wait for the result. So better use any queuing mechanism so that the requests is in queue and queue can process in a different time based on the processing capacity. Of course when we say queuing we need to think about queue failure, requeue etc...But its worth if the application is big and needs to scale.
I have an IIS hosted WCF webservice.
It has a method on it (let's call it "ConfirmOrder"). When this method is called, I want to
1. Do some quick stuff to the database, resulting in an OrderId
2. Start a new thread that will do some slow work (e.g. generate an email and send it)
3. Return the OrderId from 1. synchronously to the client.
4. Eventually, when it's finished, the new thread created in 2. will have done all the rest of the processing and sent the email.
Questions:
(1) I did have code like:
// do printing and other tasks
OrderConfirmedThreadHelper helper = new OrderConfirmedThreadHelper(userSession, result);
// some things first (like generating barcodes) in this thread
Logger.Write(basket.SessionId, String.Format("Before ConfirmOrderSync"), LogCategoryEnum.Sales, System.Diagnostics.TraceEventType.Verbose);
helper.ConfirmOrderSync();
Logger.Write(basket.SessionId, String.Format("After ConfirmOrderSync"), LogCategoryEnum.Sales, System.Diagnostics.TraceEventType.Verbose);
// slower things (like rendering, sending email) in a separate thread
Thread helperThread = new Thread(new ThreadStart(helper.ConfirmOrderAsync));
helperThread.Start();
return result;
but it seemed to cause problems; at least, the service kept locking up. Is this a bad thing to do?
(2) I tried changing it to
// slower things (like rendering, sending email) in a separate thread
ThreadPool.QueueUserWorkItem(new WaitCallback(helper.ConfirmOrderAsync));
but the ThreadPool thread seems to be being killed as soon as the main thread has finished, because it's a Background thread.
Is there a better way of doing this - short of writing a whole new windows service to communicate with?
If the second thead finishes after the request thread (the one that comes from the browser) you're in problems, since it'll get reclaimed by the runtime and terminated.
If you can afford to wait (if it's only going to send an email i'll be a couple of seconds) you can use ManualResetEvent to synchronize one thread to wait for the other to finish and clean up gracefully.
If you can't wait, well the best choice in this case for the mail process is one of the following
A Windows Service.
An .ashx you can call from your client code with a jquery ajax call passing the necessary data to send the mail.
A batch job (a scheduled task, a sql server job, etc) that reads pending mails to be sent from the DB and sends them. It would run every X minutes, so you wouldn't have to worry
Hope that helps!
Problem: Some 300 candidates make a test using Flex. A test consist of some 100 exercises. After each exercise a .NET service is called to store the result. If a candidate finishes a test, all the data of his/her test is denormalized by Asp.NET. This denormalization can take some cpu and can take 5 to 10 seconds. Now, most of the times, some of the candidates have finished their test earlier than the rest, but still some 200 of them wait until their time is up. At that moment, 200 candidates finish their test and 200 sessions are denormalized at the same time. At this point, server load (cpu) is too high and cause calls to the webserver to go wrong. Now, instead of all these sessions being normalized concurrently, I would like to add them to a queue using MSMQ.
Question:
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax that listens to the queue? If there are messages, they are dealt one at the time.
Is it necessary to do this in a separate thread? What if in the global.asax you just call a singleton for instance that starts listening to the queue? In what thread will this singleton run? (what's the thread that calls global.asax)
What are best practices to implement this? Links? Resources? Tutorials? Examples?
I don't like the idea, but could you put an exe on the root of your website, an exe that starts a process listening to the queue...
If you get a message out of the queue, do you remove it when you pull it out or do you remove it if denormalization for this session was successful? If you remove it when you pull it out and something goes wrong...
I could also create my own queue in memory, but restarting the webserver would empty the queue and a lot of sessions would end up not being normalized, so I guess this is really a bad idea.
Is MSMQ a good choice or are there better alternatives?
You could consider using a WCF-Service with MSMQ transport. I used this approach in an application that calculates commissions:
User completes asp.net wizard configuring calculation parameters
Calculation Job is sent to WCF-Service using MSMQ transport
Service transaction is completed as soon as Job entered MSMQ
New transaction scope is created for processing Job instances
One drawback is that the transaction will require MSDTC which will add some overhead when targeting MS SQL Server and even more when dealing with Oracle.
IDesign provides a lot of useful samples and best practices on WCF queueing.
Personally, I use a servicebus for scenario's like that. I know this sounds like an overkill, but I think the .net servicebusses are so good that they require the least amount of code written by you, because it's not easy to create a good scheduler for background processes without disturbing the threads of the application pool the webapp is running in. NServicebus and MassTransit are both good an well enough documented servicebuses for your scenario. With a servicebus, you have a framework that writes to msmq and listens to msmq in several apps connected by the messagequeue. The bus makes it easy for you to create a separate app that runs as a background service and is connected with your web-app by the message queue. When you use topself (included in nservicebus and masstransit), an installer/uninstaller for the seperate apps is automatically generated by the service bus.
Question: Why don't you like the idea of having a separate exe?
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax
that listens to the queue? If there are messages, they are dealt one at
the time.
Is it necessary to do this in a separate thread? What if in the
global.asax you just call a singleton for instance that starts listening to
the queue? In what thread will this singleton run? (what's the thread that
calls global.asax)
[skip]
I don't like the idea, but could you put an exe on the root of your website, an exe that > starts a process listening to the queue...
Normally another program processes the queue - not ASP.NET. Either a windows service or an executable that you run under a scheduler (and there's no reason to put it in the root of your website).
If you get a message out of the queue, do you remove it when you pull
it out or do you remove it if denormalization for this session was
successful? If you remove it when you pull it out and something goes
wrong...
For critical work, you perform a transactional read. Items aren't removed from the queue until you commit your read operation, but while the transaction is open, no other process can get the item.
What are best practices to implement this? Links? Resources? Tutorials? Examples?
This tutorial is a good introduction and John Breakwell's blog is excellent and offers a lot of good links (including the ones in his easy-to-find sidebar "MSMQ Documentation").