Entity Framework Database Initialization: Timeout when initializing new Azure SqlDatabase - asp.net

I have an ASP.NET MVC application. When a new customer is created via CustomerController I run a new background task (using HostingEnvironment.QueueBackgroundWorkItem) to create a new Azure SqlDatabase for that customer.
I use Entity Framework Code First to create/initialize the new database. Here's the code:
// My ConnectionString
var con = "...";
// Initialization strategy: create db and execute all Migrations
// MyConfiguration is just a DbMigrationsConfiguration with AutomaticMigrationsEnabled = true
Database.SetInitializer(strategy: new MigrateDatabaseToLatestVersion<CustomerDataContext, MyConfiguration>(useSuppliedContext: true));
using (var context = new CustomerDataContext(con))
{
// Neither 'Connection Timeout=300' in ConnectionString nor this line helps -> TimeoutException will rise after 30-40s
context.Database.CommandTimeout = 300;
// create the db - this lines throws the exception after ~40s
context.Database.Initialize(true);
}
My Problem is that I always get a TimeoutException after about 40secs. I think that happens because Azure cannot initialize the new database within this short period of time. Don't get me wrong: The database will be created well by Azure but I want to wait for that point / get rid of the TimeoutException.
Edit1:
I'm using Connection Timeout=300 in my ConnectionString but my app doesn't really care about that; after about 40s I'm always running into an SqlError.
Edit2:
The exception that raises is an SqlException. Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Source: .Net SqlClient Data Provider
Edit3:
I can confim now that this has nothing to do with ASP.NET/IIS. Even in a simple UnitTest method the code above fails.

It seems that there is another CommandTimeout setting that is involved in database initialization process when using Code First Migrations. I want so share my solution here just in case anybody encounters this problem too.
Thanks to Rowan Miller for his hint pointing me to the solution.
Here's my code:
// Initialisation strategy
Database.SetInitializer(strategy: new CreateDatabaseIfNotExists<MyDataContext>());
// Use DbContext
using (var context = new MyDataContext(myConnectionString))
{
// Setting the CommandTimeout here does not prevent the database
// initialization process from raising a TimeoutException when using
// Code First Migrations so I think it's not needed here.
//context.Database.CommandTimeout = 300;
// this will create the database if it does not exist
context.Database.Initialize(force: false);
}
And my Configuration.cs class:
public sealed class Configuration : DbMigrationsConfiguration<MyDataContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
AutomaticMigrationDataLossAllowed = false;
// Very important! Gives me enough time to wait for Azure
// to initialize (Create -> Migrate -> Seed) the database.
// Usually Azure needs 1-2 minutes so the default value of
// 30 seconds is not big enough!
CommandTimeout = 300;
}
}

The command timeout and the connection timeout are two different settings. In this case you only increase the commandtimeout. You can increase the connection timeout in the web.config: Connection Timeout=120. The only time you want to increase the connection timeout is when you are creating the database.

Related

Add Quartz Job&Trigger to running Razor Pages application

I have a razor pages app that implements Quartz.NET to store jobs in a MySQL Database. The implementation works fine so far, I can connect to the db, store jobs and they are executed at the specified times. The issue I'm having currently is that I need to schedule and execute jobs based on user inputs(without restarting the app) and that I can't get to work. I'm very new to Quartz&Asp.net and I haven't been coding for very long either, so apologies if I've made any stupid mistakes.
I've read somewhere that I shouldn't initialize multiple schedulers so I've tried storing the scheduler object I've got so I can access and use it later. However when I try to access it from another class later then I get a Null reference exception. Tbh, this feels like it shouldn't even work so I'm not surprised it doesn't...can anyone please look at my code below and tell me if this can work? Or is there a better way to do this?
I've found one other solution where they basically create a job on startup that periodically checks a db for new jobs and adds them to the scheduler. I guess that would work, seems a bit clunky, though. Plus it's from 10 years ago so maybe there's a better way today? How to add job with trigger for running Quartz.NET scheduler instance without restarting server?
One other idea I've had was to open(and close) a new app whenever I need to create a job. I'm not sure I like that idea but seems less resource intensive than the recurring job described above. Would that be a viable option?
The code for my current solution:
Scheduler:
//Creating Scheduler
Scheduler = await schedulerFactory.GetScheduler();
Scheduler.JobFactory = jobFactory;
var key = new JobKey("Notify Job", "DEFAULT");
if (key == null)
{
//Create Job
IJobDetail jobDetail = CreateJob(jobMetaData);
//Create Trigger
ITrigger trigger = CreateTrigger(jobMetaData);
//Schedule Job
//await Scheduler.ScheduleJob(jobDetail, trigger, cancellationToken);
await Scheduler.AddJob(jobDetail, true);
}
//Start Scheduler
await Scheduler.Start(cancellationToken);
//Copying the scheduler object into a different class where it's easier to access.
ScheduleStore scheduleStore = new ScheduleStore();
scheduleStore.tempScheduler = Scheduler;
ScheduleStore:
public class ScheduleStore
{
public IScheduler tempScheduler { get; set; }
public ScheduleStore()
{
}
}
runtime Scheduler:
public class RunningScheduler : IHostedService {
public IScheduler scheduler { get; set; }
private readonly JobMetadata jobMetaData;
public RunningScheduler(JobMetadata job)
{
ScheduleStore scheduleStore = new ScheduleStore();
this.scheduler = scheduleStore.tempScheduler;
this.jobMetaData = job;
}
public async Task StartAsync(CancellationToken cancellationToken)
{
IJobDetail jdets = CreateJob(jobMetaData);
if (jobMetaData.CronExpression == "--")
{
ITrigger jtriggz = CreateSimpleTrigger(jobMetaData);
//the next line throws the exception.
await scheduler.ScheduleJob(jdets, jtriggz, cancellationToken);
//It's definitely the scheduler that's throwing the null pointer exception.
}
// the else does basically the same as the if, only with a cron trigger instead of a simple one so I've omitted it.
I see that you are using a hosted service. Have you noticed that Quartz has that support already built-in?
Quartz cannot handle new jobs "new code that runs" dynamically, but triggers for sure. You just need to obtain a reference to IScheduler and then you can add new triggers pointing to existing job or just call scheduler.TriggerJob which will call your job once with given parameters (job data map is powerful feature to pass execution parameters).
I'd advice checking the GitHub repository and its examples, there a specific ones for different features and ASP.NET Core and worker integrations.
Generatlly Quartz already has database persistence support which you can use. Just call scheduler methods to add jobs and triggers - they will be persisted and available between application restarts (and take effect immediately without the need for restart).

Not able to read uncommitted offsets after restart Listener in Spring boot

I am trying to commit the offsets in Kafka on basis of certain conditions.
Here is my listener code.
#KafkaListener(topics = "test")
public void getTopics(#RequestBody String emp,Acknowledgment acknowledgment) {
Employee model = gson.fromJson(emp, Employee.class);
if(model.getId()%2!=0)
{
System.out.println("Kafka event consumed is: " + emp);
acknowledgment.acknowledge();
}
else {
System.out.println("Model converted value: " + model);
}
}
Here is my application.prop file configurations
spring.kafka.listener.ack-mode=manual-immediate
spring.kafka.consumer.auto-offset-reset=earliest
At first when i start the springboot app then i am passing some even employee id in topic.
Then i restart my program after making if(model.getId()%2==0) which is opposite condition.I am able to fetch the values which i had not commited first.
I retry the same process by adding spring.kafka.consumer.enable-auto-commit=false in application.properties.But this time consumer doesn't retry for earlier.I had thought that i should have been able to get what i was getting earlier and in first case it should not have worked as auto-commit should be false for committing.
Thanks for any advice.

Hangfire causes Timeout expired exception when executing SQL Code to optimzie database - ASP.NET Core

I have the following code setup as a scheduled task:
public class OptimizeDatabase : IJob {
#region Constructor
public OptimizeDatabase(DataContext dataContext) {
DbContext = dataContext;
}
#endregion
#region Fields
private readonly DataContext DbContext;
#endregion
#region Methods
public async Task Execute() {
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
string result = "Ok";
try {
// Rebuild Indexes
DbContext.Database.ExecuteSqlCommand("EXEC sp_MSforeachtable \"ALTER INDEX ALL ON ? REBUILD WITH (ONLINE=OFF)\"");
// Update Statistics
DbContext.Database.ExecuteSqlCommand("EXEC sp_updatestats;");
}
catch (Exception ex) {
result = ex.Message + Environment.NewLine + ex.StackTrace;
}
stopWatch.Stop();
DbContext.TaskLogs.Add(new TaskLog {
Date = DateTime.Now,
ElapsedSeconds = stopWatch.Elapsed.TotalSeconds,
Result = result,
Task = "Optimize Database"
});
await DbContext.SaveChangesAsync();
}
#endregion
}
And it's configured to run in Startup.cs
RecurringJob.AddOrUpdate<OptimizeDatabase>(x => x.Execute(), Cron.Daily(10));
All other scheduled tasks execute without issue, however, this one always throws the following error:
Timeout expired. The timeout period elapsed prior to completion of the
operation or the server is not responding. at
System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
Boolean breakConnection, Action`1 wrapCloseInAction)
Any ideas or insights are appreciated.
The answer to your question is pretty simple. You are trying to rebuild the indices for all tables in your database and executing this statement via a hangfire job. The Hangfire jobs now try to rebuild the index for their own tables and that creates a deadlock.
You have to rebuild the indices explicitly for all your tables one after one like:
ALTER INDEX ALL ON [dbo].[A] REBUILD;
ALTER INDEX ALL ON [dbo].[B] REBUILD;
The timeout issue is because one of the queries is taking longer that it should.
In .Net there are 2 timeout as far as I know, the connection timeout (ConnectionTimeout Property) and the command timeout (CommandTimeout Property). Both timeout default time is 30 seconds.
I recommend you to:
Run your queries at the SQL Management Studio to have an idea about the time needed to run both queries in sequences as your code shows.
Set the connection timeout and command timeout when the amount of second from the previous run at SQL Management Studio. If the timeout continues to appears, try adding more time to each timeout, add 30 seconds to each timeout until you find the minimal needed time to execute your queries. Once you find it, add 30 seconds extra to each timeout just to be sure.
Taking part of your code, the change would be something like:
try {
// Change CommandTimeout
DbContext.Database.CommandTimeout = 120;
// Rebuild Indexes
DbContext.Database.ExecuteSqlCommand("EXEC sp_MSforeachtable \"ALTER INDEX ALL ON ? REBUILD WITH (ONLINE=OFF)\"");
// Update Statistics
DbContext.Database.ExecuteSqlCommand("EXEC sp_updatestats;");
}
catch (Exception ex) {
result = ex.Message + Environment.NewLine + ex.StackTrace;
}
You can take a look at this article, it explains good scenarios of possible timeout causes https://stackoverflow.com/a/8603111/2654879 .
We had face same issue, where we have to do somany table operations in a single transaction scope. The hangfire jobs get failed with the same error when multiple jobs runs parallel. The way we solved this issue is by increase the command timeout value(By default it's 30 second)

How to find leaking db connection pool handle?

I'm seeing the dreaded "The timeout period elapsed prior to obtaining a connection from the pool" error.
I've searched the code for any unclosed db connections, but couldn't find any.
What I want to do is this: the next time we get this error, have the system dump a list of which procs or http requests are holding all the handles, so I can figure out which code is causing the problem.
Even better would be to see how long those handles had been held, so I could spot used-but-unclosed connections.
Is there any way to do this?
If you are lucky enough that connection creation/opening is centralized then the following class should make it easy to spot leaked connections. Enjoy :)
using System.Threading; // not to be confused with System.Timer
/// <summary>
/// This class can help identify db connection leaks (connections that are not closed after use).
/// Usage:
/// connection = new SqlConnection(..);
/// connection.Open()
/// #if DEBUG
/// new ConnectionLeakWatcher(connection);
/// #endif
/// That's it. Don't store a reference to the watcher. It will make itself available for garbage collection
/// once it has fulfilled its purpose. Watch the visual studio debug output for details on potentially leaked connections.
/// Note that a connection could possibly just be taking its time and may eventually be closed properly despite being flagged by this class.
/// So take the output with a pinch of salt.
/// </summary>
public class ConnectionLeakWatcher : IDisposable
{
private readonly Timer _timer = null;
//Store reference to connection so we can unsubscribe from state change events
private SqlConnection _connection = null;
private static int _idCounter = 0;
private readonly int _connectionId = ++_idCounter;
public ConnectionLeakWatcher(SqlConnection connection)
{
_connection = connection;
StackTrace = Environment.StackTrace;
connection.StateChange += ConnectionOnStateChange;
System.Diagnostics.Debug.WriteLine("Connection opened " + _connectionId);
_timer = new Timer(x =>
{
//The timeout expired without the connection being closed. Write to debug output the stack trace of the connection creation to assist in pinpointing the problem
System.Diagnostics.Debug.WriteLine("Suspected connection leak with origin: {0}{1}{0}Connection id: {2}", Environment.NewLine, StackTrace, _connectionId);
//That's it - we're done. Clean up by calling Dispose.
Dispose();
}, null, 10000, Timeout.Infinite);
}
private void ConnectionOnStateChange(object sender, StateChangeEventArgs stateChangeEventArgs)
{
//Connection state changed. Was it closed?
if (stateChangeEventArgs.CurrentState == ConnectionState.Closed)
{
//The connection was closed within the timeout
System.Diagnostics.Debug.WriteLine("Connection closed " + _connectionId);
//That's it - we're done. Clean up by calling Dispose.
Dispose();
}
}
public string StackTrace { get; set; }
#region Dispose
private bool _isDisposed = false;
public void Dispose()
{
if (_isDisposed) return;
_timer.Dispose();
if (_connection != null)
{
_connection.StateChange -= ConnectionOnStateChange;
_connection = null;
}
_isDisposed = true;
}
~ConnectionLeakWatcher()
{
Dispose();
}
#endregion
}
There are some good links for monitoring connection pools. Do a google search for ".net connection pool monitoring".
One article I referred to a while back was Bill Vaughn's article (Note this is old but still contains useful info). It has some info on monitoring connection pools, but some great insights as to where leaks could be occuring as well.
For monitoring, he suggests;
"Monitoring the connection pool
Okay, so you opened a connection and closed it and want to know if the
connection is still in place—languishing in the connection pool on an
air mattress. Well, there are several ways to determine how many
connections are still in place (still connected) and even what they
are doing. I discuss several of these here and in my book:
· Use the SQL Profiler with the SQLProfiler TSQL_Replay
template for the trace. For those of you familiar with the Profiler,
this is easier than polling using SP_WHO.
· Run SP_WHO or SP_WHO2, which return information from the
sysprocesses table on all working processes showing the current status
of each process. Generally, there’s one SPID server process per
connection. If you named your connection, using the Application Name
argument in the connection string, it’ll be easy to find.
· Use the Performance Monitor (PerfMon) to monitor the pools
and connections. I discuss this in detail next.
· Monitor performance counters in code. This option permits
you to display or simply monitor the health of your connection pool
and the number of established connections. I discuss this in a
subsequent section in this paper."
Edit:
As always, check out some of the other similar posts here on SO
Second Edit:
Once you've confirmed that connections aren't being reclaimed by the pool, another thing you could try is to utilise the StateChange event to confirm when connections are being opened and closed. If you are finding that there are a lot more state changes to opened than to closed, then that would indicate that there are leaks somewhere. You could also then log the data in the statechanged event along with a timestamp, and if you have any other logging on your application, you could start to parse the log files for instances where there appears to be state changes of closed to open, with no corresponding open to closed. See this link for more info on how to handle the StateChangedEvent.
i've used this
http://www.simple-talk.com/sql/performance/how-to-identify-slow-running-queries-with-sql-profiler/
to find long running stored procedures before, i can then work back and find the method that called the SP.
dont know if that'll help

Can I rely on the using statement to close my MySQL connections?

I have an ASP.NET website that uses mysql as it's database. I notice there are a lot of connections "sleeping" when I show a full process list. Today we had some errors "The timeout period elapsed prior to obtaining a connection from the pool.". If the processes are "sleeping" then are they still open from the code? All MySQL connections in the code are in using statements. Can I rely on the using statement to properly close connections?
Edit Code:
I am using this class to create my connection:
public class DbAccess
{
public static MySqlConnection OpenConnection(string connectionStringName)
{
string connectionString = ConfigurationManager.ConnectionStrings[connectionStringName].ConnectionString;
if (string.IsNullOrEmpty(connectionString))
{
throw new ArgumentException("Connection string " + connectionStringName + " does not exist.");
}
MySqlConnection connection = new MySqlConnection(connectionString);
connection.Open();
return connection;
}
}
Then I am calling it like this:
using (MySqlConnection connection = DbAccess.OpenConnection(connectionString))
{
//Code Here
}
Some additional info: Resetting MySql did not make the errors go away, but resetting my app pool did..
C# using blocks are guaranteed to call the .Dispose() method of the object, even if an exception is thrown. That means it's safe, as long as your provider uses the .Dispose() method to close the connection. Looking in the documentation for that type, I see this excerpt (down in section 25.2.3.3.5):
From Open to Closed, using either the Close method or the Dispose method of the connection object.
This tells me you can close the connection via the Dispose method, and so a using block should be all you need.
hope this could be of any help: MySqlConnection really not close
and
Using MySQLConnection in C# does not close properly
and this one
http://social.msdn.microsoft.com/Forums/en/adodotnetdataproviders/thread/c57c0432-c27b-45ab-81ca-b2df76c911ef
yes, absolutely.
using (MySqlConnection connection = DbAccess.OpenConnection(connectionString))
{
//Code Here
}
is exactly the same as
MySqlConnection connection = null
try
{
connection = DbAccess.OpenConnection(connectionString)
//Code Here
}
finally
{
if (connection is IDisposable)
connection.Dispose
}
provided the MySqlConnection class implements IDisposable, then it'll get cleaned up properly. If you need to call another methos, such as close instead of or as well as, then consider the more verbose syntax above and add the method in the finally.

Resources