I have a .net mvc / sql server website for a high-traffic application. I'm using async database connections.
Prior to launch I ran a load test, and it blew up pretty quickly under load. The connection pool quickly ran out of connections.
Running sp_who at the database level shows a pile of connections sleeping awaiting command. If I switch to non-async, this does not happen.
So, it appears that new database calls are not using connections sitting in the connection pool, instead they are opening their own new connection. This quickly exhausts the connection pool limit, and begins timing out queries.
The following is the helper I'm using to execute an async datareader. Does anyone see any issue here?
private async Task<List<T>> ExecuteReaderAsync<T>(SqlCommand command, Func<SqlDataReader, T> rowConverter)
{
List<T> ret = new List<T>();
using (SqlConnection connection = new SqlConnection(this.ConnectionString))
{
command.Connection = connection;
await connection.OpenAsync();
using (SqlDataReader reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
ret.Add(rowConverter(reader));
}
reader.Close();
}
connection.Close();
}
return ret;
}
I'm calling the code like the datareader helper like the following:
public async Task<Content> FindContentAsync(int id)
{
Content content = null;
using (SqlCommand command = CreateProcedure("dbo.FindContent"))
{
AddParam(command, "Id", SqlDbType.Int, id);
List<Content> items = await ExecuteReaderAsync<Content>(command, x => BindContent(x));
if (items.Count > 0)
{
content = items[0];
}
}
return content;
}
And calling that from a helper:
public async Task<Content> FindAsync(int id)
{
var db = new DataAccess();
var content = await db.FindContentAsync(id);
return content;
}
Related
I have this method:
public async static Task ExecuteSpAndForget(string sp, Location location, params SqlParameter[] parameters)
{
using (SqlConnection conn = await GetConnectionAsync(location))
{
SqlCommand cmd = new SqlCommand(sp, conn);
cmd.Parameters.AddRange(parameters);
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandTimeout = Int32.Parse(ConfigurationManager.AppSettings["CommandTimeout"]);
await cmd.ExecuteNonQueryAsync().ConfigureAwait(false);
}
}
public async static Task<SqlConnection> GetConnectionAsync(Location location)
{
SqlConnection conn = new SqlConnection(ConfigurationManager.AppSettings["ConnectionString" + location]);
await conn.OpenAsync().ConfigureAwait(false);
return conn;
}
And in my log filter I applied to WebApiConfig.cs:
config.Filters.Add(new LogFilter());
And in LogFilter(), I call the database to write log
public static int WriteLog(HttpActionContext actionContext, Exception exception = null)
{
\\logic
var logTask = DatabaseHelper.ExecuteSpAndForget("writelogSP", Location.Log, sqlParams.ToArray());
\\return logic
}
When I am debugging locally, this never writes to the database, when I deployed it to IIS, it sometimes write to the database, sometimes not. I tested it locally in a console app, if I wait several seconds after calling before exit, data can be inserted into the database, otherwise no insert happens if no wait.
Why? How can I use it?
What you are referring to is a background task.
There are different ways of going about it. The simplest one is Task.Run with no await
public static int WriteLog(HttpActionContext actionContext, Exception exception = null)
{
\\logic
// fire and forget, no await
var logTask = Task.Run(async () =>
{
await DatabaseHelper.ExecuteSpAndForget("writelogSP", Location.Log, sqlParams.ToArray());
});
\\return logic;
}
You can check out this link https://blog.stephencleary.com/2014/06/fire-and-forget-on-asp-net.html for a better solution.
My user send dynamic entity from client-project so, I have to write methods like this
public Task<TUser> FindByNameAsync(string normalizedUserName, CancellationToken cancellationToken)
{
cancellationToken.ThrowIfCancellationRequested();
throw new NotImplementedException();
//string sql = "SELECT * FROM \"IdentityUsers\" WHERE \"NormalizedUserName\" = #NormalizedUserName;";
//using (var connection = _databaseConnectionFactory.CreateConnectionAsync())
//{
// connection.QueryFirstOrDefaultAsync<TUser>(sql,
// new { NormalizedUserName = normalizedUserName });
//}
}
My IDatabaseConnectionFactory class bind ConnectionString like below:
public interface IDatabaseConnectionFactory
{
Task<IDbConnection> CreateConnectionAsync();
}
public class ConnectionFactory : IDatabaseConnectionFactory
{
private readonly string _connectionString;
public ConnectionFactory(string connectionString) => _connectionString = connectionString ?? throw new ArgumentNullException(nameof(connectionString));
public async Task<IDbConnection> CreateConnectionAsync()
{
try
{
var connString = new NpgsqlConnection(_connectionString);
await connString.OpenAsync();
return connString;
}
catch
{
throw;
}
}
}
Now, how can I execute following query using generic-type entity TUser
string sql = "SELECT * FROM \"IdentityUsers\" WHERE \"NormalizedUserName\" = #NormalizedUserName;";
using (var connection = _databaseConnectionFactory.CreateConnectionAsync())
{
connection.QueryFirstOrDefaultAsync<TUser>(sql,
new { NormalizedUserName = normalizedUserName });
}
Note: QueryFirstOrDefaultAsync not found under connection here
You aren't awaiting the CreateConnectionAsync. Unfortunately it isn't obvious in this case, because Task<T> is disposable (so the using doesn't complain); try instead:
using (var connection = await _databaseConnectionFactory.CreateConnectionAsync())
{
var user = await connection.QueryFirstOrDefaultAsync<TUser>(sql,
new { NormalizedUserName = normalizedUserName });
}
As a tip: the compiler output (against the original code) helps make this clear:
Error CS1929 'Task<IDbConnection>' does not contain a definition for 'QueryFirstOrDefaultAsync' and the best extension method overload 'SqlMapper.QueryFirstOrDefaultAsync<TUser>(IDbConnection, string, object, IDbTransaction, int?, CommandType?)' requires a receiver of type 'IDbConnection'
which tells us that:
it found some QueryFirstOrDefaultAsync method, but it wasn't usable, because
the target expression is a Task<IDbConnection>, not an IDbConnection
As a side note: it is worth knowing that if you're only doing one operation with the connection, Dapper can deal with opening and closing the connection for you - which can help reduce the number of async/await operations. Consider, for example, if you had a CreateClosedConnection() method that did not open the connection, and thus had no need to be async; the following would still work:
using (var connection = _databaseConnectionFactory.CreateClosedConnection())
{
var user = await connection.QueryFirstOrDefaultAsync<TUser>(sql,
new { NormalizedUserName = normalizedUserName });
}
with Dapper dealing with the await OpenAsync() for you as part of the QueryFirstOrDefaultAsync.
I'm using the Dapper ORM in an ASP.Net Core 2.1 CRUD application, with a SQL Server database. For a new form I'm developing, I want to retrieve all reference data (to populate SELECT field options) in one go when the screen is invoked. This involves a series of stored procedure calls, which I want to do asynchronously for best performance.
This Dapper tutorial suggests I should look at using QueryMultipleAsync, but I can't find an example of its use with stored procedures, only hard-coded SQL statements.
My C# code currently looks like this:
public async Task<ContactReferenceData> Get()
{
ContactReferenceData refData = new ContactReferenceData();
try
{
using (IDbConnection dbConnection = _connection)
{
dbConnection.Open();
var countryData = await dbConnection.QueryAsync<Country>(sql: "usp_GetCountries", commandType: CommandType.StoredProcedure);
refData.CountryDetails = countryData.AsList();
var companyData = await dbConnection.QueryAsync<Company>(sql: "usp_GetCompanies", commandType: CommandType.StoredProcedure);
refData.CompanyDetails = companyData.AsList();
var groupData = await dbConnection.QueryAsync<Group>(sql: "usp_GetGroups", commandType: CommandType.StoredProcedure);
refData.GroupDetails = groupData.AsList();
var groupPositionData = await dbConnection.QueryAsync<GroupPosition>(sql: "usp_GetGroupPositions", commandType: CommandType.StoredProcedure);
refData.GroupPositionDetails = groupPositionData.AsList();
}
return refData;
}
catch (Exception ex)
{
_logger.LogError(ex.ToString());
throw;
}
}
This works OK in my test environment, but I'm not sure it's the correct way to execute async queries. In particular, I have the following concerns:
is it robust enough to be trusted in live operation?
in its current form, is it maximising the benefits (if any) of asynchronous operation, or should I be using QueryMultipleAsync to properly achieve this?
Have you tried something like this?
public async Task<ContactReferenceData> Get()
{
var sql = "EXEC usp_GetCountries; EXEC usp_GetCompanies; EXEC usp_GetGroups; EXEC usp_GetGroupPositions";
ContactReferenceData refData = new ContactReferenceData();
try
{
using (IDbConnection dbConnection = _connection)
{
using (var multi = connection.QueryMultipleAsync(sql: sql, commandType: CommandType.StoredProcedure ).Result)
{
refData.CountryDetails = multi.Read<countryDetails>().ToList();
refData.CompanyDetails = multi.Read<CompanyDetails>().ToList();
refData.GroupData = multi.Read<Groups>().ToList();
refData.GroupPositionsData= multi.Read<GroupPositions>().ToList();
}
}
return refData;
}
catch (Exception ex)
{
_logger.LogError(ex.ToString());
throw;
}
}
I have an asp.net MVC website which is consuming a rest api to receive it's data. I'm using asynchronous tasks to perform the requests as there can be many on each page. After a while of uptime the website has been throwing the following error when trying to receive data.
The underlying connection was closed: An unexpected error occurred on a send.
I read that this could be due to the maxconnection settings on the web.config but increasing this doesn't seem to make much difference.
I'm also using caching to reduce the load on the api. The task is cached so the result can be used later.
The only way I've found to fix this is by recycling the application pool. Any help would be appreciated.
/* Code from page_load */
var currenciesTask = ApiClient.GetAsync<CurrencyListWrapper>("currencies");
var blogArticleTask = ApiClient.GetAsync<BlogArticleListWrapper>("blog/articles", "limit=10");
var seoPageTask = ApiClient.GetAsync<SEOPageListWrapper>("seopages");
await Task.WhenAll(currenciesTask, blogArticleTask, seoPageTask);
/* Code from data access later */
public class ApiClient : HttpClient
{
public static Task<T> GetAsync<T>(string operation, string query = null, bool cache = true)
{
// Check if task is in cache
string cacheName = null;
if (cache)
{
cacheName = String.Format("{0}_{1}_{2}", operation, query ?? String.Empty, App.GetLanguage());
var cachedTask = HttpRuntime.Cache[cacheName];
if (cachedTask != null)
{
return (Task<T>)cachedTask;
}
}
// Get data task
var task = GetAsyncData<T>(operation, query);
// Add to cache if required
if (task != null && cache)
{
App.AddToCache(cacheName, task);
}
return task;
}
public static async Task<T> GetAsyncData<T>(string operation, string query = null)
{
using (ApiClient client = new ApiClient())
{
string url;
if (query != null)
{
url = String.Format("{0}?{1}", operation, query);
}
else
{
url = String.Format("{0}", operation);
}
var response = await client.GetAsync(url);
return (await response.Content.ReadAsAsync<T>());
}
}
}
This is wrong,
The task is cached so the result can be used later.
You are supposed to cache result, not the task. At end of first execution, your HttpClient is closed and when you try to retrieve cached task, it will not work.
public class ApiClient : HttpClient
{
public static async Task<T> GetAsync<T>(string operation, string query = null, bool cache = true)
{
// Check if task is in cache
string cacheName = null;
if (cache)
{
cacheName = String.Format("{0}_{1}_{2}", operation, query ?? String.Empty, App.GetLanguage());
T cachedResult = (T)HttpRuntime.Cache[cacheName];
if (cachedResult!= null)
{
return Task.FromResult(cachedResult);
}
}
// Get data task
var result = await GetAsyncData<T>(operation, query);
// Add to cache if required
if (result != null && cache)
{
App.AddToCache(cacheName, result);
}
return result;
}
public static async Task<T> GetAsyncData<T>(string operation, string query = null)
{
using (ApiClient client = new ApiClient())
{
string url;
if (query != null)
{
url = String.Format("{0}?{1}", operation, query);
}
else
{
url = String.Format("{0}", operation);
}
var response = await client.GetAsync(url);
return (await response.Content.ReadAsAsync<T>());
}
}
}
Akash could be right.
But it seems more or less connection issue with application pool. Set the connection limit 0 to make it unlimited at application pool.
Have a finally block in you code, and
gc.collect();
garbage collection method to be called to remove unused connections to make space for other connection.
I have been experimenting with WP7 apps today and have hit a bit of a wall.
I like to have seperation between the UI and the main app code but Ive hit a wall.
I have succesfully implemented a webclient request and gotten a result, but because the call is async I dont know how to pass this backup to the UI level. I cannot seem to hack in a wait for response to complete or anything.
I must be doing something wrong.
(this is the xbox360Voice library that I have for download on my website: http://www.jamesstuddart.co.uk/Projects/ASP.Net/Xbox_Feeds/ which I am porting to WP7 as a test)
here is the backend code snippet:
internal const string BaseUrlFormat = "http://www.360voice.com/api/gamertag-profile.asp?tag={0}";
internal static string ResponseXml { get; set; }
internal static WebClient Client = new WebClient();
public static XboxGamer? GetGamer(string gamerTag)
{
var url = string.Format(BaseUrlFormat, gamerTag);
var response = GetResponse(url, null, null);
return SerializeResponse(response);
}
internal static XboxGamer? SerializeResponse(string response)
{
if (string.IsNullOrEmpty(response))
{
return null;
}
var tempGamer = new XboxGamer();
var gamer = (XboxGamer)SerializationMethods.Deserialize(tempGamer, response);
return gamer;
}
internal static string GetResponse(string url, string userName, string password)
{
if (!string.IsNullOrEmpty(userName) && !string.IsNullOrEmpty(password))
{
Client.Credentials = new NetworkCredential(userName, password);
}
try
{
Client.DownloadStringCompleted += ClientDownloadStringCompleted;
Client.DownloadStringAsync(new Uri(url));
return ResponseXml;
}
catch (Exception ex)
{
return null;
}
}
internal static void ClientDownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
if (e.Error == null)
{
ResponseXml = e.Result;
}
}
and this is the front end code:
public void GetGamerDetails()
{
var xboxManager = XboxFactory.GetXboxManager("DarkV1p3r");
var xboxGamer = xboxManager.GetGamer();
if (xboxGamer.HasValue)
{
var profile = xboxGamer.Value.Profile[0];
imgAvatar.Source = new BitmapImage(new Uri(profile.ProfilePictureMiniUrl));
txtUserName.Text = profile.GamerTag;
txtGamerScore.Text = int.Parse(profile.GamerScore).ToString("G 0,000");
txtZone.Text = profile.PlayerZone;
}
else
{
txtUserName.Text = "Failed to load data";
}
}
Now I understand I need to place something in ClientDownloadStringCompleted but I am unsure what.
The problem you have is that as soon as an asynchronous operation is introduced in to the code path the entire code path needs to become asynchronous.
Because GetResponse calls DownloadStringAsync it must become asynchronous, it can't return a string, it can only do that on a callback
Because GetGamer calls GetResponse which is now asynchronous it can't return a XboxGamer, it can only do that on a callback
Because GetGamerDetails calls GetGamer which is now asynchronous it can't continue with its code following the call, it can only do that after it has received a call back from GetGamer.
Because GetGamerDetails is now asynchronous anything call it must also acknowledge this behaviour.
.... this continues all the way up to the top of the chain where a user event will have occured.
Here is some air code that knocks some asynchronicity in to the code.
public static void GetGamer(string gamerTag, Action<XboxGamer?> completed)
{
var url = string.Format(BaseUrlFormat, gamerTag);
var response = GetResponse(url, null, null, (response) =>
{
completed(SerializeResponse(response));
});
}
internal static string GetResponse(string url, string userName, string password, Action<string> completed)
{
WebClient client = new WebClient();
if (!string.IsNullOrEmpty(userName) && !string.IsNullOrEmpty(password))
{
client.Credentials = new NetworkCredential(userName, password);
}
try
{
client.DownloadStringCompleted += (s, args) =>
{
// Messy error handling needed here, out of scope
completed(args.Result);
};
client.DownloadStringAsync(new Uri(url));
}
catch
{
completed(null);
}
}
public void GetGamerDetails()
{
var xboxManager = XboxFactory.GetXboxManager("DarkV1p3r");
xboxManager.GetGamer( (xboxGamer) =>
{
// Need to move to the main UI thread.
Dispatcher.BeginInvoke(new Action<XboxGamer?>(DisplayGamerDetails), xboxGamer);
});
}
void DisplayGamerDetails(XboxGamer? xboxGamer)
{
if (xboxGamer.HasValue)
{
var profile = xboxGamer.Value.Profile[0];
imgAvatar.Source = new BitmapImage(new Uri(profile.ProfilePictureMiniUrl));
txtUserName.Text = profile.GamerTag;
txtGamerScore.Text = int.Parse(profile.GamerScore).ToString("G 0,000");
txtZone.Text = profile.PlayerZone;
}
else
{
txtUserName.Text = "Failed to load data";
}
}
As you can see async programming can get realy messy.
You generally have 2 options. Either you expose your backend code as an async API as well, or you need to wait for the call to complete in GetResponse.
Doing it the async way would mean starting the process one place, then return, and have the UI update when data is available. This is generally the preferred way, since calling a blocking method on the UI thread will make your app seem unresponsive as long as the method is running.
I think the "Silverlight Way" would be to use databinding. Your XboxGamer object should implement the INotifyPropertyChanged interface. When you call GetGamer() it returns immediately with an "empty" XboxGamer object (maybe with GamerTag=="Loading..." or something). In your ClientDownloadStringCompleted handler you should deserialize the returned XML and then fire the INotifyPropertyChanged.PropertyChanged event.
If you look at the "Windows Phone Databound Application" project template in the SDK, the ItemViewModel class is implemented this way.
Here is how you can expose asynchronous features to any type on WP7.