I have this problem in my ASP.NET application where I'm seeing some of my Oracle queries fired off to the server then not returning. Ever. It happens in several places in my app and I can't explain it. Here's one specific scenario where I'm seeing this behavior:
During application start-up I am pre-fetching data asynchronously into the application state (the choice was made to use app state instead of cache b/c the data never changes during the lifetime of the app).
Action<string, object> AddApplicationState = (string name, object data) =>
{
Application.Lock();
Application.Add(name, data);
Application.UnLock();
};
Func<DataTable> GetFullNames = () => Database.GetAllNames();
Func<DataTable> GetProvinceNames = () => Database.GetProvinceNames();
Func<DataTable> GetTribeNames = () => Database.GetTribeNames();
GetFullNames.BeginInvoke(result => AddApplicationState("AllNames", GetFullNames.EndInvoke(result)), null);
GetProvinceNames.BeginInvoke(result => AddApplicationState("ProvinceNames", GetProvinceNames.EndInvoke(result)), null);
GetTribeNames.BeginInvoke(result => AddApplicationState("TribeNames", GetTribeNames.EndInvoke(result)), null);
The second two return just fine, but the first either never returns or returns after about 10 minutes. After firing up Oracle SQL Developer I go to the 'monitor sessions' tool and can see a single session for the query. It looks like it has completed, b/c the wait time is (null) and the session is inactive. Here's the ADO.NET code used to query the database:
public static DataTable GetAllNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
var sql = new StringBuilder();
sql.AppendLine("SELECT NAME_ID, NATIVE_NAME, NVL(FREQUENCY,0) \"FREQUENCY\", CULTURE_ID,");
sql.AppendLine("ENGLISH_NAME, REGEXP_REPLACE(ENGLISH_NAME, '[^A-Za-z]', null) \"ENGLISH_NAME_STRIPPED\"");
sql.AppendLine("FROM NAMES");
oracmd.CommandText = sql.ToString();
var orada = new OracleDataAdapter(oracmd);
var dtAllNames = new DataTable();
orada.Fill(dtAllNames);
return dtAllNames;
}
}
}
public static DataTable GetTribeNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
var sql = new StringBuilder();
sql.AppendLine("SELECT DISTINCT NAME_ID, English_Name \"TRIBE_NAME_ENGLISH\",");
sql.AppendLine("REGEXP_REPLACE(English_Name, '[^A-Za-z]',null) \"TRIBE_ENGLISH_NAME_STRIPPED\",");
sql.AppendLine("NATIVE_NAME \"TRIBE_NATIVE_NAME\"");
sql.AppendLine("FROM NAMES");
sql.AppendLine("WHERE NAME_ID IN ");
sql.AppendLine("(SELECT NAME_ID_TRIBE FROM TRIBES UNION SELECT NAME_ID_FAMILY FROM TRIBES)");
sql.AppendLine("ORDER BY English_Name");
oracmd.CommandText = sql.ToString();
var orada = new OracleDataAdapter(oracmd);
var dt = new DataTable();
orada.Fill(dt);
return dt;
}
}
}
public static DataTable GetProvinceNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
oracmd.CommandText = "SELECT DISTINCT PROVINCE_ID, PROVINCE_NAME_NATIVE, PROVINCE_NAME_ENGLISH FROM PROVINCES";
var orada = new OracleDataAdapter(oracmd);
var dtRC = new DataTable();
orada.Fill(dtRC);
return dtRC;
}
}
}
As you can see, the ADO.NET code is pretty standard (and boring!) stuff. When run in SQL Developer, the queries return less than a second. The first query returns x rows, the second x rows, and the third x rows. But this problem of queries being fired off then never returning happens often and I can't seem to track down the issue. Anyone have any thoughts?
And finally, since I realize it could be something completely unrelated to code, I am running the app locally (from Visual Studio) on a Windows XP SP3 machine and connecting via VPN to a remote Oracle 10g Enterprise instance running on Windows 2003 Server. Locally, I have installed Oracle Data Access Components v11.1.0.6.20.
Thanks!
Are you watching your output window for any exceptions? I don't see any catch blocks in your code.
Oracle's ODP.net has almost exactly the same syntax as ADO, but performs better in many situations. If you're only using Oracle, it might be worth a look.
Is there a reason to use StringBuilder? A single string variable will perform better and makes code easier to read.
It seems that the queries were actually returning, just taking a very long time due to poor query performance, low bandwidth, and the enormous amount of rows being returned. The VS debugger seems to give up after a few seconds for these long-running queries. However, if I let it sit for a few minutes my breakpoints would get hit and things would work as expected.
Thanks for the replies / comments!
Related
So for a project, I need to read data from Acomba odbc driver. Acomba is an old accounting software. Behind the scene, acomba stores his data in hexa in flat files. They provide an odbc driver that works decently, but it is very slow.
For a particular client, they have 17000 products. Nothing too big. But getting the 17000 products from the odbc driver takes over 2m30. Some clients have close to 1 million products, so the performance becomes a huge issue.
Basically, this code is in a webapi. It gets the data, make a csv with it and return the csv file in the httpResponse. The csv generation takes more or less 1 second. So that is not the issue.
I tried a few things :
using (var db = new OdbcConnection($"DSN={_settings.DsnName}"))
{
await db.OpenAsync();
OdbcCommand comm = new OdbcCommand(sql, db);
OdbcDataReader dr = comm.ExecuteReader();
while (dr.Read())
{
var col1 = dr.GetValue(0).ToString();
var col2 = dr.GetValue(1).ToString();
var col3 = dr.GetValue(2).ToString();
}
}
Same thing, but with
dr.GetValues(destinationArray)
Same thing, but with an odbc adapter
var dt = new DataTable();
using (var cmd = new OdbcCommand(sql, db))
using (var adapter = new OdbcDataAdapter(cmd))
{
adapter.Fill(dt);
}
Those all end up taking between 2min and 2min20.
The issue with this particular table is that there is 150 columns. Processing all those 150 columns is what takes all the time.
Tried to search for a way to optimize this, but at the end of the day, all the code I found is basically the same thing as what I wrote.
Here is a twist!
If I open the odbc connection in Excel or Access, both of them can build a table and display it in under 5 seconds. I scrolled all the way down, so the data is actually loaded. It's not only showing the first 50 rows.
Does anybody knows how to get the same kind of performance in C#?
Thanks for your time!
I found the way to make the ODBC connection go MUCH faster. Basically, this old odbc driver is a COM. It turns out that COM objects goes much faster when you are in Single Threaded apartment vs multi-threaded. So I used the solution posted in this article :
http://ryanhaugh.com/archive/2014/05/24/supporting-sta-threads-in-web-api/
public static class TaskFactoryExtensions
{
private static readonly TaskScheduler _staScheduler = new StaTaskScheduler(numberOfThreads: 1);
public static Task<TResult> StartNewSta<TResult>(this TaskFactory factory, Func<TResult> action)
{
return factory.StartNew(action, CancellationToken.None, TaskCreationOptions.None, _staScheduler);
}
}
And im my api call, I do this :
Task<DataTable> responseTask = Task.Factory.StartNewSta(() => GetData(sqlQuery));
The actual get data, which is identical to the old one :
private DataTable GetData(string value)
{
var dt = new DataTable();
using (var db = new OdbcConnection($"DSN={_settings.DsnName}"))
{
db.Open();
OdbcCommand comm = new OdbcCommand(value, db);
using (OdbcDataAdapter da = new OdbcDataAdapter(comm))
{
da.Fill(dt);
}
}
return dt;
}
I use the extension method on all the calls made to get the data using StartNewSta. With the exact same code, it went from 2min 30 to 9 seconds.
So the key for fast COM objects, is STA threads!
I use FluentMigrator to create a SqlLite DB in C# using FluentMigrator.Runner.MigrationRunner. I wonder is there any way to use the SetPassword command o the SqlConnection only when the DB needs to be created ? There's a SqLiteRunnerContextFactory object but it don't seem to be a property that I can use to specify password.
public MigrationRunner CreateMigrationRunner(string connectionString, string[] migrationTargets, Assembly assembly, long version)
{
var announcer = new TextWriterAnnouncer(Console.WriteLine) { ShowSql = true };
var options = new ProcessorOptions { PreviewOnly = false, Timeout = 60 };
var runnerContext = new SqLiteRunnerContextFactory().CreateRunnerContext(connectionString, migrationTargets, version, announcer);
var sqlLiteConnection = new SQLiteConnection(connectionString);
//If the DB has already been created, it crashes later on if I specify this
sqlLiteConnection.SetPassword("ban4an4");
return new MigrationRunner(assembly,
runnerContext,
new SQLiteProcessor(sqlLiteConnection,
new SQLiteGenerator(),
announcer,
options,
new SQLiteDbFactory()));
}
I would like to avoid having to look if the file exists before setting password on connection.
Well, finally the code below works perfectly by using SetPassword everytime you create de runner. No need to check if the file exists or not. First time it creates it with the password and second time it opens it with it seems to use it to open DB. Which is exactly what I was looking for.
Using Async-based Webservice and Async framework in WinRT (Win8) to get a large recordsets(1000 to 5000) from a remote Ms SQL Server.
I want to know :
1) Which is the best approach to handle to insert large recordsets into SQLite?
2) Using RollBack transaction will start all over again if there is connection error. The below method will insert whatever and I can update the data later if the records are not complete. Is this a good approach?
3) Any better way to enhance my below solution?
This foreach statement to handle
each reords in returned result which returned from Async-Based WebService:
foreach (WebServiceList _List in IList)
{
InsertNewItems(_List.No, _List.Description, _List.Unit_Price, _List.Base_Unit_of_Measure);
}
private void InsertNewItems(string ItemNo, string ItemName, decimal ItemPrice, string ItemBUoM)
{
var existingItem = (db2.Table().Where(c => c.No == ItemNo)).SingleOrDefault();
if (existingItem != null)
{
existingItem.No = ItemNo;
existingItem.Description = ItemName;
existingItem.Unit_Price = ItemPrice;
existingItem.BaseUnitofMeasure = ItemBUoM;
int success = db2.Update(existingItem);
}
else
{
int success = db2.Insert(new Item()
{
No = ItemNo,
Description = ItemName,
Unit_Price = ItemPrice,
Base_Unit_of_Measure = ItemBUoM
});
}
}
You should use RunInTransaction from sqlite-net. The documentation for it says,
Executes action within a (possibly nested) transaction by wrapping it
in a SAVEPOINT. If an exception occurs the whole transaction is rolled
back, not just the current savepoint. The exception is rethrown.
using (var db = new SQLiteConnection(DbPath))
{
db.RunInTransaction(() =>
{
db.InsertOrReplace(MyObj);
});
}
Wiki article for Transactions at GitHub
The most important performance aspect for bulk inserts is to use a single transaction. If you want to handle aborts, I suggest that you feed the data in sufficiently large parts and restart from that point on next time. An SQL transaction either finishes completely or rolls back completely, so unless the input data changes between two runs, there should be no need to do an insert-or-update.
See, for example, here for a discussion of SQLite bulk insert performance using different methods.
System Scope
I have a database with a lot of users (over 50,000). At any time there may be 100-200 people logged in and actively using the system. The system is ASP.NET MVC 4, with Sql Server 2008 backend. For data access we are using Dapper.
Requirements
I am trying to build a notification component that has the following attributes:
When a new record is created in the [dbo.Assignment] table (with OwnerId = [Currently logged in user]), I need to update the Cache inside of an asp.net application.
I don't want to receive any notifications for users who are not actively online, as this would be a massive waste of resources)
Specific Questions:
Should I be using SqlDependency, SqlCacheDependency, or SqlNotification?
Assuming that we are using SqlDependency, how would I remove the Dependency.OnChange handler when user has logged out.
Any code samples would be much appreciated, as this has consumed the whole part of my day trying to figure it out.
Here is the current code
public IList<Notification> GetNotifications(string userName)
{
Cache o = HttpContext.Current.Cache;
if (o["Notifications_" + userName] == null)
{
var notifications = new List<Notification>();
using (var cn = new SqlConnection(getSQLString()))
{
using (var cmd = cn.CreateCommand())
{
var parameter = new SqlParameter("Employee_Cd", SqlDbType.Char, 30) { Value = userName };
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = "Notifications.Assignments";
cmd.Parameters.Add(parameter);
cmd.Notification = null;
var dependency = new SqlCacheDependency(cmd);
cn.Open();
using (var dr = cmd.ExecuteReader())
{
// this is where you build your cache
while (dr.Read())
{
var obj = new Notification();
obj.Name = dr["Name"].ToString();
notifications.Add(obj);
}
dr.Close();
}
HttpContext.Current.Cache.Insert("Notifications_" + userName,
notifications,
dependency,
DateTime.Now.AddDays(1D),
Cache.NoSlidingExpiration);
}
}
}
return (List<Notification>) o["Notifications_" + userName];
}
Note: I am not experienced with using SqlDependencies, as I have never really needed to use them until today. It's very possible that I am overlooking something important.
I didn’t really use any of these techniques but here are some alternatives that you can create yourself that will do the job just as good.
If you need to update cache every time new record is inserted into dbo.Assignment table why not create OnInserted event in your data access layer that will notify the cache object to refresh?
Another thing you can to is to create INSERT trigger in Assignemt table and another table that can look like this dbo.Cache (LastUpdate datetime). Trigger will insert value into Cache table and your application cache can ping this table like every X mins or X seconds to see if cache update is required.
If you need to refresh the cache immediately after record is inserted triggers might be an overkill because you’d have to ping Cache table probably every second but if you have 200 online users at a time that probably won’t make much of a difference in DB performance.
There is a lot of work if you want to implement these for a lot of tables but since this is only one table this might turn out to be faster way than implementing built in cache mechanisms.
I've been building this project as the solo dev for a while, and while I'm comfortable in the front end and middle tier, I don't really think I'm doing the database the way I should be, and the reason why is because I simply don't really know of any other way. The way I'm currently getting data is by testing out queries in my MySQL workbench and copying and pasting the SQL as a string literal into a method that makes a call to the DB, pulls the data and hydrates my objects.
This hasn't really been a problem until recently, when I had to create a monster of a query and it got me thinking that maybe there's a better way to do this. I don't have a formal DAL separated out, so I know there's room for improvement there, but I was curious about what the correct way would be to store SQL strings. I assume there is a tool somewhere built into VS10 where I can manipulate and work with SQL as SQL instead of as a string.
You should be doing this in stored procedures. That will basically format and store your query. You set parameters that are passed in from your code, then read out the results.
Example:
The C# method:
private void SetNote()
{
const string sql = "sp_SelectControllerNoteByID";
using (var conn = MocSystem.GetMocDbConnection())
{
using (var comm = new SqlCommand(sql, conn))
{
comm.CommandType = CommandType.StoredProcedure;
comm.Parameters.Add(new SqlParameter("#ControllerNoteID", ControllerNoteId));
try
{
conn.Open();
using (var rdr = comm.ExecuteReader())
{
while (rdr.Read())
{
CommentText = rdr["NoteText"].ToString();
_commentor = new User(new Guid(rdr["NoteAuthor"].ToString()));
CommentDate = (DateTime)rdr["NoteDate"];
MocRequestId = (int)rdr["MocRequestID"];
}
}
}
catch (Exception ex)
{
HasError = true;
ErrorMessage += "\nThere was a problem building the note: " + ex.Message;
}
}
}
}
The stored procedure on the DBMS (sql server in this example):
ALTER proc [dbo].[sp_SelectControllerNoteByID]
#ControllerNoteID int
AS
SELECT
ControllerNoteID,
NoteText,
NoteDate,
NoteAuthor,
MocRequestID
FROM
ControllerNotes
WHERE
ControllerNoteID = #ControllerNoteID
So here we call the stored procedure which in this case is just a simple select statement, then we read it out into an object via ADO. Now, this way, you can modify your query without recompiling. Unless you add parameters, in which case you'll have to update those in your code as well.