So for a project, I need to read data from Acomba odbc driver. Acomba is an old accounting software. Behind the scene, acomba stores his data in hexa in flat files. They provide an odbc driver that works decently, but it is very slow.
For a particular client, they have 17000 products. Nothing too big. But getting the 17000 products from the odbc driver takes over 2m30. Some clients have close to 1 million products, so the performance becomes a huge issue.
Basically, this code is in a webapi. It gets the data, make a csv with it and return the csv file in the httpResponse. The csv generation takes more or less 1 second. So that is not the issue.
I tried a few things :
using (var db = new OdbcConnection($"DSN={_settings.DsnName}"))
{
await db.OpenAsync();
OdbcCommand comm = new OdbcCommand(sql, db);
OdbcDataReader dr = comm.ExecuteReader();
while (dr.Read())
{
var col1 = dr.GetValue(0).ToString();
var col2 = dr.GetValue(1).ToString();
var col3 = dr.GetValue(2).ToString();
}
}
Same thing, but with
dr.GetValues(destinationArray)
Same thing, but with an odbc adapter
var dt = new DataTable();
using (var cmd = new OdbcCommand(sql, db))
using (var adapter = new OdbcDataAdapter(cmd))
{
adapter.Fill(dt);
}
Those all end up taking between 2min and 2min20.
The issue with this particular table is that there is 150 columns. Processing all those 150 columns is what takes all the time.
Tried to search for a way to optimize this, but at the end of the day, all the code I found is basically the same thing as what I wrote.
Here is a twist!
If I open the odbc connection in Excel or Access, both of them can build a table and display it in under 5 seconds. I scrolled all the way down, so the data is actually loaded. It's not only showing the first 50 rows.
Does anybody knows how to get the same kind of performance in C#?
Thanks for your time!
I found the way to make the ODBC connection go MUCH faster. Basically, this old odbc driver is a COM. It turns out that COM objects goes much faster when you are in Single Threaded apartment vs multi-threaded. So I used the solution posted in this article :
http://ryanhaugh.com/archive/2014/05/24/supporting-sta-threads-in-web-api/
public static class TaskFactoryExtensions
{
private static readonly TaskScheduler _staScheduler = new StaTaskScheduler(numberOfThreads: 1);
public static Task<TResult> StartNewSta<TResult>(this TaskFactory factory, Func<TResult> action)
{
return factory.StartNew(action, CancellationToken.None, TaskCreationOptions.None, _staScheduler);
}
}
And im my api call, I do this :
Task<DataTable> responseTask = Task.Factory.StartNewSta(() => GetData(sqlQuery));
The actual get data, which is identical to the old one :
private DataTable GetData(string value)
{
var dt = new DataTable();
using (var db = new OdbcConnection($"DSN={_settings.DsnName}"))
{
db.Open();
OdbcCommand comm = new OdbcCommand(value, db);
using (OdbcDataAdapter da = new OdbcDataAdapter(comm))
{
da.Fill(dt);
}
}
return dt;
}
I use the extension method on all the calls made to get the data using StartNewSta. With the exact same code, it went from 2min 30 to 9 seconds.
So the key for fast COM objects, is STA threads!
Related
I use FluentMigrator to create a SqlLite DB in C# using FluentMigrator.Runner.MigrationRunner. I wonder is there any way to use the SetPassword command o the SqlConnection only when the DB needs to be created ? There's a SqLiteRunnerContextFactory object but it don't seem to be a property that I can use to specify password.
public MigrationRunner CreateMigrationRunner(string connectionString, string[] migrationTargets, Assembly assembly, long version)
{
var announcer = new TextWriterAnnouncer(Console.WriteLine) { ShowSql = true };
var options = new ProcessorOptions { PreviewOnly = false, Timeout = 60 };
var runnerContext = new SqLiteRunnerContextFactory().CreateRunnerContext(connectionString, migrationTargets, version, announcer);
var sqlLiteConnection = new SQLiteConnection(connectionString);
//If the DB has already been created, it crashes later on if I specify this
sqlLiteConnection.SetPassword("ban4an4");
return new MigrationRunner(assembly,
runnerContext,
new SQLiteProcessor(sqlLiteConnection,
new SQLiteGenerator(),
announcer,
options,
new SQLiteDbFactory()));
}
I would like to avoid having to look if the file exists before setting password on connection.
Well, finally the code below works perfectly by using SetPassword everytime you create de runner. No need to check if the file exists or not. First time it creates it with the password and second time it opens it with it seems to use it to open DB. Which is exactly what I was looking for.
I am new in database using Delphi. Actually I need to display all the tables and their data in a gridview using firedac. Several videos on youtube and docwiki teaches how to access database through visual components i.e. throwing FDConnetion, FDTable, DataSource, etc. on the form.
But I would like to do the same thing programmatically so that I can focus more on coding and separate form from the business logic.
Here is what I have done in C# while accessing MSAccess data:
public void LoadUsersInfo()
{
try {
OleDbConnection Connection = new OleDbConnection();
Connection.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;DataSource=dbCyberNet.accdb;Persist Security Info=False;";
Connection.Open();
OleDbCommand command = new OleDbCommand();
command.Connection = Connection;
String query = "SELECT * FROM tblUsersInfo";
command.CommandText = query;
OleDbDataAdapter adapter = new OleDbDataAdapter(command);
DataTable dt = new DataTable();
adapter.Fill(dt);
dbGrid.DataSource = dt;
}
catch (Exception ex) {
MessageBox.Show("Error occured while loading users information . " + ex.Message.ToString());
}
finally { Connection.Close(); }
}
I would like to do similar in delphi for sqlite database. Here is what I have tried (edited).
procedure TForm1.Button1Click(Sender: TObject);
var
con: TFDConnection;
table: TFDTable;
ds: TDataSource;
begin
con := TFDConnection.Create(nil);
con.LoginPrompt := False;
table := TFDTable.Create(nil);
table.TableName := 'Students';
ds := TDataSource.Create(nil);
ds.DataSet := table;
try
con.Open('DriverID=SQLite;Database=studentsDB.sqlite;');
table.Connection := con;
table.Active := True;
con.Connected := True;
DBGrid1.DataSource := ds;
finally
con.Close;
con.Free;
table.Free;
ds.Free;
end;
end;
Whenever I run this code, error dialog appears showing: First chance exception at $759AF097. Exception class ESQLiteNativeException with message '[FireDAC][Phys][SQLite] ERROR: no such table: STUDENTS'. Process Project1.exe (3836)
whereas the same table name exists inside the table and using visual firedac components, I can see the data grid fills easily.
You can do the same as with C#: assign your TFDQuery instance as DataSource of the VCL grid!
Add a TDataSource to the form, set its DataSet property to your TFDQuery, and assign it to your grid.
See the official Delphi documentation and other SO questions.
You can execute your Select query by calling the query's Open method so your code will look like this :-
procedure TStudents.executeQuery;
var
con: TFDConnection;
query: TFDQuery;
begin
con := TFDConnection.Create(nil);
query := TFDQuery.Create(con);
con.LoginPrompt := False;
con.Open('DriverID=SQLite;Database=studentsDB.sqlite;');
query.Connection := con;
query.SQL.Text := 'SELECT * FROM Students';
query.Open;
query.First;
While Not Query.EOF Do
Begin
// Populate your object or do whatever processing you like here.
Query.Next;
End;
end;
You can then programatically create a TDataSource and a Grid and hook those up in code in a similar fashion. You'll need to move the Connection and Query objects out of your object's executeQuery method as they are local variables and will not be accessible to anything outside that method.
Note that Delphi does not garbage collect like C# does, so as it stand, the code leaks memory.
As an aside, your code appears to be an object method and it looks like you're attempting to create a set of business objects that know how to load and save themselves. Rather than roll your own, you might be better served by looking at one of the existing frameworks that exist (tiOPF, InstantObjects, mORMot or TMS Aurelius) to achieve this. These will allow you to concentrate on solving the problem at hand and not worry about the minutae of creating, loading and saving of objects.
System Scope
I have a database with a lot of users (over 50,000). At any time there may be 100-200 people logged in and actively using the system. The system is ASP.NET MVC 4, with Sql Server 2008 backend. For data access we are using Dapper.
Requirements
I am trying to build a notification component that has the following attributes:
When a new record is created in the [dbo.Assignment] table (with OwnerId = [Currently logged in user]), I need to update the Cache inside of an asp.net application.
I don't want to receive any notifications for users who are not actively online, as this would be a massive waste of resources)
Specific Questions:
Should I be using SqlDependency, SqlCacheDependency, or SqlNotification?
Assuming that we are using SqlDependency, how would I remove the Dependency.OnChange handler when user has logged out.
Any code samples would be much appreciated, as this has consumed the whole part of my day trying to figure it out.
Here is the current code
public IList<Notification> GetNotifications(string userName)
{
Cache o = HttpContext.Current.Cache;
if (o["Notifications_" + userName] == null)
{
var notifications = new List<Notification>();
using (var cn = new SqlConnection(getSQLString()))
{
using (var cmd = cn.CreateCommand())
{
var parameter = new SqlParameter("Employee_Cd", SqlDbType.Char, 30) { Value = userName };
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = "Notifications.Assignments";
cmd.Parameters.Add(parameter);
cmd.Notification = null;
var dependency = new SqlCacheDependency(cmd);
cn.Open();
using (var dr = cmd.ExecuteReader())
{
// this is where you build your cache
while (dr.Read())
{
var obj = new Notification();
obj.Name = dr["Name"].ToString();
notifications.Add(obj);
}
dr.Close();
}
HttpContext.Current.Cache.Insert("Notifications_" + userName,
notifications,
dependency,
DateTime.Now.AddDays(1D),
Cache.NoSlidingExpiration);
}
}
}
return (List<Notification>) o["Notifications_" + userName];
}
Note: I am not experienced with using SqlDependencies, as I have never really needed to use them until today. It's very possible that I am overlooking something important.
I didn’t really use any of these techniques but here are some alternatives that you can create yourself that will do the job just as good.
If you need to update cache every time new record is inserted into dbo.Assignment table why not create OnInserted event in your data access layer that will notify the cache object to refresh?
Another thing you can to is to create INSERT trigger in Assignemt table and another table that can look like this dbo.Cache (LastUpdate datetime). Trigger will insert value into Cache table and your application cache can ping this table like every X mins or X seconds to see if cache update is required.
If you need to refresh the cache immediately after record is inserted triggers might be an overkill because you’d have to ping Cache table probably every second but if you have 200 online users at a time that probably won’t make much of a difference in DB performance.
There is a lot of work if you want to implement these for a lot of tables but since this is only one table this might turn out to be faster way than implementing built in cache mechanisms.
I've been building this project as the solo dev for a while, and while I'm comfortable in the front end and middle tier, I don't really think I'm doing the database the way I should be, and the reason why is because I simply don't really know of any other way. The way I'm currently getting data is by testing out queries in my MySQL workbench and copying and pasting the SQL as a string literal into a method that makes a call to the DB, pulls the data and hydrates my objects.
This hasn't really been a problem until recently, when I had to create a monster of a query and it got me thinking that maybe there's a better way to do this. I don't have a formal DAL separated out, so I know there's room for improvement there, but I was curious about what the correct way would be to store SQL strings. I assume there is a tool somewhere built into VS10 where I can manipulate and work with SQL as SQL instead of as a string.
You should be doing this in stored procedures. That will basically format and store your query. You set parameters that are passed in from your code, then read out the results.
Example:
The C# method:
private void SetNote()
{
const string sql = "sp_SelectControllerNoteByID";
using (var conn = MocSystem.GetMocDbConnection())
{
using (var comm = new SqlCommand(sql, conn))
{
comm.CommandType = CommandType.StoredProcedure;
comm.Parameters.Add(new SqlParameter("#ControllerNoteID", ControllerNoteId));
try
{
conn.Open();
using (var rdr = comm.ExecuteReader())
{
while (rdr.Read())
{
CommentText = rdr["NoteText"].ToString();
_commentor = new User(new Guid(rdr["NoteAuthor"].ToString()));
CommentDate = (DateTime)rdr["NoteDate"];
MocRequestId = (int)rdr["MocRequestID"];
}
}
}
catch (Exception ex)
{
HasError = true;
ErrorMessage += "\nThere was a problem building the note: " + ex.Message;
}
}
}
}
The stored procedure on the DBMS (sql server in this example):
ALTER proc [dbo].[sp_SelectControllerNoteByID]
#ControllerNoteID int
AS
SELECT
ControllerNoteID,
NoteText,
NoteDate,
NoteAuthor,
MocRequestID
FROM
ControllerNotes
WHERE
ControllerNoteID = #ControllerNoteID
So here we call the stored procedure which in this case is just a simple select statement, then we read it out into an object via ADO. Now, this way, you can modify your query without recompiling. Unless you add parameters, in which case you'll have to update those in your code as well.
I have this problem in my ASP.NET application where I'm seeing some of my Oracle queries fired off to the server then not returning. Ever. It happens in several places in my app and I can't explain it. Here's one specific scenario where I'm seeing this behavior:
During application start-up I am pre-fetching data asynchronously into the application state (the choice was made to use app state instead of cache b/c the data never changes during the lifetime of the app).
Action<string, object> AddApplicationState = (string name, object data) =>
{
Application.Lock();
Application.Add(name, data);
Application.UnLock();
};
Func<DataTable> GetFullNames = () => Database.GetAllNames();
Func<DataTable> GetProvinceNames = () => Database.GetProvinceNames();
Func<DataTable> GetTribeNames = () => Database.GetTribeNames();
GetFullNames.BeginInvoke(result => AddApplicationState("AllNames", GetFullNames.EndInvoke(result)), null);
GetProvinceNames.BeginInvoke(result => AddApplicationState("ProvinceNames", GetProvinceNames.EndInvoke(result)), null);
GetTribeNames.BeginInvoke(result => AddApplicationState("TribeNames", GetTribeNames.EndInvoke(result)), null);
The second two return just fine, but the first either never returns or returns after about 10 minutes. After firing up Oracle SQL Developer I go to the 'monitor sessions' tool and can see a single session for the query. It looks like it has completed, b/c the wait time is (null) and the session is inactive. Here's the ADO.NET code used to query the database:
public static DataTable GetAllNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
var sql = new StringBuilder();
sql.AppendLine("SELECT NAME_ID, NATIVE_NAME, NVL(FREQUENCY,0) \"FREQUENCY\", CULTURE_ID,");
sql.AppendLine("ENGLISH_NAME, REGEXP_REPLACE(ENGLISH_NAME, '[^A-Za-z]', null) \"ENGLISH_NAME_STRIPPED\"");
sql.AppendLine("FROM NAMES");
oracmd.CommandText = sql.ToString();
var orada = new OracleDataAdapter(oracmd);
var dtAllNames = new DataTable();
orada.Fill(dtAllNames);
return dtAllNames;
}
}
}
public static DataTable GetTribeNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
var sql = new StringBuilder();
sql.AppendLine("SELECT DISTINCT NAME_ID, English_Name \"TRIBE_NAME_ENGLISH\",");
sql.AppendLine("REGEXP_REPLACE(English_Name, '[^A-Za-z]',null) \"TRIBE_ENGLISH_NAME_STRIPPED\",");
sql.AppendLine("NATIVE_NAME \"TRIBE_NATIVE_NAME\"");
sql.AppendLine("FROM NAMES");
sql.AppendLine("WHERE NAME_ID IN ");
sql.AppendLine("(SELECT NAME_ID_TRIBE FROM TRIBES UNION SELECT NAME_ID_FAMILY FROM TRIBES)");
sql.AppendLine("ORDER BY English_Name");
oracmd.CommandText = sql.ToString();
var orada = new OracleDataAdapter(oracmd);
var dt = new DataTable();
orada.Fill(dt);
return dt;
}
}
}
public static DataTable GetProvinceNames()
{
using (OracleConnection oraconn = GetConnection())
{
using (OracleCommand oracmd = GetCommand(oraconn))
{
oracmd.CommandText = "SELECT DISTINCT PROVINCE_ID, PROVINCE_NAME_NATIVE, PROVINCE_NAME_ENGLISH FROM PROVINCES";
var orada = new OracleDataAdapter(oracmd);
var dtRC = new DataTable();
orada.Fill(dtRC);
return dtRC;
}
}
}
As you can see, the ADO.NET code is pretty standard (and boring!) stuff. When run in SQL Developer, the queries return less than a second. The first query returns x rows, the second x rows, and the third x rows. But this problem of queries being fired off then never returning happens often and I can't seem to track down the issue. Anyone have any thoughts?
And finally, since I realize it could be something completely unrelated to code, I am running the app locally (from Visual Studio) on a Windows XP SP3 machine and connecting via VPN to a remote Oracle 10g Enterprise instance running on Windows 2003 Server. Locally, I have installed Oracle Data Access Components v11.1.0.6.20.
Thanks!
Are you watching your output window for any exceptions? I don't see any catch blocks in your code.
Oracle's ODP.net has almost exactly the same syntax as ADO, but performs better in many situations. If you're only using Oracle, it might be worth a look.
Is there a reason to use StringBuilder? A single string variable will perform better and makes code easier to read.
It seems that the queries were actually returning, just taking a very long time due to poor query performance, low bandwidth, and the enormous amount of rows being returned. The VS debugger seems to give up after a few seconds for these long-running queries. However, if I let it sit for a few minutes my breakpoints would get hit and things would work as expected.
Thanks for the replies / comments!