We're trying to insert data into a clipper database file (DBF file with NTX index file).
For a variety of reasons, we cannot change the database format.
We are currently seeing two issues:
With our existing code, we are not able to update or utilize the NTX index file (I believe). We would like to be able to do this. Do you know of an OLE or ODBC driver that can do this?
We are able to insert a row into the clipper database file (DBF) as long as we do not include a date. If we include a date, in any format, we get an error.
Some example code:
OleDbConnection con = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + path + ";Extended Properties=dBASE IV");
string sql = "insert into TABLE (E, J, DATE, STARTTIME, ENDTIME) values ('1', '2', '2010-01-13' ,'08:12:12', '18:12:12')";
OleDbCommand myCommand = new OleDbCommand(sql);
myCommand.Connection = con;
con.Open();
myCommand.ExecuteNonQuery();
myCommand.Connection.Close();
and the exception is something like:
01/15/2010 12:50:31 {ERROR} ASITranslator.GUI.ASITranslatorGUI.insertSCH - Error in: Syntax error in INSERT INTO statement.-- StackTrace: at System.Data.OleDb.OleDbCommand.ExecuteCommandTextErrorHandling(OleDbHResult hr)
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
Again, without the DATE column, it works fine.
Is there a better provider to use for Clipper files (that provider works great for other DBF files).
Any ideas?
Seems the issue is primarily related to the OLE DBF / dbase driver is unable to write to the Clipper native format, which is a modified version of dbase III.
To write to the Clipper format, this string needs to be used:
Provider=MSDASQL.1;Persist Security Info=False;Mode=ReadWrite;Extended Properties="CollatingSequence=ASCII;DBQ=C:\DATA\8110FULL;DefaultDir=C:\DATA\8110FULL;Deleted=1;Driver={Microsoft dBase Driver (*.dbf)};DriverId=21;FIL=dBase III;FILEDSN=C:\Program Files\Common Files\ODBC\Data Sources\test.dsn;MaxBufferSize=2048;MaxScanRows=8;PageTimeout=600;SafeTransactions=0;Statistics=0;Threads=3;UID=admin;UserCommitSync=Yes;";Initial Catalog=C:\DATA\8110FULL
This will allow one to write to the file, including the DATE format.
However, this does NOT use the NTX index files (nor does it update them). For that, it would appear that we would need to use the CodeBase (or similar) Clipper driver.
First, it appears you are trying to add 'Text values' for the date columns regardless of them being in a date format. Additionally, if used in web-based applications where you would be using variables, you'd be best to use parameterized queries.
String sql = "insert into YourTable ( fld1, fld2, DateFld1, DateFld2 ) "
+ "value ( ?, ?, ?, ? )";
OleDbCommand myCommand = new OleDbCommand(sql);
OleDbParameter NewParm = new OleDbParameter( "parmFld1", 1 );
NewParm.DbType = DbType.Int32;
myCommand.Parameters.Add( NewParm );
NewParm = new OleDbParameter( "parmFld2", 2 );
NewParm.DbType = DbType.Int32;
myCommand.Parameters.Add( NewParm );
NewParm = new OleDbParameter( "parmDate1", DateTime.Now );
NewParm.DbType = DbType.DateTime;
myCommand.Parameters.Add( NewParm );
NewParm = new OleDbParameter( "parmDate2", DateTime.Now );
NewParm.DbType = DbType.DateTime;
myCommand.Parameters.Add( NewParm );
Then continue with your connection, open, execute and close...
DSN-less connection : (Note the driver and FIL string changed to what called in Windows 7)
It use Microsoft OLE DB Provider for ODBC Drivers (MSDASQL).
The code use the .Net Framework Data Provider for ODBC (System.Data.Odbc),
Not the .Net Framework Data Provider for OLEDB (System.Data.OleDb).
Clipper type N-> OdbcType Double
"Provider=MSDASQL.1;Persist Security Info=False;Mode=ReadWrite;Extended Properties=\"CollatingSequence=ASCII;DBQ=F:\\Folder;DefaultDir=F:\\Folder;Deleted=1;DRIVER=Microsoft Access dBASE Driver (*.dbf, *.ndx, *.mdx);DriverId=21;FIL=dBASE III;MaxBufferSize=2048;MaxScanRows=8;PageTimeout=600;SafeTransactions=0;Statistics=0;Threads=3;UID=admin;UserCommitSync=Yes;\";Initial Catalog=F:\\Folder";
Related
I would like to encrypt an existing database column with always encrypted. My project is a ASP.NET project using code first and database is SQL Server. The database has already data. I created a migration to achieve my goal.
First I tried to alter the column type, using the following.
ALTER TABLE [dbo].[TestDecrypted] ALTER COLUMN [FloatCol] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
I got the following error.
Operand type clash: float is incompatible with float encrypted with (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt')
Then I decided to created another column and migrate the data.
ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = [FloatCol]
And I got the same error.
After I looked at this, I noticed that it is possible to insert data like the following
DECLARE #floatCol FLOAT = 1.1
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #floatCol
But if I try to obtain the value from my existing column, it fails.
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
The error follows.
Encryption scheme mismatch for columns/variables '#floatCol'. The encryption scheme for the columns/variables is (encryption_type = 'PLAINTEXT') and the expression near line '4' expects it to be (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt').
Does anyone knows how can I achieve my goal?
Update 1
#Nikhil-Vithlani-Microsoft did some interesting suggestions.
Always Encrypted Wizard in SSMS - I would like to achieve my goal with code first migrations, so this idea does not fit.
SqlBulkCopy - It does not work inside migrations, because the new column will only exist after all 'Up' method is run. Therefore we cannot insert data into this column in this way inside this method.
Anyway, his suggestions drove me to another attempt: obtain the decrypted values and update the encrypted column with them.
var values = new Dictionary<Guid, double>();
var connectionString = ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString;
using (var sourceConnection = new SqlConnection(connectionString))
{
var myCommand = new SqlCommand("SELECT * FROM dbo.TestDecrypted", sourceConnection);
sourceConnection.Open();
using (var reader = myCommand.ExecuteReader())
{
while (reader.Read())
{
values.Add((Guid)reader["Id"], (double)reader["FloatCol"]);
}
}
}
Sql("ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL");
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
In fact, I did not try to create another column and to migrate the data, as mentioned in an example above. I tried it only on SSMS.
And now I got a different error.
Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I tried to do it without encrypting the new column, and it worked properly.
Any idea why this error occurs?
You will have to do the always encrypted related migration outside of entity framework. This blog should help
https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/
If you want to encrypt an existing column, you can use Always Encrypted Wizard in SSMS, or use this article that explains how to migrate existing data.
Also, please note that doing bulk inserts through a C# (.NET 4.6.1+ client) app is supported.
You can do this in c# using SqlBulkCopy specifically using SqlBulkCopy.WriteToServer(IDataReader) Method.
Create a new table (encryptedTable) with the same schema as that of your plaintext table (unencryptedTable) but with the encryption turned on for the desired columns.
Do select * from unencryptedTable to load the data in a SqlDataReader then use SqlBulkCopy to load it to the encryptedTable using SqlBulkCopy.WriteToServer(IDataReader) Method
For example,
Plaintext Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) NOT NULL)
Encrypted Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (ENCRYPTION_TYPE = DETERMINISTIC,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256',
COLUMN_ENCRYPTION_KEY = CEK1) NOT NULL)
As for why your method does not work,
when you use parameterization for always encrypted, the right hand side (RHS) of the declare statement needs to be a literal. Because the driver will identify the literal and encrypt it for you. So, the following will not work, since RHS is a sql expression and cannot be encrypted by the driver
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
Update:
The following code will not work because parameterization for Always Encrypted only applies to SSMS
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
However, if you rewrite your code as follows, that should work
foreach (var valuePair in values)
{
SqlCommand cmd = _sqlconn.CreateCommand();
cmd.CommandText = #"UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #FloatVar WHERE Id = '{valuePair.Key}'");";
SqlParameter paramFloat = cmd.CreateParameter();
paramFloat.ParameterName = #"#FloatVar";
paramFloat.DbType = SqlDbType.Float;
paramFloat.Direction = ParameterDirection.Input;
paramFloat.Value = floatValue;
cmd.Parameters.Add(paramFloat);
cmd.ExecuteNonQuery();
}
Hope that helps, if you have additional question, please leave them in the comments.
DbDataAdapter.Fill() is extremly slow when performing parameters!
I have a query with 2 parameters inside, and when I put those parameters hardcoded in the query it takes 1 second to execute (in a 470k table rows, returning only 20 rows).
I found many posts similars here and I tried all those solutions (set arithabort, option recompile, option optimize for, ...) with no luck.
I just perform a query (sql server 2008) and not a stored procedure, so the query with arithabort is like this:
string strSql = #"set ARITHABORT ON;
select TOP 20 ....
Also I tried to call set arithabort in the same transaction but performing that query first..
I don't know if I'm doing something wrong, but the sensation is the ado.net is performing a very bad execution plan in ado.net when I have defined parameters on it.
As a result of this bad choice, the execution time in SSMS is 1 second (after being cached) but in asp is like 9 seconds!
The query is something like this:
strSQL #="
select *
from Table1
where Name like #name";
And then:
DbProviderFactory factory = DbProviderFactories.GetFactory(mProvider);
DbCommand dbcmd = factory.CreateCommand();
if (CommandTimeout != null)
dbcmd.CommandTimeout = CommandTimeout.Value;
if(this.transaccion != null)
dbcmd.Transaction = this.transaccion;
dbcmd.Connection = dbc;
dbcmd.CommandText = strSQL;
if (parametros != null)
dbcmd.Parameters.AddRange(parametros);
DbDataAdapter dbda = factory.CreateDataAdapter();
dbda.SelectCommand = dbcmd;
DataTable dt = new DataTable();
dbda.Fill(dt);
return dt;
EDIT 14/01/2013 (18:44)
I'm not longer retrieve the connection from DbProviderFactory, insted I'm using directly SqlConnection and SqlCommand. I know DbCommand and DbProvider are a base clase... but I think there is something more in there.. because the performance drasticaly increase like 300%!
It's not the fill method, because I already tried in the code shown before..
Anyway, I don't know the reason why but using a SqlConnection is much faster! Any idea? Maybe isn't making that bad execution plan made before?
SqlCommand objCmd = new SqlCommand(strSQL, sqlConn);
if (CommandTimeout != null)
objCmd.CommandTimeout = CommandTimeout.Value;
if (this.transaccion != null)
objCmd.Transaction = SQLtransaccion;
if (parametros != null)
objCmd.Parameters.AddRange(parametros);
DbDataReader dbReader = objCmd.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dbReader);
dbReader.Close();
return dt;
Any help will be greatly appreciated,
Thanks,
I found the solution!
It was parameters!
I was using a wrong type in the the List!
Parametross.Add(bd.MakeParameter("#val", "%" + txtFind.Text + "%",
DbType.String));
DbType.String vs. DbType.AnsiString
Although both DbType.String and DbType.AnsiString deal with character data, these datatypes are processed differently, and using the wrong data type can have a negative effect on the application’s performance. DbType.String identifies the parameter as a 2-byte Unicode value and is sent to the server as such.DbType.AnsiString causes the parameter to be sent as a multibyte character string. To avoid excessive string conversions, use:
DbType.AnsiString for char or varchar columns and parameters.
DbType.String for unichar and univarchar columns and parameters.
Source:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc20066.0115/html/adonet/adonet49.htm
In my query there is a:
....
where Table.Col1 like #val
But the column type was varchar and I should use DbType.AnsiString, instead of DbType.String
Parametross.Add(bd.MakeParameter("#val", "%" + txtFind.Text + "%",
DbType.AnsiString));
In my huge table I was making a lot of unnecesary casts and this is the reason why the performance drastically fall down!
Hope this will help someone,
I got similar questions to this but not get satisfied answers. I want to store paragraph in table in sql server 2005. I tried all data types (nvarchar, varchar, text etc.) but when I copy the content, it store only 1st line. Help me.
I am using Sql Server Management Studio and directly copy paste the content into table. And for retrieving I use this code.
String query2 = String.Format("select data from try where roll=55");
SqlCommand cmd2 = new SqlCommand(query2, conn);
SqlDataReader reader2 = cmd2.ExecuteReader();
if (reader2.Read())
{
Label1.Text = reader2.GetValue(0).ToString();
}
where 'data' is multiline content.
I did a little test using Sql Server 2008-R2 (So it might be different on 2005):
insert into Test1 (Col1) VALUES('THIS
INSERTS
A
Multiline
String')
When selecting, the result DOES look like there are no line-breaks, BUT if you check for char(10) and char(13) - (CRLF) they DO exist in the string...
select col1, REPLACE(REPLACE(col1,CHAR(10), 'X'),CHAR(13), 'Y') as LineBreaks from Test1
Result:
If you set SSMS to return values as text rather than grid (button on toolbar) you can see the line breaks in the text, they aren't displayed in the grid view.
I have a web application that I help maintain, we store the data from a multiline textbox in SQL server 2008 R2 -- inserted with ADO.Net and Pulled using ADO.net At first glance it appears that the breaks in the "paragraph" are missing. To recover the line breaks for a printable version of the data, I utilize a quick regular expression to place the breaks back in the web app. The method used to do this is below:
public static string lineBreakCharsToHTMLBreak(string sourceString)
{
// This regex says look for any tab, carriage return or linefeed
Regex rexNewLine = new Regex("[\\t\\n\\r]");
sourceString = rexNewLine.Replace(sourceString, "<br>");
Regex rexDoppleBreak = new Regex("(<br>){2,}");
sourceString = rexDoppleBreak.Replace(sourceString, "<br>");
return sourceString;
}
I have a .aspx page that has query to and informix database. This query is done via an odbc connection and is put into a datatable. Then this datatable is used as the datasource for a radio button group.
My problem is that for whatever reason the time is being appended to the radio button as "12:00:00 AM". This is odd because the informix field is a date field that does not include the time. If I were to run the query outside of the webpage it returns it without the time... "2012-06-15"
So in summary... what I am getting is: "6/15/2012 12:00:00 AM" and what I want is "06/15/2012"
The query is as follows:
"select DATE(attend_date) as attend_date from soar_major_table where major =? and active<>'N'"
The code that creates the datatable:
string connString;
connString = ConfigurationManager.ConnectionStrings [ "ERP" ].ConnectionString;
OdbcConnection conn = new OdbcConnection ( );
conn.ConnectionString = connString;
string sql = "select DATE(attend_date) as attend_date from soar_major_table where major =? and active<>'N' ";
OdbcCommand command = new OdbcCommand ( );
command.CommandText = sql;
command.Parameters.Add ( new OdbcParameter ( "major", major ) );
command.Connection = conn;
DataTable dt = new DataTable ( );
OdbcDataAdapter dataAdapter = new OdbcDataAdapter ( );
dataAdapter.SelectCommand = command;
try
{
conn.Open ( );
dataAdapter.Fill ( dt );
}
finally
{
if ( conn != null && conn.State == ConnectionState.Open )
{
command.Dispose ( );
dataAdapter.Dispose ( );
conn.Close ( );
}
}
return dt;
And lastly the population of the radio btn group:
if ( dt.Rows.Count > 0 )
{
rdoDate.DataSource = dt;
rdoDate.DataTextField = "attend_date";
rdoDate.DataValueField = "attend_date";
rdoDate.DataBind ( );
}
The problem is upstream of the Informix data server, I believe.
When you execute:
SELECT DATE(attend_date) ...
the server will return that value as a 4-byte integer representing the number of days since 1899-12-31 (so 1900-01-01 was day 1), which is the internal representation of a DATE in Informix.
Something in a higher layer is then treating it as a 'date + time' value and assuming midnight is the time since there was no time component in the date, and is then rubbing salt in the wound by formatting it in am/pm notation.
This will involve client-side tracing of what's going on. My suspicion (not founded on anything except limited knowledge of the ODBC drivers) is that the problem is occurring in the .NET layers rather than the ODBC driver. However, you're way outside my area of expertise once you're above the ODBC layer (and I don't claim great expertise in ODBC).
You may be able to isolate the problem to the client code by using SQLIDEBUG=2:xyz in the environment (you might need to set that with SETNET32 for Windows). If it works at all on Windows (it does on Unix), then you'll end up with a file with a name starting xyz_ followed by various groups of digits and letters. That file can be analyzed by sqliprint and will show you what was sent to the Informix data server and returned to your client. Assuming the SQL was not hacked en route to the server, then you'll see the date returned as a simple date, and the problem is definitively client-side. If the SQL is hacked en route, then that too is a client-side problem.
It at least gives a starting point for debugging.
Look very carefully at the data types of the types your code is using. In particular, some DBMS have a DATE type that includes time information, and you may need to avoid that interpretation. The SQL standard has DATE (no time), TIME (no date) and TIMESTAMP (date and time) types.
I'm currently using the following snippet to insert data into a table in my database. It works great. But, I want to start adding filename data and not sure how to proceed.
I have the following:
// Create command
comm = new SqlCommand(
"INSERT INTO Entries (Title, Description) " +
"VALUES (#Title, #Description)", conn);
// Add command parameters
comm.Parameters.Add("#Description", System.Data.SqlDbType.Text);
comm.Parameters["#Description"].Value = descriptionTextBox.Text;
comm.Parameters.Add("#Title", System.Data.SqlDbType.NVarChar, 50);
comm.Parameters["#Title"].Value = titleTextBox.Text;
I also have a File Upload option. But, I don't know how to use this to do the following:
move the file to my images directory and
store the filename value in my table.
I have added the correct enctype to my form but now a little lost.
Can someone explain the best way to do this?
Many thanks for any help with this.
To store the file in an images folder, it should be:
FileUpload1.SaveAs(Server.MapPath("~/Images/" + FileUpload1.FileName));
and then add the command parameters in the fileName
comm.Parameters["#FileName"].Value = FileUpload1.FileName;
Note: you must have the FileName field in your DB table.
I suggest storing file in the db too. This will guarantee data consistency.
Add column to the DB. Replace X with the suitable size if the image is less than 8000, or specify varbinary(MAX) if it is not.
alter table Entries
add FileContent varbinary(X) not null
C# code:
byte[] fileContent = yourFileContent;
using(var connection = new SqlConnection(connectionString))
using (var command = connection.CreateCommand())
{
command.CommandText = #"
INSERT INTO Entries (Title, Description, FileContent)
VALUES (#Title, #Description, #FileContent)
";
command.Parameters.AddWithValue("Description", descriptionTextBox.Text);
command.Parameters.AddWithValue("Title", titleTextBox.Text);
command.Parameters.AddWithValue("FileContent", fileContent);
connection.Open();
command.ExecuteScalar();
}