My MySQL database can store the euro symbol just fine (as I have tested with a native MySQL client (HeidiSQL)). But with the MySQL .NET connector my ASP.NET application is using I can't insert nor read it back from the database: I only get a ? character back. What could be the possible cause?
I would suggest explicitly specifying the encoding in your connection string:
Server=localhost;Database=schema;Uid=foo;Pwd=bar;CharSet=utf8;
It usually resolves most encoding-related issues with MySQL's Connector/NET.
I'd say that MySQL .NET connector sets some collation-related environment variables on connection; compare output of these queries both on HeidiSQL and .NET:
SHOW VARIABLES LIKE "character_set_%";
SHOW VARIABLES LIKE "collation_%";
They all should contain utf8-something. If not, you can alter them at runtime as in the manual.
The following .Net command can encode these for your database, and then decode them into the original symbol...
System.Web.HttpUtility.HtmlEncode()
System.Web.HttpUtility.HtmlDecode()
Presumably you are inserting into the euro sign into a field that is typed as NVARCHAR or you wouldn't be able to insert and retrieve the euro sign correctly with other clients?
Are you using the correct syntax in your SQL statements for inserting and retrieving Unicode data?
To do this you must use the N specifier character before any string which is unicode or it will get silently converted to a single byte encoding and become an unreadable character e.g.
//Inserting Unicode data
MySqlCommand cmd = new MySqlCommand();
cmd.CommandText = "INSERT INTO SOME_TABLE (someField) VALUES (N'#text')";
cmd.Parameters.Add("text", "Some Unicode Text");
cmd.Connection = conn;
cmd.ExecuteNonQuery();
//Selecting Unicode data
MySqlCommand select = new MySqlCommand();
select.CommandText = "SELECT * FROM SOME_TABLE WHERE someField=N'#text'";
cmd.Parameters.Add("text", "Some Unicode Text");
cmd.Connection = conn;
//Execute your query and process your Results...
I had a similar problem, trying to import data from Sybase 12.5 to MS SQL Server 2005.
Basically MSSQL stores £ and euro signs fine in varchar if they're encoded right, which is great, but what I didn't realise is that Sybase was outputting the data as windows-1252 (western european) and not UTF-8 and this caused data like £ and the euro symbol to get translated by the import programme as '?'. Telling the import programme that the incoming data is encoded as 1252 fixed the issue.
If you encode the output from the DB as 1252 as you're reading it, you should get the right symbols out.
For example:
System.Text.Encoding enc = System.Text.Encoing.GetEncoding(1252);
MySqlDataReader msdr = command.ExecuteReader();
while(msdr.Read())
{
...
string val = enc.GetString(enc.GetBytes(msddr.GetString(mssdr.GetOrdinal("varhcar_field"))));
...
}
Have you correctly set the charset in your output page ?
i.e. something like <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
or charset=iso-8859-15 but not charset=iso-8859-1
Related
I am having a look on an old web application written in the 90s in VB6 and Active Server Pages (JScript). The application retrieves some data from the database and stores it in a recordset which it is using to update. When it attempts to update a field (see below) it gives an '80040e21' error.
rsSave.Fields('text') = Request.Form('strText').Item(i); // this line fails
I have checked the type of the field and it is adVarWChar (202). I have checked the size of the 'text' field which is 2000, way bigger than what is comming from the form. I checked the status of all fields and they are all adFieldOK (0). In other words any of the usual suspects giving normally this error are ok.
The COM+ object that is creating, filling and then returning the recordset is doing the following:
'Initialize command object
Set oCmd = CreateObject("ADODB.Command")
With oCmd
.CommandType = adCmdText
.CommandText = strsql
End With
Set cn = CreateObject("ADODB.Connection")
'Open connection to database
cn.Open strConn
oCmd.ActiveConnection = cn
Set rs = CreateObject("ADODB.Recordset")
With rs
Set .Source = oCmd
.LockType = adLockBatchOptimistic
.CursorLocation = adUseClient
.CursorType = adOpenStatic
.Open
Set .ActiveConnection = Nothing
End With
I tried using adLockOptimistic, but no luck.
Last but not least, this application was using initially an old Microsoft OleDb provider for Oracle that was no longer compatible with windows server 2008. We had to use a new provider and since then some things needed to be adjusted in order to work properly.
Any ideas?
I used to get these when I was writing from VB6 to a DB2 database using IBM OLE DB Provider. It was always related to either the data being incompatible with the underlying field type, or the content was too large. Check dates, strings being written into integers, vice versa, etc.
I would try a process of elimination. Write data only one field, defaulting the others to the minimum thy will accept, then keep adding fields with the expected data until it triggers an error. At least it will tell you which field it is.
Sorry I don't have much more to offer.
I have problem with data which I want to fetch from SQLite database in my Adobe AIR application. I use Javascript.
In sqlite3 CLI when I use this query: select * from notes;
I get this result:
Cześć|8
Boa noite|12
Até logo|13
But when I want to get this data in AIR using:
dbQuery = new air.SQLStatement();
dbQuery.sqlConnection = db;
dbQuery.text = "SELECT hello, id FROM sometable";
I have something like that:
Cze|8
Boa noite|12
At logo|13
All specific characters are removed.
Where could be problem and how can I resolve it?
I discovered where was problem.
I added new entities using sqlite3 CLI and probably the values were inserted in wrong encoding, because when I inserted values like "łąśćźóż" using AIR Adobe application or importing data to database:
sqlite3 test.db < test.sql
values are saved in correct encoding and when I get them to application, they are displayed in correct encoding.
I'm looking to encode and store Unicode in a Sqlite database. Is there any way to raw encode a UTF-8 (unicode) string literal in a sql query.
I'm looking for something similar to java where I can toss a \u00E9 into a string and have it automagically upconvert to Unicode.
What language are you using? SQLite handles Unicode just fine, creating the literals in your hosting language is less obvious.
$ sqlite3 junk.sqlite
SQLite version 3.6.22
sqlite> create table names (id integer primary key, name string);
sqlite> insert into names values (null,
'î℉ yõù g𐌹ѷЄ ΣϘГくטƏ UTF-8, it stores it');
sqlite> select * from names;
1|î℉ yõù g𐌹ѷЄ ΣϘГくטƏ UTF-8, it stores it
SQLite doesn't have escape sequences. But your programming language probably does.
# in Python
db.execute("INSERT INTO MyTable(MyColumn) VALUES('\u00E9')")
or
db.execute("INSERT INTO MyTable(MyColumn) VALUES(?)", ['\u00E9'])
If for some reason you have to write a UTF-8 literal in pure SQL, you can do something like:
sqlite> SELECT CAST(X'C3A9' AS TEXT);
é
Edit: Since this answer was originally written, a CHAR function has been added to SQLite. So now, you could write
INSERT INTO MyTable(MyColumn) VALUES(CHAR(233))
If your problem is reinterpretation of escape sequences in sqlite you can (ab)use json_extract eg.
UPDATE `tableToFix` SET `columnToFix` = json_extract('"' || `columnToFix` || '"', '$');
INSERT INTO test VALUE (json_extract('"P\u0159\u00edli\u0161 \u017elu\u0165ou\u010dk\u00fd k\u016f\u0148 \u00fap\u011bl \u010f\u00e1belsk\u00e9 \u00f3dy."', '$'));
Notice: quotes handling. Valid json string starts and ends with " so you must add them before use of json_extract
If you configure your database to use UTF-8 (I believe this is default for many installations; do PRAGMA encoding="UTF-8"; at schema creation time to be certain), this shouldn't be an issue.
If you send SQLite3 a set of characters encoded in UTF-8, it should have no problem dealing with it.
If Java has the ability to allow you to "toss a \u0039 into a string", I'd just use that, and ensure that when you try to place the string into the database, that you have the string convert to a UTF-8 byte encoding using whatever mechanism Java provides. I do not believe SQLite provides or needs to provide this for you.
I'm using this code to store korean string in my database:
Dim username As String = Request.QueryString.Get("Some Korean String")
Using dg As New DataContext()
Dim newfriend As New FriendsTable With {.AskingUser = User.Identity.Name, .BeingAskedUser = username, .Pending = True}
dg.FriendsTables.InsertOnSubmit(newfriend)
dg.SubmitChanges()
end using
Checking my database, the username stored is a string"????"...
anybody got an idea how this happened or any workarounds?
What is your database collation? Are you able to store Korean strings with any other data access technology? What is the type of the username column, and is it accurately mapped in LINQ to SQL?
I suspect that something in the database isn't set up correctly to allow full Unicode. I very much doubt that this has anything to do with LINQ itself.
The other thing to check is that you're actually getting the right data in the first place. There are often several places where things can go wrong - you need to validate each place separately to see where the data is being corrupted. I have a short article on this which you may find helpful.
It sounds like you are storing Korean text in a varchar/text column which is not using a Korean collation. Thea easiest fix is to change the column type to nvarchar/ntext.
The nchar column types store Unicode data, whereas the char and varchar types store single byte characters in the specified collation.
I have an application in asp.net which worked fine untill recently when i changed the datetime format in Regional and Language Settings in Control Panel.
The date time format was default when i installed XP. I chose Indian standard time while installing XP.
I changed the date time format to dd/MM/yyyy HH:mm:ss. And my application started to throw an exception whenever i tried to insert any datetime in to the table.
The exception i get is:
System.Data.SqlClient.SqlException: Error converting data type varchar to datetime.
Please help me on this
Hard to know exactly what's going on without seeing the code that's throwing. However, if you need to communicate dates to SQL Server, it is generally good practice to use the ISO 8601 standard for representation because it is unambiguous and locale-independent. The most important formats are:
yyyy-MM-dd for dates
hh:mm:ss for time
yyyy-MM-dd hh:mm:ss for date/time
My guess is that you have a query that's sending over dates in the current locale, and the locale on the server does not match.
Edit: And for the record, this doesn't preclude anything that Rob said in his answer, i.e. try to avoid passing hard-coded dates or hard-coded SQL at all. This only applies if you need to for some reason.
Edit 2: I've been informed that the yyyy-MM-dd format can still be wrong for some locales; so instead of this, if you need to pass in a literal date string, you should instead use yyyyMMdd.
As per my comment, you'll probably want to make sure you're using code that behaves in a similar way to the code below (i.e. using parameters rather than string concatenation)
var myConnectionString = "connection string goes here";
var myDateValue = DateTime.Now;
using (var connection = new SqlConnection(myConnectionString))
{
using (var command = new SqlCommand("SELECT COUNT(1) FROM dbo.table WHERE datecolumn = #datevalue", connection))
{
var dateValueParameter = new SqlParameter("#datevalue", myDateValue);
command.Parameters.Add(dateValueParameter);
var result = Convert.ToInt32(command.ExecuteScalar());
}
}
Try adding "Current Language=YourLanguage" to the SQL server connection string. Where YourLanguage is the language you want SQL to use when reading values such as the dates.
You can see a list of all languages supported by SQL by executing the following SQL command:
select * from master.dbo.syslanguages