I am having trouble importing into sql lite.
I am exporting a table from Sql Server into a flat file encoded in UTF-8. And then trying to import the flat file into sqlite db. DB is UTF-8 encoded.
These lines are troublesome (tab delimited, line ends with CRLF):
ID w posid def
1234 bracket 40 "(" and ")" spec...
1234 bracket 40 Any of the characters "(", ")", "[", "]", "{", "}", and, in the area of computer languages, "<" and ">".
Error:
unescaped " character
I have tried replacing the quotes " with double quotes "", still doesn't work.
Import settings: tab separator
.separator " "
.import data.txt words
sqlite Table Schema:
CREATE TABLE words (ID integer NOT NULL, w TEXT NOT NULL, posid integer NOT NULL, def TEXT NOT NULL);
Update:
Somehow, adding a hash at the beginning of the def field in Sql Server worked:
update words set def = '#' + def
Not sure why that is. This worked, but it added an unwanted character in the field.
It turned out import can mess up when there are new line characters, or quotes, or commas.
One solution would be to replace these characters with some other character sequences, or character codes (e.g. char(1), char(2)...) , and make sure fields don't contain these sequences or codes, before you run the import. For example, replace quotes with --, then import, then replace -- with quotes again. I have another table with some text fields that have new line characters, and this solution seems to work.
before import:
update [table] set comment = REPLACE(comment, CHAR(13), '-*-')
update [table] set comment = REPLACE(comment, CHAR(10), '%-$-%')
update [table] set comment = REPLACE(comment, '"', '%-&-%')
after import:
update [table] set comment = REPLACE(comment, '-*-', CHAR(13))
update [table] set comment = REPLACE(comment, '%-$-%', CHAR(10))
update [table] set comment = REPLACE(comment, '%-&-%', '"')
To do that without changing the input data, use ascii mode and set the column separator to tab and the row separator to CRLF.
.mode ascii
.separator "\t" "\r\n"
See my answer to this other question for an explanation of why.
Related
My SQLite query :
let data = db.getAllRows(sql"""
SELECT name, source, uploaded_at, canonical_url, size
FROM table
WHERE name like ?
ORDER BY ? DESC
LIMIT ?
OFFSET ?
""", &"%{query}%", order, limit, offset)
Nim adds single quotes to any string replacing ?. I can manually build the SQL string and then use sql(string), but the input then isn't escaped. Is there some other token apart from ? that does not add '?
To answer your title question: "How to safely add a raw string to a query", you can use dbQuote:
import db_sqlite
let
a = "unes'caped %' string"
b = "my prefix ->"
c = "<- my suffix"
d = b & dbQuote(a) & c
echo d
This will print my prefix ->'unes''caped %'' string'<- my suffix, adding quotes before/after and escaping any ones inside. This string is presumably safe to pass as an sql statement. You should get the same result as if you had used ? with extra parameters.
I create a SQLite function which takes a string and returns another string, then I can use the return value as match strings. Code is here
It works very well except for the single quotes. In this case, it can't match any rows, but if I directly use the returned string, it can match. Anyone know what's the problem here?
sqlite> select simple_query('''');
"''"
sqlite> select ' ', simple_highlight(t1, 0, '[', ']') from t1 where x match simple_query('''');
sqlite> select ' ', simple_highlight(t1, 0, '[', ']') from t1 where x match '"''"';
|#English &special _characters."[']bacon-&and[']-eggs%
Full example here
This question finally answered in sqlite-forum, and I'd like to post the reason in here.
The reason is SQLite will try to escape the string for us, we can verify that after turn on quote mode, as you can see, our return value will be escaped from "'" to "''" by SQLite. Which means we don't need to escape single quote in our function.
sqlite> select simple_query('''');
"'"
sqlite> select simple_query('"');
""""
sqlite> .mode quote
sqlite> select simple_query('"');
'""""'
sqlite> select simple_query('''');
'"''"'
I used the following command
SELECT json_extract(data,'$.address') FROM data;
and output as CSV file.
Output in CSV file is
enter image description here
Field (column) in CSV file is saved as 2 lines for 1 field (column).
Eg-
"71 CHOA CHU KANG LOOP
NORTHVALE"
How could I save field(column) as 1 line ?
That is I don't want to include new line character in filed(column).
Eg-
"71 CHOA CHU KANG LOOP NORTHVALE"
Thanks.
Just replace the new line character:
select replace(json_extract(data,'$.address'), char(10), '') from data;
This will catch the newline character ('\n'). If you want '\r' and '\r\n' too:
select replace(
replace(json_extract(data,'$.address'), char(10), ''),
char(13),
''
) from data;
while inserting i can insert danish character in proper format in sqlite Db but while retrieving my query returns no result
String searchQuery= "SELECT * FROM article,product where article.ItemNo=product.ItemNo ";
if(searchText.length()>0)
{
searchQuery += " AND (article.itemNo like '"+ searchText +"%' OR product.Description like '"+ searchText +"%')";
}
in debug mode query is
`SELECT * FROM article,product where article.ItemNo=product.ItemNo AND (article.itemNo like '%ø%' OR product.Description like '%ø%')..`
No result returns
Proper query will be
SELECT * FROM article,product where article.ItemNo=product.ItemNo AND (article.itemNo like '%Ø%' OR product.Description like '%Ø%');
the desired description field value in Db is MØNTPUNG.
I am wondering is there any issue of case sensitivty?I am using UTF8 encoding for my raw file that will insert data to DB.
The documentation says:
SQLite only understands upper/lower case for ASCII characters by default. The LIKE operator is case sensitive by default for unicode characters that are beyond the ASCII range. For example, the expression 'a' LIKE 'A' is TRUE but 'æ' LIKE 'Æ' is FALSE.
To handle non-ASCII characters correctly, store an uppercase version of your string(s) in a separate column, and search in that with an uppercase search pattern.
I wrote the database schema (only one table so far), and the INSERT statements for that table in one file. Then I created the database as follows:
$ sqlite3 newdatabase.db
SQLite version 3.4.0
Enter ".help" for instructions
sqlite> .read ./schema.sql
SQL error near line 16: near "s": syntax error
Line 16 of my file looks something like this:
INSERT INTO table_name (field1, field2) VALUES (123, 'Hello there\'s');
The problem is the escape character for a single quote. I also tried double escaping the single quote (using \\\' instead of \'), but that didn't work either. What am I doing wrong?
Try doubling up the single quotes (many databases expect it that way), so it would be :
INSERT INTO table_name (field1, field2) VALUES (123, 'Hello there''s');
Relevant quote from the documentation:
A string constant is formed by enclosing the string in single quotes ('). A single quote within the string can be encoded by putting two single quotes in a row - as in Pascal. C-style escapes using the backslash character are not supported because they are not standard SQL. BLOB literals are string literals containing hexadecimal data and preceded by a single "x" or "X" character. ... A literal value can also be the token "NULL".
I believe you'd want to escape by doubling the single quote:
INSERT INTO table_name (field1, field2) VALUES (123, 'Hello there''s');
for replace all (') in your string, use
.replace(/\'/g,"''")
example:
sample = "St. Mary's and St. John's";
escapedSample = sample.replace(/\'/g,"''")
Just in case if you have a loop or a json string that need to insert in the database. Try to replace the string with a single quote . here is my solution. example if you have a string that contain's a single quote.
String mystring = "Sample's";
String myfinalstring = mystring.replace("'","''");
String query = "INSERT INTO "+table name+" ("+field1+") values ('"+myfinalstring+"')";
this works for me in c# and java
In C# you can use the following to replace the single quote with a double quote:
string sample = "St. Mary's";
string escapedSample = sample.Replace("'", "''");
And the output will be:
"St. Mary''s"
And, if you are working with Sqlite directly; you can work with object instead of string and catch special things like DBNull:
private static string MySqlEscape(Object usString)
{
if (usString is DBNull)
{
return "";
}
string sample = Convert.ToString(usString);
return sample.Replace("'", "''");
}
In bash scripts, I found that escaping double quotes around the value was necessary for values that could be null or contained characters that require escaping (like hyphens).
In this example, columnA's value could be null or contain hyphens.:
sqlite3 $db_name "insert into foo values (\"$columnA\", $columnB)";
Demonstration of single quoted string behavior where complexity or double quotes are not desired.
Test:
SELECT replace('SAMY''S','''''','''');
Output:
SAMY'S
SQLite version:
SELECT sqlite_version();
Output:
3.36.0