I index a file in a SQLite DB.
I create my table with this:
CREATE TABLE Record (RecordID INTEGER,Data TEXT,PRIMARY KEY (RecordID))
I read a file, and for each line I add a row on the table.
Each line can have binary data at end. It's not a problem for many chars, but \0 char make a problem.
If I have a line like this : "My data \0with binary".
When I try to get data after the \0 not worked (SELECT substr(Data, 11, 5) FROM Record return an empty string or SELECT substr(Data, 4, 10) FROM Record return data)
When I try to search a data (SELECT Data FROM Record WHERE Data LIKE '%binar%') return 0 rows returned.
How can I solve this problem ? I try to replace \0 by an other char sequence, but it's not a good idea because can I have this sequence in my file.
Thank you
Related
String record : Blueoba Mountain Company
SQL Query :
SELECT from table
WHERE CATSEARCH(account_partner_name_type,'%Blueoba% %Mountain% %Company%', NULL) > 0)
where rn <=500;
If I write the full name of the string in the query (i.e.%Blueoba% %Mountain% %Company%) then it gives me the record.
But if I write %Blueoba% %Mountain% %Comp% or %Blue% %Company% or %Comp% then its not returning any record.
So ideally, if I write a word %comp% then it should search all the records which contains 'comp' word and show the records but its not showing.
Can anybody suggest something?
You can try using wild card characters
SELECT * from table
WHERE account_partner_name_type like '%Blueoba%'
and rn <=500;
I want use this procedure to display the username and moblephone number,the result sets is this when I use select :
declare enter image description here
when the procedure runs,I get this :
enter image description here
error ORA-01722: invalid number
ORA-06512: at "ABPROD.SHAREPOOL", line 24.
when I use unique or distinct in the cursor,nothing display.
the code source :
create or replace procedure sharepool (assignment in varchar2,myorgname in varchar2) is
rightid T_CLM_AP30_RIGHT.RIGHT_ID%type;
orgid t_clm_ap30_org.org_id%type;
begin
select t.right_id into rightid from T_CLM_AP30_RIGHT t where t.rightdesc=trim(assignment);
dbms_output.put_line(rightid||trim(myorgname)||assignment);
select t.org_id into orgid from t_clm_ap30_org t where t.orgname=trim(myorgname);
dbms_output.put_line(orgid);
declare
cursor namelist is select distinct a.username,a.mobile from t_clm_ap30_user a, T_CLM_AP30_RIGHT_AUTH t where a.user_id=t.user_id and t.right_id=rightid and t.poolorgrange=orgid ;
begin
for c in namelist
loop
dbms_output.put_line(c.username||' '||c.mobile);
end loop;
end;
end sharepool;
INVALID_NUMBER errors indicate a failed casting of a string to a number. That means one of your join conditions is comparing a string column with a number column, and you have values in the string column which cannot be cast to a number.
ORA-06512: at "ABPROD.SHAREPOOL", line 24
Line 24 doesn't align with the code you've posted, presumably lost in translation from your actual source. Also you haven't posted table descriptions so we cannot tell which columns to look at.
So here is a guess.
One (or more) of these joins has an implicit numeric conversion:
where a.user_id = t.user_id
and t.right_id = rightid
and t.poolorgrange = orgid
That is, either t_clm_ap30_user.user_id is numeric and T_CLM_AP30_RIGHT_AUTH.user_id is not, or vice versa. Or T_CLM_AP30_RIGHT_AUTH.right_id is numeric and T_CLM_AP30_RIGHT.right_id is not, or vice versa. Or T_CLM_AP30_RIGHT_AUTH.poolorgrange is numeric and t_clm_ap30_org.org_id is not, or vice versa.
Only you can figure this out, because only you can see your schema. Once you have identified the join where you have a string column being compared to a numeric column you need to query that column to find the data which cannot be converted to a number.
Let's say that T_CLM_AP30_RIGHT_AUTH.poolorgrange is the rogue string. You can see which are the troublesome rows with this query:
select * from T_CLM_AP30_RIGHT_AUTH
where translate (poolorgrange, 'x1234567890', 'x') is not null;
The translate() function strips out digits. So anything which is left can't be converted to a number.
"T_CLM_AP30_RIGHT_AUTH.poolorgrange is varchar2,and t_clm_ap30_org.org_id is numeric ."
You can avoid the error by explicitly casting the t_clm_ap30_org.org_id to a string:
select distinct a.username, a.mobile
from t_clm_ap30_user a,
T_CLM_AP30_RIGHT_AUTH t
where a.user_id = t.user_id
and t.right_id = rightid
and t.poolorgrange = to_char(orgid) ;
Obviously you're not going to get matches on those alphanumeric values but the query will run.
I am working on a project with a database. This database is very simple. There is only one table with 2 columns : id (int) and text (string).
To fill this base I want to create a .sql script file.
(this database isn't created inside an android project because I want an already filled database to insert in my android project)
I want my script to create the table and then read a .txt file with a string value (for text column) on each row.
For each row, it should insert the string value into the table.
I am not very familiar with SQLite and SQL in general.
I already found a way to auto-increment the id using an iterator (but I dind't test it yet), but I couldn't found how to read a .txt file line by line.
So my question is : Is it possible to read a .txt file line by line in a SQLite script ?
And if it is, could you please tell me how to do it.
Here's a solution in pure sqlite
CREATE TEMP TABLE input (value STRING);
INSERT INTO input VALUES (TRIM(readfile('input.txt'), char(10)));
CREATE TABLE lines (s STRING);
WITH RECURSIVE
nn (s, rest)
AS (
SELECT
(SELECT SUBSTR(input.value, 0, INSTR(input.value, char(10))) FROM input),
(SELECT SUBSTR(input.value, INSTR(input.value, char(10)) + 1) FROM input)
UNION ALL
SELECT
CASE INSTR(nn.rest, char(10))
WHEN 0 THEN nn.rest
ELSE SUBSTR(nn.rest, 0, INSTR(nn.rest, char(10)))
END,
CASE INSTR(nn.rest, char(10))
WHEN 0 THEN ''
ELSE SUBSTR(nn.rest, INSTR(nn.rest, char(10)) + 1)
END
FROM nn
WHERE LENGTH(nn.rest) > 0
)
INSERT INTO lines (s)
SELECT nn.s FROM nn;
DROP TABLE input;
A few subtleties here:
sqlite does not have a \n escape so you have to use char(10)
this doesn't work well for mixed newlines or \r\n newlines (though you can adjust some + 1s to + 2s and char(10) to char(13) || char(10)
most of the magic is in the recursive union in the middle which nibbles off a line at a time
note that I'm using this approach to solve advent of code -- https://github.com/anthonywritescode/aoc2020
SQLite is an embedded database; it is designed to be used together with some 'real' programming language.
There are no functions to access and parse text files.
You have to write your own script in whatever language you like, or use some existing tool.
If there is a character that is guaranteed not to occurr in the text file, you can use the sqlite3 command-line shell and a temporary, one-column table for importing:
CREATE TEMP TABLE i(txt);
.separator ~
.import MyFile.txt i
INSERT INTO TheRealTable(text) SELECT txt FROM i; -- assumes id is autoincrementing
DROP TABLE i;
I think the simplest way is work on the txt file to convert it to a csv file. Then you can import it directly in Sqlite3 or by a programming language.
sqlite> .mode csv table_name
sqlite> .import file_name.csv table_name
You can use a BufferedReader for that. the code could look like:
InputStream in = context.getResources().openRawResource( R.raw.your_txt_file );
BufferedReader reader = new BufferedReader( new InputStreamReader( in ) );
String line = null;
while( null != ( line = reader.readLine() ) ){
doStuffWithLine( line );
}
reader.close();
Yes, reading a .txt file line by line in a SQLite script is possible. But you'll need to use an extension. Specifically, sqlean-fileio can do the job.
Its fileio_scan(path) function reads the file specified by path line by line without loading the whole file into memory.
For example:
$ echo 'one' > data.txt
$ echo 'two' >> data.txt
$ echo 'three' >> data.txt
create table data(id integer primary key, txt text);
insert into data(txt)
select value from fileio_scan('data.txt');
select * from data;
┌────┬───────┐
│ id │ txt │
├────┼───────┤
│ 1 │ one │
│ 2 │ two │
│ 3 │ three │
└────┴───────┘
That's it!
So my question is : Is it possible to read a .txt file line by line in a SQLite script ?
Yes.
And if it is, could you please tell me how to do it.
There we go:
Pseudo-code algorithm:
Open the file.
Read line by line and insert new row in the database.
Close resources and commit transactions.
1) Open the file
InputStream instream = new FileInputStream("myfilename.txt");
InputStreamReader inputreader = new InputStreamReader(instream);
BufferedReader buffreader = new BufferedReader(inputreader);
2) Read line by line and insert new row in database
List<String> nameList = new ArrayList<>();
String line;
do {
line = buffreader.readLine();
if (line != null){
nameList.add(line);
}
} while (line != null);
Now you should insert all names in database:
storeNamesInDB(nameList);
Where
private void storeNamesInDB(nameList){
String sql = "INSERT INTO table (col1) VALUES (?)";
db.beginTransaction();
SQLiteStatement stmt = db.compileStatement(sql);
for (int i = 0; i < nameList.size(); i++) {
stmt.bindString(1, values.get(i));
stmt.execute();
stmt.clearBindings();
}
db.setTransactionSuccessful();
db.endTransaction();
}
3) Close resources
Don't forget to close resources:
instream.close();
inputreader.close();
DISCLAIMER!
You shouldn't copy&paste this code. Replace each var name and some instructions with someone that make sense in your project. This is just an idea.
I am creating an SQL db and trying to iterate over an excel file and put all the data in to the SQL table as follows but I keep getting an annoying error. I have looked at the data types and still can't get my head around it please let me know if anyone spots what the problem is my code is:
import sqlite3
from openpyxl import load_workbook
#wb = load_workbook(r"LeaguePlayers.xlsx")
#read workbook to get data
wb = load_workbook(filename = r"LeaguePlayers.xlsx", use_iterators = True)
ws = wb.get_sheet_by_name(name = 'Sheet1')
#ws = wb.worksheets
conn = sqlite3.connect("players.db") # or use :memory: to put it in RAM
cursor = conn.cursor()
# create a table
cursor.execute("""CREATE TABLE players
(player TEXT,
team TEXT,
points INTEGER,
cost REAL,
position TEXT)
""")
#Iterate through worksheet and print cell contents
for row in ws.iter_rows():
for cell in row:
cursor.execute("INSERT INTO players VALUES (?,?,?,?,?)", row)
conn.commit()
#----------------------------------------
# display SQL data
#----------------------------------------
c.execute('SELECT * FROM players')
for row in c:
print (row)
The error i get says:
cursor.execute("INSERT INTO players VALUES (?,?,?,?,?)", row)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
I really think you need to do some kind of introduction to Python.
You are making two elementary mistakes: looping of the cells in a row but passing the row to the query; passing a complex object as opposed to a native Python type such as an integer or string.
Something like the following is what you want:
player = [cell.value for cell in row]
cursor.execute(query, player)
Note, that execute takes a sequence (tuple or list) as the second argument.
Ok, so I have a basic table called "ledger", it contains fields of various types, integers, varchar, etc.
In my program, I used to use a query with no "from" predicate to collect all of the rows, which of course works fine. But... I changed my code to allow selecting one row at a time using "where acctno = x" (where X is the account number I want to select at the time).
I thought this must be a bug in the client library for my programming language, so I tested it in the SQLite command-line client - and it still doesn't work!
I am relatively new to SQLite, but I have been using Oracle, MS SQL Server, etc. for years and never seen this type of issue before.
Other things I can tell you:
* Queries using other integer fields also don't work
* Queries on char fields work
* Querying it as a string (with the account number on quotes) still doesn't work. (I thought maybe the numbers were stored as a string inadvertently).
* Accessing rows by rowid works fine - which is why I can edit the database with GUI tools with no noticeable problem.
Examples:
Query with no WHERE (works fine):
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
4|0|0|JPY|14|Japan Post Bank|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
...
Query with WHERE clause: (no results)
sqlite> select * from ledger where acctno=1;
sqlite>
putting quotes around the 1 above changes nothing.
Interestingly enough, "select * from ledger where acctno > 1" returns results! However since it returns ALL results, it's not terrible useful.
I'm sure someone will ask about the table structure, so here goes:
sqlite> .schema ledger
CREATE TABLE "LEDGER" (
"ACCTNO" integer(10,0) NOT NULL,
"drbal" integer(20,0) NOT NULL,
"crbal" integer(20,0) NOT NULL,
"CURRKEY" char(3,0) NOT NULL,
"TEXTKEY" integer(10,0),
"TEXT" VARCHAR(64,0),
"ACCTYPECD" CHAR(1,0) NOT NULL,
"ACCSTCD" CHAR(1,0),
"PACCTNO" number(10,0) NOT NULL,
"CATCD" number(10,0),
"TRANSNO" number(10,0) NOT NULL,
"extrefno" number(10,0),
"UPDATEUSER" VARCHAR(32,0),
"UPDATEDATE" text(8,0),
"UPDATETIME" TEXT(6,0),
"PAYEECD" number(10,0) NOT NULL,
"drbal2" number(10,0) NOT NULL,
"crbal2" number(10,0) NOT NULL,
"delind" boolean,
PRIMARY KEY("ACCTNO"),
CONSTRAINT "fk_curr" FOREIGN KEY ("CURRKEY") REFERENCES "CURRENCY" ("CUR
RKEY") ON DELETE RESTRICT ON UPDATE CASCADE
);
The strangest thing is that I have other similar tables where this works fine!
sqlite> select * from journalhdr where transno=13;
13|Test transaction ATM Withdrawel 20130213|20130223||20130223||
TransNo in that table is also integer (10,0) NOT NULL - this is what makes me thing it is something to do with the values.
Another clue is that the sort order seems to be based on ascii, not numeric:
sqlite> select * from ledger order by acctno;
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
10|0|0|USD|20|Sallie Mae|L|X|0|X|0|0|SYSTEM|20121209|153900|0|0|0|
21|0|0|USD|21|Skrill|A|X|0|X|0|0|SYSTEM|20121209|154000|0|0|0|
22|0|0|USD|22|AES|L|X|0|X|0|0|SYSTEM|20121209|154200|0|0|0|
23|0|0|JPY|23|Marui|L|X|0|X|0|0|SYSTEM|20121209|154400|0|0|0|
24|0|0|JPY|24|Amex JP|L|X|0|X|0|0|SYSTEM|20121209|154500|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
Of course the sort order on journalhdr (where the select works properly) is numeric.
Solved! (sort-of)
The data can be fixed like this:
sqlite> update ledger set acctno = 23 where rowid = 13;
sqlite> select * from ledger where acctno = 25;
25|0|0|JPY|0|Test|L|X|0|X|0|0|SYSTEM|20130224|132500|0|0|0|
Still, if it was stored as strings, then that leave a few questions:
1. Why couldn't I select it as a string using the quotes?
2. How did it get stored as a string since it is a valid integer?
3. How would you go about detecting this problem normally besides noticing bizzarre symptoms?
Although the data would normally be entered by my program, some of it was created by hand using Navicat, so I assume the problem must lie there.
You are victim of SQLite dynamic typing.
Even though SQLite defines system of type affinity, which sets some rules on how input strings or numbers will be converted to actual internal values, but it does NOT prevent software that is using prepared statements to explicitly set any type (and data value) for the column (and this can be different per row!).
This can be shown by this simple example:
CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16));
INSERT INTO ledger VALUES(1, 'John'); -- INTEGER '1'
INSERT INTO ledger VALUES(2 || X'00', 'Zack'); -- BLOB '2\0'
I have inserted second row not as INTEGER, but as binary string containing embedded zero byte. This reproduces your issue exactly, see this SQLFiddle, step by step. You can also execute these commands in sqlite3, you will get the same result.
Below is Perl script that also reproduces this issue
This script creates just 2 rows with acctno having values of integer 1 for first, and "2\0" for second row. "2\0" means string consisting of 2 bytes: first is digit 2, and second is 0 (zero) byte.
Of course, it is very difficult to visually tell "2\0" from just "2", but this is what script below demonstrates:
#!/usr/bin/perl -w
use strict;
use warnings;
use DBI qw(:sql_types);
my $dbh = DBI->connect("dbi:SQLite:test.db") or die DBI::errstr();
$dbh->do("DROP TABLE IF EXISTS ledger");
$dbh->do("CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16))");
my $sth = $dbh->prepare(
"INSERT INTO ledger (acctno, name) VALUES (?, ?)");
$sth->bind_param(1, "1", SQL_INTEGER);
$sth->bind_param(2, "John");
$sth->execute();
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->bind_param(2, "Zack");
$sth->execute();
$sth = $dbh->prepare(
"SELECT count(*) FROM ledger WHERE acctno = ?");
$sth->bind_param(1, "1");
$sth->execute();
my ($num1) = $sth->fetchrow_array();
print "Number of rows matching id '1' is $num1\n";
$sth->bind_param(1, "2");
$sth->execute();
my ($num2) = $sth->fetchrow_array();
print "Number of rows matching id '2' is $num2\n";
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->execute();
my ($num3) = $sth->fetchrow_array();
print "Number of rows matching id '2<0>' is $num3\n";
Output of this script is:
Number of rows matching id '1' is 1
Number of rows matching id '2' is 0
Number of rows matching id '2<0>' is 1
If you were to look at resultant table using any SQLite tool (including sqlite3), it will print 2 for second row - they all get confused by trailing 0 inside a BLOB when it gets coerced to string or number.
Note that I had to use custom param binding to coerce type to BLOB and permit null bytes stored:
$sth->bind_param(1, "2\0", SQL_BLOB);
Long story short, it is either some of your client programs, or some of client tools like Navicat which screwed it up.