In an SQLite database, I created a table and imported CSV data into it.
The last column only contains integers:
ZIP;AMOUNT
78123;4272
95456;154
etc.
I used the following commands:
.mode csv
.separator ';'
CREATE TABLE MyTable("ZIP" TEXT, "AMOUNT" INTEGER);
.import input.csv MyTable
sqlite> select SUM(AMOUNT) from MyTable;
25270.0
Why is SQLite displaying SUM with a decimal?
Thank you.
===
Edit: Here's the infos:
sqlite> select typeof(AMOUNT) from MyTable LIMIT 10;
text
integer
integer
integer
integer
etc.
sqlite> select typeof(SUM(AMOUNT)) from MyTable;
real
==
Edit: Here's the top of input.csv as exported from LibreOffice Calc:
ZIP;AMOUNT
78123;4272
95456;154
etc.
Maybe I didn't use the right commands to import data into SQLite.
#PetSerAl has it. Because you're importing into an existing table, every line is imported including the header. That string makes sum() return a floating point result.
I work around this case with:
.import '| tail -n +2 input.csv' mytable
which strips the first line.
If the first character of the filename is a |, it's treated as a shell command to run (Using the C popen() function) and its output is used as the data to import.
I am new to RSQLite.
I have an input document in text format in which values are seperately by '|'
I created a table with the required variables (dummy code as follows)
db<-dbconnect(SQLite(),dbname="test.sqlite")
dbSendQuery(conn=db,
"CREATE TABLE TABLE1(
MARKS INTEGER,
ROLLNUM INTEGER
NAME CHAR(25)
DATED DATE)"
)
However I am struck at how to import values into the created table.
I cannot use INSERT INTO Values command as there are thousands of rows and more than 20+ columns in the original data file and it is impossible to manually type in each data point.
Can someone suggest an alternative efficient way to do so?
You are using a scripting language. The deal of this is literally to avoid manually typing each data point. Sorry.
You have two routes:
1: You have corrected loaded a database connection and created an empty table in your SQLite database. Nice!
To load data into the table, load your text file into R using e.g. df <-
read.table('textfile.txt', sep='|') (modify arguments to fit your text file).
To have a 'dynamic' INSERT statement, you can use placeholders. RSQLite allows for both named or positioned placeholder. To insert a single row, you can do:
dbSendQuery(db, 'INSERT INTO table1 (MARKS, ROLLNUM, NAME) VALUES (?, ?, ?);', list(1, 16, 'Big fellow'))
You see? The first ? got value 1, the second ? got value 16, and the last ? got the string Big fellow. Also note that you do not enclose placeholders for text in quotation marks (' or ")!
Now, you have thousands of rows. Or just more than one. Either way, you can send in your data frame. dbSendQuery has some requirements. 1) That each vector has the same number of entries (not an issue when providing a data.frame). And 2) You may only submit the same number of vectors as you have placeholders.
I assume your data frame, df contains columns mark, roll, and name, corrsponding to the columns. Then you may run:
dbSendQuery(db, 'INSERT INTO table1 (MARKS, ROLLNUM, NAME) VALUES (:mark, :roll, :name);', df)
This will execute an INSERT statement for each row in df!
TIP! Because an INSERT statement is execute for each row, inserting thousands of rows can take a long time, because after each insert, data is written to file and indices are updated. Insert, enclose it in an transaction:
dbBegin(db)
res <- dbSendQuery(db, 'INSERT ...;', df)
dbClearResult(res)
dbCommit(db)
and SQLite will save the data to a journal file, and only save the result when you execute the dbCommit(db). Try both methods and compare the speed!
2: Ah, yes. The second way. This can be done in SQLite entirely.
With the SQLite command utility (sqlite3 from your command line, not R), you can attach a text file as a table and simply do a INSERT INTO ... SELECT ... ; command. Alternately, read the text file in sqlite3 into a temporary table and run a INSERT INTO ... SELECT ... ;.
Useful site to remember: http://www.sqlite.com/lang.html
A little late to the party, but DBI provides dbAppendTable() which will write the contents of a dataframe to an SQL table. Column names in the dataframe must match the field names in the database. For your example, the following code would insert the contents of my random dataframe into your newly created table.
library(DBI)
db<-dbConnect(RSQLite::SQLite(),dbname=":memory")
dbExecute(db,
"CREATE TABLE TABLE1(
MARKS INTEGER,
ROLLNUM INTEGER,
NAME TEXT
)"
)
df <- data.frame(MARKS = sample(1:100, 10),
ROLLNUM = sample(1:100, 10),
NAME = stringi::stri_rand_strings(10, 10))
dbAppendTable(db, "TABLE1", df)
I don't think there is a nice way to do a large number of inserts directly from R. SQLite does have a bulk insert functionality, but the RSQLite package does not appear to expose it.
From the command line you may try the following:
.separator |
.import your_file.csv your_table
where your_file.csv is the CSV (or pipe delimited) file containing your data and your_table is the destination table.
See the documentation under CSV Import for more information.
I would like to run a query involving joining a table to a manually generated list but am stuck trying to generate the manual list. There is an example of what I am attempting to do below:
SELECT
*
FROM
('29/12/2014', '30/12/2014', '30/12/2014') dates
;
Ideally I would want my output to look like:
29/12/2014
30/12/2014
31/12/2014
What's your Teradata release?
In TD14 there's STRTOK_SPLIT_TO_TABLE:
SELECT *
FROM TABLE (STRTOK_SPLIT_TO_TABLE(1 -- any dummy value
,'29/12/2014,30/12/2014,30/12/2014' -- any delimited string
,',' -- delimiter
)
RETURNS (outkey INTEGER
,tokennum INTEGER
,token VARCHAR(20) CHARACTER SET UNICODE) -- modify to match the actual size
) AS d
You can easily put this in a Derived Table and then join to it.
inkey (here the dummy value 1) is a numeric or string column, usually a key. Can be used for joining back to the original row.
outkey is the same as inkey.
tokennum is the ordinal position of the token in the input string.
token is the extracted substring.
Try this:
select '29/12/2014'
union
select '30/12/2014'
union
...
It should work in Teradata as well as in MySql.
I added a new column to my table, so there are now 4 instead of 3, and am now getting the following error when do a parametric insert (looped):
table 'test' has 4 columns but 3 values were supplied
Does this mean that you have to code your query for EVERY column the table has (as opposed to just the columns you want populated) when doing inserts, and that SQLite won't just add a default value if a column is missing from the query?
My query is:
"INSERT OR IGNORE INTO test VALUES (NULL, #col2, #col3)"
And this is the code that controls what's inserted in the recursive lopp:
sqlStatement.clearParameters();
var _currentRow:Object = _dataArray.shift();
sqlStatement.parameters["#col2"] = _currentRow.val2;
sqlStatement.parameters["#col3"] = _currentRow.val3;
sqlStatement.execute();
Ideally, I'd like column 4 to be left blank, without having to code it into the query.
Thanks for taking a look.
If you're inserting less values than there are columns, you need to explicitly specify the columns you are inserting to. For example
INSERT INTO test(firstcolumn,secondcolumn) VALUES(1,2);
Those columns that are not specified will get the default value, or NULL if there is no default value.
I have a sqlite3 full text search table defined like this:
CREATE VIRTUAL TABLE entries USING fts4 ( entry TEXT )
Each entry row has a line of text. How can I write a query to count the total amount of words in the table? Thanks
I don't know of a built-in function to do that, but you could re-use the answer to
" Query to count words SQLite 3 " to get the total count of words:
select sum(length(trim(entry))
- length(replace(trim(entry), ' ', '')) + 1) from entries;
(Modified the original answer by adding the trim.)
If you have sqlite3 version 3.7.6 or later, you can do something cleaner with an fts4aux table.
create virtual table terms using fts4aux(entries);
select count(distinct term) from terms;