I am trying to create a SQL database using a data set with a column that has both date and time included in it. The problem that I am running into is when the data is written into a SQL database, and read back into R the date and time column end up having a numeric structure rather than a Posixct structure or does not show the date and times correctly.
I have been using the RSQlite and DBI package to work between the two. I just started working with SQL, is there an appropriate way in reading date and time columns into a SQL database?
Thank you for your time.
SQLite does not support date and time types. Here are some options:
convert the date/time fields back to R classes yourself. You could write a separate function for each table read into R that reads in the table and does the conversion transparently or you could adopt a naming convention for the columns that lets a single function perform the conversion according to the naming rules if you control the database itself. Another way to implement a naming convention, other than writing your own function, is to use the sqldf package. If you use sqldf("select ...", dbname = "mydb", connection = con, method = "name__class") it will convert every column whose name has two underscores to the class name after the two underscores. Note that name__class has two underscores as well.
the dbmisc package (also see this) can perform conversions. You must prepare a schema, i.e. layout specification, for each table, as described there, in order to use it.
Use a different database that does support date/time types. I usually use the H2 database in such cases. The RH2 package includes the entire H2 database software right in the R driver package in a similar manner to RSQLite.
As per a comment below, the latest version of RSQLite has support for time and date fields; however, note that that that is on the R side, like the other solutions above (except using H2), and does not change the fact that SQLite itself has no such support so, for example, if you use SQL to modify such a field such as adding 1 to get the next date it will no longer be of the same type.
Related
For example, if I want to create a table that stores the mtcars data set in a remote database, I can do the following with DBI:
dbWriteTable(database_connection, "MTCARS", mtcars)
I think behind the scenes, DBI (or perhaps dbplyr?) generates some SQL and send it to the database to complete the task. Then how can I get the SQL so that I can tweak it to better suit my use case?
The APIs from the DBI (and other R SQL) package do not necessarily correspond to just one SQL operation. From the documentation for DBI, dbWriteTable does the following:
Writes, overwrites or appends a data frame to a database table, optionally converting row names to a column and specifying SQL data types for fields.
That is, depending on how you call dbWriteTable, using parameters such as append and overwrite, it may generate either an INSERT, UPDATE, or even an upsert.
How do I compare dates in SQLite database which are stored in DD-MMM-YYYY format e.g. 10-OCT-2017?
I want to compare dates and select rows of particular date and adjacent dates.
Short answer:
Painfully. You must convert the datestring, particularly the character month, into a numeric value, then you can compare the date values.
... Or more easily, by storing your dates using a format like “2018-01-01” which can be natively compared within the SQL via SQLite date functions.
Longer Answer
This page shows all the SQLite SQL date-time functions available. None of them produce or manipulate character month values. So your options are:
1) Select a group of records via SQL into a dataset; and then use your programming language to convert the date values in the dataset to a format whose values are comparable; then compare them.
This would have poor performance for anything except very small data queries, but would probably be reasonably simple to implement, and have no data conversion necessary. (To specifically answer the question you asked, this is the best solution for an app selecting few and small datasets.)
2) Create a SQLite function to do the date format conversion. To do that, you use SQLite's C API. See more how-to discussion here.
I haven't done this myself, and I wouldn’t recommend this route just due to the learning curve, but also due to the type & purpose of SQLite and its capabilities. (To specifically answer the question you asked, this is the best solution for not few and not small datasets.)
3) (This option does not answer your specific question) Convert your stored date values to a natively SQL comparable format. Like: “2018-01-01”. Then you can use the SQLite date-time functions for adjacent date comparisons.
Sample:
select mySQLTableDate, myOtherSQLTableDate
date(mySQLTableDate,'+1 day'), -- YYYY-DD-MM stored format
date(strftime('%Y-%m-%d', mySQLTableDate),'+1 day') -- many stored formats
from mySQLTable
where select mySQLTableDate = date(myOtherSQLTableDate,'+1 day')
Answering your question in terms of the goal rather than the specific question :) , my recommendation is to Use This Solution, especially if you are scanning a lot of data. See more about SQLite Date types here, but for dates with no time, I just store them as the string “2018-01-01”. I’m working with .js, and this is very simple to convert to/from a .js Date object.
I am trying to put a large data frame into a new table of a database. It could be done simply done via:
dbWriteTable(conn=db,name="sometablename",value=my.data)
However, I want to specify the Primary keys, foreign keys and the column Types like Numeric, Text and so on.
Is there any thing I can do? Should I create a table with my columns first and then add the data frame into it?
RSQlite assumes you have already your data.frame table all set before writing it to disk. There is not much to specify in the writing query. So, I visualise two ways, either before firing a query to write it, or after. I usually write the table from R to disk, then I polish it using dbGetQuery to alter table attributes. The only problem with this workflow is that Sqlite has very limited feature for altering tables.
I have a database name Team which has 40 tables . How can I connect to that database and refer to particular table without using sqlquerry. By the use of R data Structures.
I am not sure what do you mean with "How can I connect to that database and refer to particular table without using sqlquerry".
I am not aware of a way to "see" DB tables as R dataframes or arrays or whatever without importing the tuples first through some sort of query (in SQL) - this seems to be the most practical way to use R with DB data (without going to the hassle of exporting these as .csv files first, and re-read them in R).
There are a couple ways to import data from a DB to R, so that the result of a query becomes a R data structure (including proper type conversion, ideally).
Here is a short guide on how to do that with SQL-R
A similar brief introduction to the DBI family
I am using sqlite3 (maybe sqlite4 in the future) and I need something like dynamic tables.
I have many tables with the same format: values_2012_12_27, values_2012_12_28, ... (number of tables is dynamic) and I want to select dynamically the table that receives some data.
I am using _sqlite3_prepare with INSERT INTO ? VALUES(?,?,?). Ofcourse this fails to compile (syntax error near ?). There is a nice and simple way to do this in sqlite ?
Thanks
Using SQL parameters is not possible for identifiers such as table or column names.
If you don't want to keep so many prepared statements around, just prepare them on the fly whenever you need one.
If your database were properly normalized, you would have a single big values table with an extra date column.
This organization is usually to be preferred, unless you have measured both and found that the better performance (if it actually exists) outweighs the overhead of managing multiple tables.