I have a SQLite 3 database on MacOS with timestamp data (typically something like 279020203.539467).
The documentation says that dates can be stored as
REAL - as Julian day numbers, the number of days since noon in Greenwich
on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
And it looks like this is what I've got.
I want to export this and import it into other databases.
I am assuming the timestamp datatype is not compatible between database engines, so some conversion has to happen somewhere, and I am working under the assumption that it'd be best to do that while exporting the data from the SQLite database.
But I can't figure out how to do this conversion.
I've looked at this answer which refers to this forum post which indicates that in SQLite you could do something like
select datetime('40660.9454658044', '+2415018 days', '+12 hours', 'localtime');
(Maybe 2415018 is the number of days between November 24, 4714 B.C. and some other magical date...)
However, replacing the timestamp string in this example with what I have results in null. Presumably because '279020203.539467' is some other kind of timestamp. It is also some magnitudes larger than the example.
But how to convert this to a usable date? I know it should be around 2011/2012.
Interpreting the data as an "integer" (seconds since 1970-01-01) gives 1978 so that is not correct either.
UPDATE: I've found that
288773834.371606 should be 2010-02-25 07.57
296636121.950064 should be 2010-05-27 08.55
(CET if that matters).
The good news is: To convert a Julian date to "regular" date format you could use datetime(strftime('%J',jtime)). FYI Here's the doc for sqlite date and time functions. But there's bad news.
A NASA Calculator computes the Julian date of 2010-02-25 to be 2455246. It computes the civil date of 288773834 as Sept 2, 785907 A.D. sqlite doesn't give that same result using the above notation, but it doesn't give a "date".
Even though the numbers look like Julian date notation, they are not any dates in our lifetimes.
DATEDIF("2010-02-25 07.57"; "2010-05-27 08.55"; "d")
This gives 91 days which works out to be 7862400 seconds, almost exactly the same as the difference between the two timestamps (296636121.950064 - 288773834.371606 = 7862287.578458).
On MacOS native timestamps are seconds from 2001-01-01 (Jan 1 2001).
And
DATEDIF("2001-01-01 00:00"; "2010-02-25 07.57"; "d")
gives 3342 days which is 288748800 seconds, near, but not exactly the timestamp but that difference is caused by
DATEDIF only caters for whole days
CET is one hour off
Correcting for that we get around 25000 seconds more to add, which takes us almost exactly to the timestamp for 2010-02-25 07.57.
So the gist of this is that SQLite on MacOS stores timestamps as MacOS native timestamps which are seconds since start of MacOS epoch (2001-01-01 00.00). This is probably caused by the application that created the data was not using SQLite timestamps but MacOS native dates and storing them in the database as any other data.
Converting this to some other format should be trivial, either during the export, a conversion of the exported file or during imports.
Possibly it would be easiest to convert the date on export from the original database using
select ...
datetime(table.timestamp_field, 'unixepoch', '+31 years')
from ...
Related
I'm creating book writing software.
The problem is, when user create story that happen say in year 200 BCE [ CE / BCE wiki], how to store this date in SQLite so I can sort it like normal DateTime.
The documentation for the built-in date functions says:
These functions only work for dates between 0000-01-01 00:00:00 and 9999-12-31 23:59:59.
However, SQLite does not have a separate data type for dates; it just uses numbers or strings, and interprets them as dates only when you apply a date function to them.
If you do not actually need to use the SQLite date functions, you can use any type and format, as long as it sorts correctly.
Strings like yyyy-mm-dd do not sort correctly for BCE dates, so you have to use numbers (Julian days, or Unix timestamps, or any other format).
I'm trying to import a table from Microsoft Access (.accdb) to R.
The code that I use is:
library(RODBC)
testdb <- file.path("modelEAU Database V.2.accdb")
channel <- odbcConnectAccess2007(testdb)
WQ_data <- sqlFetch(channel, "WaterQuality")
It seems that it works but the problem is importing date and time data. Into the Access file there are two columns, one with date field (dd/mm/yyyy) and another one with time field (hh:mm:ss) and when I import them in R, in date column appears the date with yyyy-mm-dd format and into the time column the format is 1899-12-30 hh:mm:ss. Also, R can't recognise these formats as a variable and I can't work with them.
Also, I tried the mdb.get function but it didn't work as well.
Does somebody know how to import the data in R from Access defining the date and time format ? Any idea how to import the Access file as a text file?
Note: I'm working with with Office 2010 and R version 2.14.1
Thanks a lot in advanced.
Look at the result of runing str on your data frame. That will tell you more about how the data is actually stored. Generally dates and times are stored as a number from an origin date (Access uses 30 Dec. 1899 because MS thought that 1900 was a leap year). Sometimes it is stored as the number of days since the origin with time being represented as a fraction of the day, other times it is the number of seconds (or miliseconds) since the origin.
You will need to see how the data was sent (whether access and odbc converted to strings first, or sent days or seconds), then you will have a better feel for how to work with these (possibly converting) in R.
There is an article in the June 2004 edition of R News (the predecesor to the R Journal) that details the common ways to deal with dates and times in R and could be very useful to you.
You should decide what you want to end up with, a single column of DateTimes, 2 columns with numbers, 2 columns with characters, etc.
I kind of assumed it was a string, so I compared it as a string, but not surprisingly it failed. I believe thats how it works in Mysql. I could be wrong as I haven't worked on it in a while. In either case, how can I check if dates are equal in SQLite? I will be using it in a WHERE clause.
SELECT a._id, b._id, b.start_date,a.event_name, b.start_time,
b.end_date, b.end_time, b.location FROM events_info b INNER JOIN events a ON
a._id=b.event_id WHERE b.start_time = '6:00';
(added space to make it easier to look at)
SQLite doesn't have a dedicated DATETIME type. Normally what people do is make sure they store the date as a formatted string that is consistent; for example, YYYY-MM-DD hh:mm:ss. If you do so, as long as you're consistent, then you can compare dates directly:
SELECT * FROM a WHERE q_date < '2013-01-01 00:00:00';
This works because even though the comparison is technically an alphabetical comparison and not a numeric one, dates in a consistent format like this sort alphabetically as well as numerically.
For such a schema, I would suggest storing dates in 24-hour format (the above example is midnight). Pad months, days, and hours with zeros. If your dates will span multiple timezones, store them all in UTC and do whatever conversion you need client-side to convert them to the local time zone.
Normally dates and times are stored all in one column. If you have to have them separated for whatever reason, just make sure you dates are all consistent and your times are all consistent. For example, dates should all be YYYY-MM-DD and times should all be hh:mm:ss.
The reason that YYYY-MM-DD hh:mm:ss is the preferred format is because when you go from the largest date interval (years) to the smallest (seconds), you can index and sort them very easily and with high performance.
SELECT * FROM a WHERE q_date = '2012-06-04 05:06:00';
would use the index to hone in on the date/time instead of having to do a full table scan. Or if they're in two separate rows:
SELECT * FROM a WHERE q_date = '2012-06-04' AND q_time = '05:06:00';
The key is to make sure that the dates and times are in a consistent format going into the database. For user-friendly presentation, do all conversion client-side, not in the database. (For example, convert '2012-06-04 05:06:00' to "1:06am Eastern 6/4/2012".)
If this doesn't answer question, could you please post the exact format that you're using to store your dates and times, and two example dates that you're trying to compare that aren't working the way you expect them to?
Sqlite can not compare dates directly. we need to convert them in seconds as well as integer also.
Example
SELECT * FROM Table
WHERE
CAST(strftime('%s', date_field) AS integer) <=CAST(strftime('%s', '2015-01-01') AS integer) ;
From Datatypes In SQLite Version 3:
1.2 Date and Time Datatype
SQLite does not have a storage class set aside for storing dates and/or times. Instead, the built-in Date And Time Functions of SQLite are capable of storing dates and times as TEXT, REAL, or INTEGER values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can chose to store dates and times in any of these formats and freely convert between formats using the built-in date and time functions.
If you look at the examples in Date And Time Functions, something like this should get you close to what you want (which, I'm assuming, is 6:00 of the current day):
WHERE b.start_time = date('now', 'start of day', '+6 hours')
I exported my Firefox bookmarks, and the 'dateAdded' fields look like this:
1260492675000000
1260492675000000
1266542833000000
They're too big to be a Unix timestamp, and I can't make sense of them. What are they? (I want to convert it into something usable/readable.)
It is PRTime.
This type is a 64-bit integer representing the number of microseconds since the NSPR epoch, midnight (00:00:00) 1 January 1970 Coordinated Universal Time (UTC). A time after the epoch has a positive value, and a time before the epoch has a negative value.
PRTime as described on this page.
You can extract the time using the f3e tool if you can find a link to it.
Dates in DB2 AS/400 are an integer, containing the number of days since sometime around the turn of the 20th century.
Question 1: Does anyone know the IBM DB2/AS400 "zero" date? e.g.:
12/30/1899
12/31/1899
1/1/1900
Question 2: Given an "AS/400" date (e.g. 40010) how can you convert that to a CLR DateTime?
DateTime d = new DateTime(40010); //invalid
Some other "zero" dates are:
OLE Automation: 12/30/1899
SQL Server: 1/1/1900
I don't think AS/400 dates are stored internally as some number of days from an epoch date1 (this is the more common term for what you are calling "zero date"). As Tracy Probst said, this is definitely NOT what date fields in native AS/400 physical files look like.2
But that's immaterial if whatever method you are using to extract the data is giving it to you as the number of days since an epoch. Ideally, you should find out what the intended date is by looking directly at the AS/400, or asking someone who can. If the date on the AS/400 is 2009-07-30 and what you are getting is 40022, then you can be pretty confident the epoch date is Jan 1, 1900. If you are getting 40024, then the epoch is Dec 30, 1899. (Though it's of course best to compare a bunch of dates, preferably from different years to guard against possible use of Julian dates.)
Also, as Tracy commented on his own answer, it's exceedingly common for dates to be stored in generic numeric fields (which is what I would guess if your retrieval method is reporting Decimal as the data type), in which case it really has nothing to do with DB2's internal date format anyway. You should be aware that by far the most common date formats stored in AS/400 numeric fields are the following, or variations thereof:
yyyymmdd (Gregorian, ISO 4-digit year)
mmddyy (Gregorian, U.S. 2-digit year)
yyyyddd (so-called Julian, 4-digit year)
yyddd (so-called Julian, 2-digit year)
yymmdd
cyymmdd (IBM's crazy invention with century flag)
The ddd in the Julian dates is the number of days from the beginning of year. The c in IBM's crazy date is 0 for 19yy or 1 for 20yy. I have not heard of anyone who stores days-since-epoch on "The Four Hundred" but maybe you've encountered a convert from another platform. The mainframe heritage of the AS/400 strongly favors human-readable dates.
1The AS/400 (now called IBM i) does have its own data type for dates, and this data type actually does consist internally of a number of days from an epoch. But that epoch is many thousands of years in the past, not somewhere near the turn of the 20th century, and not even near the beginning of the Common Era. IBM likes to call this number of days the Scaliger number, but for most people who study this stuff, it's called the Julian Day Number. As you may have noticed from the main part of my answer, IBM uses the word "Julian" to mean something completely different (and not even related to the Julian calendar). Namely, IBM's so-called "Julian date" is really the ordinal date from ISO 8601.
2The internal format of the date data type is very low-level and mostly hidden from the user (including most programmers). The DSPPFM command, which ostensibly shows the "actual contents" of a file, is at least one step "too late": the value it reports has already been converted from the internal, 4-byte "Scaliger number" to a human-readable form.
Question 1:
I have no idea what the start date is for DB2. Google isn't very helpful anyway. Don't you have any sample data you could use to figure it out?
Update: are you sure the date is stored as a number of days? I found this page that suggests otherwise.
Question 2:
Assuming 1900-01-01 as the start date in this example, where days is the AS/400 date value.
DateTime myDate = new DateTime(1900, 1, 1).AddDays(days);
I don't know the answer for 1. But for 2, you can do something like this:
private DateTime AS400 = new DateTime(1900, 1, 1);
...
DateTime myClrDT = AS400.AddDays(days);
Question 1:
As far as I can tell, there is no "zero date" in an AS/400 phsyical file. If I do a DSPPFM on a phsyical file with a timestamp field in it, the value is stored as a readable timestamp in the format yyyy-MM-ddhh.mm.ss. For example: "2005-08-0207.06.33" for 08/02/2005 at 7:06:33 AM. There can be a zero-date within a particular programming language and that's really where you need to focus. The AS/400 ODBC driver returns the date in a SQL_TYPE_TIMESTAMP field.
Question 2:
It should be as simple as:
DateTime d = Convert.ToDateTime(reader["DateField"]);
I invite other C# experts to edit the response with better C# code.
I've just 5 months of experience in DB2(working on AS400), so i just can show you something
about the way we work with dates. It's true that we consider the 'zero' date in our calculation of the date fields. In our system, the 'zero' date =12/31/1971 0:00.
I don't know if this is the 'only' 'zero' date in AS400.
In our system files, the date we use is stored as the number of days from the 'zero' date(length=5).
So, every time we have to get the date field, from a specified file, we convert this field to get the date in the format : dd/mm/yyyy or yyyy-mm-dd(it depends from the environment where we execute the query). The function is:
date(field+719892), where field is the field where we store the date and 719892 is the number of days we add after each unconverted date we use(it seems like it is the number of days between x-12/31/1971, you can calculate x).
I'll give you on more example:
select date(15+719892) as date1 from library1.file1
The result is: date1=1972-01-15
marc_s had a comment that confused the "zero" dates with "minimum" dates in SQL Server. Just so everyone gets to see the example:
SELECT
CAST(0 AS datetime) AS dateTimeZero,
CAST(0 AS smalldatetime) AS smallDateTimeZero
dateTimeZero smallDateTimeZero
======================= ===================
1900-01-01 00:00:00.000 1900-01-01 00:00:00