I have data sources that provide date information in the format YYYYMM, for example, 201805. What is the best Impala datatype to deal with such data?
I could use string but that would restrict the ability to do date math on that data field.
Please suggest.
Related
How do I compare dates in SQLite database which are stored in DD-MMM-YYYY format e.g. 10-OCT-2017?
I want to compare dates and select rows of particular date and adjacent dates.
Short answer:
Painfully. You must convert the datestring, particularly the character month, into a numeric value, then you can compare the date values.
... Or more easily, by storing your dates using a format like “2018-01-01” which can be natively compared within the SQL via SQLite date functions.
Longer Answer
This page shows all the SQLite SQL date-time functions available. None of them produce or manipulate character month values. So your options are:
1) Select a group of records via SQL into a dataset; and then use your programming language to convert the date values in the dataset to a format whose values are comparable; then compare them.
This would have poor performance for anything except very small data queries, but would probably be reasonably simple to implement, and have no data conversion necessary. (To specifically answer the question you asked, this is the best solution for an app selecting few and small datasets.)
2) Create a SQLite function to do the date format conversion. To do that, you use SQLite's C API. See more how-to discussion here.
I haven't done this myself, and I wouldn’t recommend this route just due to the learning curve, but also due to the type & purpose of SQLite and its capabilities. (To specifically answer the question you asked, this is the best solution for not few and not small datasets.)
3) (This option does not answer your specific question) Convert your stored date values to a natively SQL comparable format. Like: “2018-01-01”. Then you can use the SQLite date-time functions for adjacent date comparisons.
Sample:
select mySQLTableDate, myOtherSQLTableDate
date(mySQLTableDate,'+1 day'), -- YYYY-DD-MM stored format
date(strftime('%Y-%m-%d', mySQLTableDate),'+1 day') -- many stored formats
from mySQLTable
where select mySQLTableDate = date(myOtherSQLTableDate,'+1 day')
Answering your question in terms of the goal rather than the specific question :) , my recommendation is to Use This Solution, especially if you are scanning a lot of data. See more about SQLite Date types here, but for dates with no time, I just store them as the string “2018-01-01”. I’m working with .js, and this is very simple to convert to/from a .js Date object.
For a booking plugin I need to store dates (start date and end date) for each booking. I want to store bookings in a custom post type, so I do not need to make my own tables in the database.
So I will store the dates in a meta.
I could store the date in mysql format, Ymd, or unix timestamp. But which way is best? Which way would be the Wordpress way?
ps: would this also work best for dates stored in settings?
When querying BETWEEN custom DATE meta fields, WordPress requires the date format to be YYYY-MM-DD. If you plan on using a compare argument of BETWEEN, I recommend storing in this format. Otherwise, it really depends on your use case.
According to this article, it's best to convert Date into Epoch time in order to use it is range query in DocumentDB. However, as recently the range query on Sting values has been added to DocumentDB, it is necessary to do convert date-time to epoch (as long as all date-time values have the same format and are in UTC format)?
This is similar to this question, where the accepted answer suggests using strings as you point out.
But to answer your question more specifically, DocumentDB cannot store JavaScript Date objects because it only stores pure JSON and Date is not a part of the JSON spec. So, you (or your client API) needs to do something with Date objects. By default, the node.js and .NET clients will convert Date objects to ISO-8601 formatted strings so using strings is actually a bit easier than Epoch. Just send the Date object to the database. The one trick to keep in mind here is that it's not converted back into a Date object when you read it. It comes back as a string. You have to do the conversion yourself. In JavaScript, this is easy. Just call new Date(yourDateString). Not sure about .NET or the other platforms.
The program I am currently designing use the dd/mm/yyyy date format, while Sqlite standard format is yyyy-mm-dd. My program make use of quite a lot of date calculations using julianday('yyyy-mm-dd'). I know I could convert the dd/mm/yyyy format to yyyy-mm-dd by using SUBSTR(X,Y) manipulation or by using the code of the language I am designing the db front-end; but i wish to avoid those. Any Idea?
You should always store dates (and timestamps) using native date format that is provided by database engine for following reasons:
Native formats permit native date arithmetic functions to work.
Native formats permit indexes to be consistently applicable, so you can use date comparisons efficiently and use operators like BETWEEN.
Native formats take less space to store on disk. For SQLite, storing date as real number of days from 4174 BC or as integer number of seconds since Jan 1st, 1970 takes 8 bytes. For your representation, it will take at least 10 bytes.
While SQLite does not really have true native date/datetime type (which is big omission in my opinion), it does have 3 permissible formats: TEXT, REAL or INTEGER that are still treated (to some extent) as native datetime formats, and all advantages outlined above still apply.
When you need to display dates in your application, you should use libraries provided by your scripting or other programming languages that know how to display dates in desired format.
In other words, use database to store, compare and retrieve data, and use your application to render it in desired format.
I don't like how SQLAlchemy treats datetime values with sqlite databases. It stores the values in plain string format. At the same time sqlite recommends using julianday for storing datetime values.
Is there an easy way to change the SQLAlchemy's behaviour here?
PS. Shall I worry about it? May be noone is dealing with julianday just because it's not necessary?
Actually it is not SQLAlchemy that stores dates as plain strings; SQLite itself does not support date types. It's important that you understand that from the outset; Sqlite does provide some functions for dealing with dates, but those are dates stored as text. That's why SQLAlchemy does some magic in transforming the dates to and from python's datetime type: Per the SQLAlchemy's documentation:
SQLite does not have built-in DATE,
TIME, or DATETIME types, and pysqlite
does not provide out of the box
functionality for translating values
between Python datetime objects and a
SQLite-supported format. SQLAlchemy’s
own DateTime and related types provide
date formatting and parsing
functionality when SQlite is used. The
implementation classes are SLDateTime,
SLDate and SLTime. These types
represent dates and times as ISO
formatted strings, which also nicely
support ordering. There’s no reliance
on typical “libc” internals for these
functions so historical dates are
fully supported.
As for using the Julian calendar as opposed to the Gregorian calendar. Are you sure you want that? Might you mean Gregorian dates?