I would like to understand how ES stores date values internally in its indexes. Does it convert to UTC?
I have a field "t" of type date. Here's the mapping:
"t": { "type" : "date" },
Now when I insert/add a document to ES, how does it store in its indexes.
"t" : "1427700477165" (milliseconds generated from Date.now() function). Does ES recognize its epoch time in UTC and stores as is?
"t" : "2015-03-29T23:59:59" (i adjust mapping date format accordingly)- how does ES store this. If it converts to UTC, how does it know what time zone this date is and convert it to UTC? Does ES get the default time zone from the machine its running on?
Thank you!
Internally (within an index) Elasticsearch stores all dates as numbers in epoch format - i.e. the number of milliseconds since 01 Jan 1970 00:00:00 GMT.
However Elasticsearch by default also stores your raw JSON posted message as well - so when returning the _source you'll see whatever was posted to Elasticsearch.
To be able to import date strings into the epoch format you need to specify the format in your mapping, for example either a predefined date format:
"t": { "type" : "date", "format" : "basic_date_time" }
for yyyyMMdd'T'HHmmss.SSSZ.
or specify a custom date format:
"t": { "type" : "date", "format" : "YYYY-MM-dd" }
If no format is specified, the default date parsing used is
ISODateTimeFormat.dateOptionalTimeParser.
Multiple date formats can be specified in the mapping - e.g. yyyy/MM/dd
HH:mm:ss||yyyy/MM/dd
If no timezone is specified then Elasticsearch assumes UTC
Related
I'm working with data flow in azure data factory and i tried to convert an epoch formatted timestamp to date.
the value of the timestamp is '1574067907751' and i tried expressions :
toDate(toTimestamp(1574067907751*1000l))
or
toDate(toTimestamp(toInteger('1574067907751')*1000l,'yyyy-MM-dd HH:mm:ss'))
there is any other way to do that ?
https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-expression-builder#convert-to-dates-or-timestamps
"To convert milliseconds from epoch to a date or timestamp, use toTimestamp(). If time is coming in seconds, multiply by 1,000.
toTimestamp(1574127407*1000l)
The trailing "l" at the end of the previous expression signifies conversion to a long type as inline syntax."
I have a flowfile attribute which is a UTC datetime in the format of yyyy-MM-dd HH:mm:ss.SSS
I need to convert this to a unix timestamp.
How can this be done? I know its possible to convert Unix to the above format using Jolt:
"time": "${time:format('yyyy-MM-dd HH:mm:ss.SSS')}"
however, im not sure how to do this in reverse?
Working with attributes in this way uses the NiFi Expression Langauge (not Jolt).
See the docs here https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
${time:format('yyyy-MM-dd HH:mm:ss.SSS')}
Uses Expression Language to format the time attribute to the given SimpleDateFormat string.
${time:toNumber()}
Uses Expression Language to convert the given Date object to Epoch Millis.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#tonumber
Can't format the below date using moment.js, the below statement returns Invalid Date
moment('20171206T062406927Z').format('D-MMM-YYYY');
Please help me on this.
You need to tell moment which format your date string is in:
moment('20171206T062406927Z', 'YYYYMMDD[T]HHmmssSSSZ', true).format('D-MMM-YYYY');
Edit: updated as per #VincenzoC comment to ensure the timestamp is parsed in UTC
Also fix: use HH for 24-hour format (not hh), and pass a third true parameter to ensure the timestamp is parsed in strict mode.
I've read thru various posts on here in regards to similiar issues but none have solved my problem.
I manipulate the moment.js date object, and then store it as timestamp.
BUT, when I try to read in that timestamp again, I get that deprecated warning.
""Deprecation warning: moment construction falls back to js Date. This is discouraged and will be removed in upcoming major release. Please refer to https://github.com/moment/moment/issues/1407 for more info."
I've tried toDate(), format(), moment(myTimeStamp, 'ddd, DD MMM YYYY HH:mm:ss ZZ'); --> all generate the warning...
So, for example, my timestamp will look like this:
const timestamp = '1458586740000'
when I read that back and try to parse out the month/day/year, then the hour/min am/pm, etc... I need to get that timestamp into a moment.js object. Nothing is working for me. Any ideas.
How can I get this timestamp: '1458586740000', into a moment.js object so I can extract date date from it as I need?
EDIT: this is how I am storing the timestamp. So I would need to retrieve it from this.
let timeStamp = Moment(state[_Date])
.add({ hour: state[AMPM] === 'PM'
? +state[Hour] + 12
: state[Hour] ,
minute: state[Min] }).format('x')
The X token indicates a unix timestamp in seconds, and the x token indicates a unix millisecond timestamp (offset).
You appear to have a millisecond timestamp, so you would make a moment out of it by doing the following:
var a = moment('1458586740000', 'x')
It works without ' as well:
var a = moment(1458586740000, 'x')
You can also not specify the x and it should work:
moment(1458586740000)
Because you have a unix offset (milliseconds), not a unix timestamp (seconds), moment.unix is not what you want.
Then you can do the following:
a.format()
"2016-03-21T13:59:00-05:00"
Or you can use any of the other formatting tokens listed here to output whatever result you would like: http://momentjs.com/docs/#/displaying/format/
Based on the code you presented, I think you may be having problems because your timestamp is stored as a string (in ''). Parsing as a string causes an invalid date error, because it attempts to match ISO 8601 format and fails. Specifying that 'x' token will cause it to assume unix offset and work correctly.
I insert my date/time data into a CHAR column in the format: '6/4/2015 2:08:00 PM'.
I want that this should get automatically converted to format:
'2015-06-04 14:08:00' so that it can be used in a query because the format of DATETIME is YYYY-MM-DD hh:mm:ss.fffff.
How to convert it?
Given that you've stored the data in a string format (CHAR or VARCHAR), you have to decide how to make it work as a DATETIME YEAR TO SECOND value. For computational efficiency, and for storage efficiency, it would be better to store the value as a DATETIME YEAR TO SECOND value, converting it on input and (if necessary) reconverting on output. However, if you will frequently display the value without doing computations (including comparisons or sorting) it, then maybe a rococo locale-dependent string notation is OK.
The key function for converting the string to a DATETIME value is TO_DATE. You also need to look at the TO_CHAR function because that documents the format codes that you need to use, and because you'll use that to convert a DATETIME value to your original format.
Assuming the column name is time_string, then you need to use:
TO_DATE(time_string, '%m/%d/%Y %I:%M %x') -- What goes in place of x?
to convert to a DATETIME YEAR TO SECOND — or maybe DATETIME YEAR TO MINUTE — value (which will be further manipulated as if by EXTEND as necessary).
I would personally almost certainly convert the database column to DATETIME YEAR TO SECOND and, when necessary, convert to the string format on output with TO_CHAR. The column name would now be time_value (for sake of concreteness):
TO_CHAR(time_value, '%m/%d/%Y %I:%M %x') -- What goes in place of x?
The manual pages referenced do not immediately lead to a complete specification of the format strings. I think a relevant reference is GL_DATETIME environment variable, but finding that requires more knowledge of the arcana of the Informix product set than is desirable (it is not the first thing that should spring to anyone's mind — not even mine!). If that's correct (it probably is), then one of %p and %r should be used in place of %x in my examples. I have to get Informix (re)configured on my machine to be able to test it.