I have log events in CloudWatch whcih have certain fields like #timestamp and #message.
#timestamp looks like this:
2022-08-09T08:49:51.971+02:00
and #message like this:
2022-08-09 06:49:49.376 INFO [xyz, 123abc456] xxxxxxx
I can extract the time in #message by using something like this:
| parse #message /(?<messagetst>.+) INFO/
What I want to do is to
convert the timestamp of #timestamp to UTC and a datetime format,
convert the #message messagetst to a datetime format,
substract #timestamp from messagetst.
Is this possible in CloudWatch and how can I do this?
Related
I'm trying to convert a string datetime, to a timestamp. try_to_timestamp isn't converting dates in the yyyy-mm-dd hh:mm:ss AM/PM format. I've only been able to solve this by striping out the time and casting it as a type time and concatenating it back with the date. Anyone know of a Snowflake function to handle this?
select try_to_timestamp('2019-11-18 4:01:29 PM +0000')
your timestamp doesn't quite fit the ISO or RTC formats so it's not automatically detecting it. You can manually put in your format though. Should look something like this:
select to_timestamp('2019-11-18 4:01:29 PM +0000', 'YYYY-MM-DD HH12:MI:SS AM TZHTZM')
For information on the various formats SnowFlake uses: https://docs.snowflake.net/manuals/user-guide/date-time-input-output.html
EDIT:
You may or may not want to cast it as a timestamp_tz (timestamp with timezone) to maintain timezone information
select to_timestamp_tz('2019-11-18 4:01:29 PM +0200', 'YYYY-MM-DD HH12:MI:SS AM TZHTZM');
Since try_to_timestamp does not support a format, you can set it before calling the
try_to_timestamp function. The example below is setting it at the session level.
alter session set TIMESTAMP_INPUT_FORMAT = 'YYYY-MM-DD HH12:MI:SS AM TZHTZM';
select try_to_timestamp('2019-11-18 4:01:29 PM +0000');
I have a vendor oracle DB storing values to a column creation_timestamp as timestamp(6) when i create an extract of that table and load it to HDFS and create hive schema with that column defined as timestamp it does not populate the field. However if I use string it is all there. I have other MSSQL datetime data that successfully stored as timestamp in hive so I am just a little unsure why the oracle timestamp data is not populating.
The oracle timestmap looks like this: 07-JAN-15 05.55.20.732754000 PM
I have a feeling hive does not like the month as letters as the documentation states:Strings: JDBC compliant java.sql.Timestamp format "YYYY-MM-DD HH:MM:SS.fffffffff" (9 decimal place precision)
How can I convert the oracle timestamp(6):07-JAN-15 05.55.20.732754000 PM to the java.sql.Timestamp format shown above?
I was able to alter my session and then export the table, hive has now accepted it as a timestamp
ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD HH24:MI:SS.FF';
07-JAN-15 05.55.20.732754000 PM is now: 2015-01-07 17:55:20.732754000
Use to_date in hive.
Example :
select TO_DATE(from_unixtime(UNIX_TIMESTAMP('7-MAR-13', 'd-MMMM-yy'))) from table_name
Refer : date functions
I cannot seem to figure out why datetime does not work for me on some data I imported from CSV. I have a column, TIMESTAMP, which is of type datetime.
Select TIMESTAMP from GPS limit 1 <-This gives me a time, "6/29/2009 00:00:00"
Select datetime(TIMESTAMP) from GPS limit 1 <- This gives me a pink field in SQLite manager, which seems empty.
Select datetime('now') from GPS limit 1 <- This gives me the current date and time. ("2012-12-19 20:45:17") It is formatted differently than my other data - is there a datatype issue?
What is going on? Did my "Timestamp" data not actually get converted into a DATETIME object? Why is it stored as text? Is there a way to fix this?
SQLite does not have a native date/time type; dates are stored either as numbers or as strings.
To be understood by SQLite's built-in date functions, date strings must have a format like yyyy-mm-dd hh:mm:ss.
I want to convert sqlite query to Ormlite query.
SELECT * FROM Test where strftime('%m-%Y',Date)='11-2001'
I could not format date column like above query.
How to format Date column in Ormlite as MM-yyyy?
Thanks.
If that is the exact SQL that you want to use then you can use the Where.raw(...) method:
QueryBuilder<Test, Integer> qb = testDao.queryBuilder();
qb.where().raw("strftime('%m-%Y',Date) = '11-2001'");
List<Test> results = qb.query();
However, this only seems to work if the date field is stored as a DATE_STRING type:
#DatabaseField(dataType = DataType.DATE_STRING)
Date date;
The issue is that by default the Xerial JDBC driver is storing the date in the format:
2012-07-19 09:58:18.36
Which does not [quite] match one of the approved Sqlite formats which are:
YYYY-MM-DD
YYYY-MM-DD HH:MM
YYYY-MM-DD HH:MM:SS
YYYY-MM-DD HH:MM:SS.SSS
YYYY-MM-DDTHH:MM
YYYY-MM-DDTHH:MM:SS
YYYY-MM-DDTHH:MM:SS.SSS
HH:MM
HH:MM:SS
HH:MM:SS.SSS
now
DDDDDDDDDD
When you change it to DataType.DATE_STRING then it will be stored as the following which seems to work:
2012-07-19 10:03:49.000991
For more info, see the Sqlite docs on date functions. Unfortunately, the documentation does not fully explain that the database values need to be in a certain format:
http://www.sqlite.org/lang_datefunc.html
I have an ASP.NET program that is writing date to an SQLExpress Database date field in DD/MM/YYYY.
When I look at the data in SQL Express it is stored as mm/dd/yyyy.
How can I configure it to store in DD/MM/YYYY format?
This is not possible, as the date is internally stored as a number, the DD/MM/YYYY or MM/DD/YYYY format is only the display format of the data. You can, however, change the way the data are converted to a string by SQL functions...
You are seeing the a rendered, localised version of an internal date representation (numbers of days since 01 Jan 1900 basically).
Don't worry about it. You'll get date back to your client (in an internal date representation) and this can be formatted how you like there.
Store the data normally. When you retrieve the data, do something like this on the code:
dateField.ToString("dd/MM/yyyy")
And your result will be: 11/04/2011
Maybe you can select that like this:
select CONVERT(varchar(12) , getdate(), 103 )