I'm having trouble parsing this datetime field in BigQuery that takes in two different time zone formats.
It is stored as a string. The timestamps look like either of these.
DateTime:
Thu Mar 03 2022 18:18:38 GMT+0000 (GMT)
Thu Mar 03 2022 00:04:32 GMT-0800 (Pacific Standard Time)
What I want:
DateTime:
Thu Mar 03 2022 10:18:38 GMT-0800 (Pacific Standard Time)
Thu Mar 03 2022 00:04:32 GMT-0800 (Pacific Standard Time)
I get errors with datetime and parse_timestamp trying to do something like this:
parse_timestamp("%a %b %d %E4Y %T", DateTime , 'US/Pacific')
How can I get this to work?
Consider below options
select *,
datetime(parse_timestamp("%a %b %d %Y %T GMT%Z", regexp_replace(DateTime, r' \([ \w]+\)', '')), 'US/Pacific'),
format_datetime("%a %b %d %Y %T", datetime(parse_timestamp("%a %b %d %Y %T GMT%Z", regexp_replace(DateTime, r' \([ \w]+\)', '')), 'US/Pacific')),
from your_table
if applied to sample data in your question - output is
Related
Through Python i'm trying to convert the future date into another format and subtract with current date but it's throwing error.
Python version = Python 3.6.8
from datetime import datetime
enddate = 'Thu Jun 02 08:00:00 EDT 2022'
todays = datetime.today()
print ('Tpday =',todays)
Modified_date1 = datetime.strptime(enddate, ' %a %b %d %H:%M:%S %Z %Y')
subtract_days= Modified_date1 - todays
print (subtract_days.days)
Output
Today = 2022-02-02 08:06:53.687342
Traceback (most recent call last):
File "1.py", line 106, in trusstore_output
Modified_date1 = datetime.strptime(enddate1, ' %a %b %d %H:%M:%S %Z %Y')
File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime
(data_string, format))
ValueError: time data ' Thu Jun 02 08:00:00 EDT 2022' does not match format ' %a %b %d %H:%M:%S %Z %Y'
During handling of the above exception, another exception occurred:
Linux server date
$ date
Wed Feb 2 08:08:36 CST 2022
Point 6 in the Documentation tells that not all Timezone formats are available to be parsed by strptime.
%Z [...]
So someone living in Japan may have JST, UTC, and GMT as valid values, but probably not EST. It will raise ValueError for invalid values.
If possible, you could get the server date with the -u flag and parse the UTC timestamp.
date -u
Mi 2. Feb 14:39:11 UTC 2022
PS:
Also watch out for the leading whitespace in your strings.
If EDT is available on your system, the Value Error could be a result of the a mixup between enddate and enddate1.
' Thu Jun 02 08:00:00 EDT 2022' vs. enddate = 'Thu Jun 02 08:00:00 EDT 2022'
Unfortunately, only a subset of timezones is supported by strptime.
If you can ensure that the input does not contain any other timezones than EDT or EST, you could replace these by the corresponding UTC offsets and use %z instead of %Z:
from datetime import datetime
date_str = "Thu Jun 02 08:00:00 EDT 2022"
date_str = date_str.replace("EDT", "-0400")
date_str = date_str.replace("EST", "-0500")
date_parsed = datetime.strptime(date_str, "%a %b %d %H:%M:%S %z %Y")
# 2022-06-02 08:00:00-04:00
print(date_parsed)
Trying to replace moment.js in my Angular application with Luxon to reduce bundle size.
I have come across a case where the two libraries produce a different output, and I am not sure why.
moment.js produces a date that is one hour ahead.
const activeToDateTimeString = '2014-08-06T13:07:04';
let foo1 = moment(activeToDateTimeString).utcOffset(-5, true);
let foo2 = DateTime.fromISO(activeToDateTimeString, {zone: 'America/New_York'}).setZone('America/New_York', { keepLocalTime: true });
let foo3 = DateTime.fromJSDate(new Date(activeToDateTimeString)).setZone('America/New_York', { keepLocalTime: true });
let foo4 = DateTime.fromISO(activeToDateTimeString).setZone('America/New_York', { keepLocalTime: true });
console.log(foo1.toDate());
console.log(foo2.toJSDate());
console.log(foo3.toJSDate());
console.log(foo4.toJSDate());
Output:
Wed Aug 06 2014 14:07:04 GMT-0400 (Eastern Daylight Time)
Wed Aug 06 2014 13:07:04 GMT-0400 (Eastern Daylight Time)
Wed Aug 06 2014 13:07:04 GMT-0400 (Eastern Daylight Time)
Wed Aug 06 2014 13:07:04 GMT-0400 (Eastern Daylight Time)
Why does moment.js produce a different output in this case?
let foo1 = moment(activeToDateTimeString).utcOffset(-4, true);
This would correct your code, but as you're moving to Luxon the daylight time changes won't effect you in the future.
Right now (19 mar 2020) New York is 4 hours behind UTC as from the 8th March 2020 it entered Eastern Daylight Time from Eastern Standard Time.
If New York was in Eastern Standard Time at the moment your code would output the same times.
I have a dataset that looks like this:
datetime count
18:28:20.602 UTC DEC 08 2016 1
20:42:32.017 UTC DEC 08 2016 5
15:33:40.691 UTC DEC 08 2016 1
17:11:54.008 UTC DEC 08 2016 3
20:28:57.861 UTC DEC 08 2016 0
.
.
.
.
The datetime column is in the string format. I'm having difficulty in converting it to a timestamp.
How do I write a Impala/Hive query so that I get the data between '18:28:00.000 UTC DEC 08 2016' to '18:33:00.000 UTC DEC 08 2016'
With Hive:
cast(from_unixtime(unix_timestamp(SHITTY_FORMAT, 'HH:mm:ss.SSS zzz MMM dd yyyy'), 'yyyy-MM-dd HH:mm:ss.SSS') as Timestamp)
...will translate your shitty String format into a UNIX timestamp, then into String standard format (in local timezone because that's the Hive convention), then into a Timestamp.
There is no easier way, unfortunately. And you may have some edge cases because of the 1h overlap in summer/winter times.
Source: the Hive documentation, of course...
With Impala (which does not support the zzz format modifier):
cast(from_unixtime(unix_timestamp(regexp_replace(SHITTY_FORMAT, ' UTC ', ' '), 'HH:mm:ss.SSS MMM dd yyyy'), 'yyyy-MM-dd HH:mm:ss.SSS') as Timestamp)
...will translate your shitty String format into a UNIX timestamp, assuming that all your inputs are in UTC, then into String standard format (in UTC timezone because that's the Impala convention), then into a Timestamp.
I'm working on a Parser which Parses log files from a game so I can do analysis on auctions made within the game, however the date format that's being written by the logger seems to be causing problems as the format seems to be custom written for the logger, an example datetime stamp looks like: [Wed Nov 23 23:26:10 2016] I try to Parse it with:
func (r *AuctionReader) extractSaleInformation(line string) {
fmt.Println("Extracting information from: ", line)
// Format mask for output
layout := "DD-MM-YYYY hh:mm:ss"
// Replace the square brackets so we're just left with the date-time string
date := strings.TrimSpace(strings.Replace((strings.Split(line, "]")[0]), "[", "", -1))
fmt.Println(time.Parse(date, layout))
}
When I attempt to Parse the above date-time string I get the following error:
0001-01-01 00:00:00 +0000 UTC parsing time "DD-MM-YYYY hh:mm:ss" as "Wed Nov 23 23:26:10 2016": cannot parse "DD-MM-YYYY hh:mm:ss" as "Wed Nov "
How am I able to get the parser to recognise this seemingly custom format, I will be saving this data to Mongo so I don't want to store the auction time as a string as I want to query the timestamps individually.
Golang handle all date formatting in a unique way - it uses the reference time Mon Jan 2 15:04:05 MST 2006 (01/02 03:04:05PM '06 -0700) to show the pattern with which to format/parse a given time/string.
So, to read the format "Wed Nov 23 23:26:10 2016" you would put the reference date into that format: "Mon Jan 2 15:04:05 2006", and then do:
t, _ := time.Parse("Mon Jan 2 15:04:05 2006", "Wed Nov 23 23:26:10 2016")
Then, to output it in the given format, if you wanted the format DD-MM-YYYY hh:mm:ss, you would put the reference time into that format: 02-01-2006 15:04:05, and then do:
t.Format("02-01-2006 15:04:05")
https://play.golang.org/p/VO5413Z7-z
So basically, the main change is
// Format mask for output
layout := "DD-MM-YYYY hh:mm:ss"
should be
// Format mask for output
layout := "02-01-2006 15:04:05"
and
time.Parse(date, layout)
should be
time.Parse(layout, date)
I have this datetime, or something that looks like it.
2014-11-17 23:02:03 +0000 UTC
I want to convert this to a time object and I've been unable to produce any output from time.Parse apart from:
0001-01-01 00:00:00 +0000 UTC
I've tried these layouts:
time.RFC3339
0001-01-01 00:00:00 0000 UTC
2016-10-10
time.UnixDate
And a few more - none have worked.
This is how I'm calling parse :
updatedAt, err := time.Parse(time.UnixDate, updatedAtVar)
How do I create a time object from a string?
Most likely you used a wrong layout, and you didn't check the returned error.
The layout must be this date/time, in the format your input time is:
Mon Jan 2 15:04:05 -0700 MST 2006
See this working code:
layout := "2006-01-02 15:04:05 -0700 MST"
t, err := time.Parse(layout, "2014-11-17 23:02:03 +0000 UTC")
fmt.Println(t, err)
Output (try it on the Go Playground):
2014-11-17 23:02:03 +0000 UTC <nil>
EDIT:
In your question you included a + sign in your input time (as part of the zone offset), but you have error with times of other formats.
Time.String() uses the following format string:
"2006-01-02 15:04:05.999999999 -0700 MST"
So either use this to parse the times, or use Time.Format() to produce your string representations where you can specify the layout, so you can use the same layout to parse the time strings.
2nd round:
You're including your time strings in URLs. The + sign is a special character in URL encoding: it denotes the space. So the + gets converted to space (and so it vanishes from your time string). Use proper URL encoding! Check out the net/url package, and this example.
Didn't see this yet but for those that don't know the formats, time has the formats builtin as constants. so you can reference them when parsing or formating.
time.Parse(time.RFC3339, <your time.Time object here>)
<time.Time object>.Format(time.RFC3339) //or other type of formats
Here they are for reference
ANSIC = "Mon Jan _2 15:04:05 2006"
UnixDate = "Mon Jan _2 15:04:05 MST 2006"
RubyDate = "Mon Jan 02 15:04:05 -0700 2006"
RFC822 = "02 Jan 06 15:04 MST"
RFC822Z = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
RFC850 = "Monday, 02-Jan-06 15:04:05 MST"
RFC1123 = "Mon, 02 Jan 2006 15:04:05 MST"
RFC1123Z = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
RFC3339 = "2006-01-02T15:04:05Z07:00"
RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
Kitchen = "3:04PM"
You are likely using the wrong layout. As explained in time.Parse, you need to specify a layout that helps Go to understand how the date passed as input is formatted.
There are predefined layouts (like the ones you were using), but none matches your input. Hence you need to define a custom layout.
A layout uses the following date as reference:
Mon Jan 2 15:04:05 MST 2006
The layout is nothing else that a representation of that date, that matches the representation of your input:
t, err := time.Parse("2006-01-02 15:04:05 -0700 MST", "2014-11-17 23:02:03 +0000 UTC")
Also remember to check err for errors. It's likely your attempts returned an error, but you didn't check it.
package main
import (
"fmt"
"log"
"time"
)
func main() {
t, err := time.Parse("2006-01-02 15:04:05 -0700 UTC", "2014-11-17 23:02:03 +0000 UTC")
if err != nil {
log.Fatalln(err)
}
fmt.Println(t)
}