In order to store a DateTime array to a HDF5 database, I first converted each DateTime to Int64 and did the reverse upon retrieval. The code looks as follows:
using Dates
a = Array{DateTime, 1}(undef, 2)
a[1] = DateTime("2020-01-01")
a[2] = DateTime("2020-01-02")
# Convert to Int64 array
msecs = [d.instant.periods.value for d in a]
# Code to store to hdf5 here (skipped for now)
# Code to retrieve from hdf5 here (skipped for now)
# Convert back to array of DateTime
b = [DateTime(0) + Millisecond(msec) for msec in msecs]
Question1: Is it the best way to convert back and forth between DateTime and Int64?
Question2: Can we directly store DateTime in hdf5 without conversion or is this the way to go?
Related
I want to convert an Int64 representing the number of microseconds passed since 12:00:00 midnight, January 1, 0001 (0:00:00 UTC on January 1, 0001, in the Gregorian calendar) into a Julia datetime.
julia> time = Dates.Microsecond(6369175082331949400)
julia> Dates.format(time, "yyyymmdd HH:MM:SS.sss")
If you need a DateTime, just make sure you have your Int64 correctly in milliseconds, and you can use the (undocumented) UTInstant constructor, and then later add back the fractional microseconds (comment: your example number, 6369175082331949400, seems big for recent Gregorian time in microseconds, it may be nanoseconds):
julia> using Dates
julia> t = now().instant
Dates.UTInstant{Millisecond}(63694318624788 milliseconds)
julia> dump(t)
Dates.UTInstant{Millisecond}
periods: Millisecond
value: Int64 63694318624788
julia> t2 = Dates.UTInstant(Millisecond(63691750823319))
Dates.UTInstant{Millisecond}(63691750823319 milliseconds)
julia> DateTime(t2)
2019-04-24T01:00:23.319
julia> t3 = DateTime(t2)+ Dates.Microsecond(494)
2019-04-24T01:00:23.319
You can get what you want using Dates.epochms2datetime and applying an adjustment to it for your case as shown below.
Lets take datetime_value as the date we are interested in getting:
datetime_value = Dates.DateTime(2019,1,1,0,0,0)
date_start = Dates.DateTime(1,1,1,0,0,0)
date_diff = datetime_value - date_start
This gives you a value of 63681897600000 milliseconds for date_diff. Now Dates.epochms2datetime considers start of epoch as 0000-01-01T00:00:00. So we need to add 1 Year and 1 Day to the result after using Dates.epochms2datetime to arrive at our datetime value from the milliseconds value:
julia> Dates.epochms2datetime(63681897600000) + Dates.Year(1) + Dates.Day(1)
2019-01-01T00:00:00
I'm not sure I completely understand the question, as Dates.Microsecond merely returns the Int64 value of a Date or Time. However, you can create the DateTime value from a specific date and then work from there. Subtraction is allowed for DateTime values and it returns the difference in milliseconds.
using Dates
dateThen = DateTime(1, 1, 1, 0, 0, 0)
dateNow = now(UTC)
diff = dateNow - dateThen
dump(diff * 1000)
Int64 63694261047549000 (or whatever time you run it.)
Using some of the ideas provided, I came up with:
function convert_datetime(time)::DateTime
num = div(time, 100000)
remainder = rem(time, 100000)
time = DateTime(Dates.UTInstant(Millisecond(num))) + Dates.Day(1)
# time = Dates.epochms2datetime(trade.date_time/100000) + Dates.Year(1) + Dates.Day(1)
time + Dates.Microsecond(remainder)
end
I am reading a line from a text file. It contains the date in YYYY-MM-DD format. I am trying to convert it to datetime object so as to find the difference between two dates.
l = datetime.strptime(last_execution_date,"%Y-%m-%d").date()
Its throwing an error:ValueError: unconverted data remains:
But when I am using below its working perfectly fine
l = datetime.strptime('2019-01-25',"%Y-%m-%d").date()
My complete code looks something like this:
def incoming_mails_duration():
f = open('last_script_execution_time.txt', 'r')
last_execution_date = f.readline()
print(last_execution_date)
print(type(last_execution_date))
l = datetime.strptime(last_execution_date,"%Y-%m-%d").date()
print(l)
print(type(l))
present_date = date.today()
delta_days = abs((present_date - l).days)
f.close()
Why I am getting the above error when I am passing the string as variable read from a file ?
It is because f.readline() returns string with \n in the end. You either have to strip the newline character or include it inside strptime format argument.
Solution 1:
last_execution_date = f.readline().strip()
Solution 2:
l = datetime.strptime(last_execution_date,"%Y-%m-%d\n").date() # Note \n
Note
Also it is good practice to open files with with statement. This is a safe way to handle files. File will be safely closed even if exception occurred inside with block.
with open(filepath) as f:
for line in f:
# Work with line here
pass
The title is all it is.
Assume I have a double:
let d = 11241241.10124012 :: Double
How do I get an instance of UTCTime corresponding to d, d as in unix seconds?
You need to convert the double to a fractional in order for it to be understood as a NominalDiffTime value (which is an instance of Fractional). Once it is understood as a NominalDiffTime value, then posixSecondsToUTCTime will handle the conversion (POSIXTime being a type synonym of NominalDiffTime).
import Data.Time.Clock
import Data.Time.Clock.POSIX(posixSecondsToUTCTime)
doubleToUTCTime d = posixSecondsToUTCTime $ realToFrac d
I'm new to Lua when I began to use OpenResty, I want to output a image and it's x,y coordinate together as one binary sequence to the clients, looks like: x_int32_bits y_int32_bits image_raw_data. At the client, I know the first 32 bits is x, the second 32 is y, and the others are image raw data. I met some questions:
How to convert number to 32 binary bits in Lua?
How to merge two 32 bits to one 64 bits sequence?
How to insert 64 bits to front of image raw data? And how to be fastest?
file:read("*a") got string type result, is the result ASCII sequence or like "000001110000001..." string?
What I'm thinking is like below, I don't know how to convert 32bits to string format same as file:read("*a") result.
#EgorSkriptunoff thank you, you opened a window for me. I wrote some new code, would you take a look, and I have another question, is the string merge method .. inefficient and expensive? Specially when one of the string is very large. Is there an alternative way to merge the bytes string?
NEW CODE UNDER #EgorSkriptunoff 's GUIDANCE
function _M.number_to_int32_bytes(num)
return ffi.string(ffi.new("int32_t[1]", num), 4)
end
local x, y = unpack(v)
local file, err = io.open(image_path, "rb")
if nil ~= file then
local image_raw_data = file:read("*a")
if nil == image_raw_data then
ngx.log(ngx.ERR, "read file error:", err)
else
-- Is the .. method inefficient and expensive? Because the image raw data maybe large,
-- so will .. copy all the data to a new string? Is there an alternative way to merge the bytes string?
output = utils.number_to_int32_bytes(x) .. utils.number_to_int32_bytes(y) .. image_raw_data
ngx.print(output)
ngx.flush(true)
end
file:close()
end
OLD CODE:
function table_merge(t1, t2)
for k,v in ipairs(t2) do
table.insert(t1, v)
end
return t1
end
function numToBits(num, bits)
-- returns a table of bits
local t={} -- will contain the bits
for b=bits,1,-1 do
rest=math.fmod(num,2)
t[b]=rest
num=(num-rest)/2
end
if num==0 then return t else return {'Not enough bits to represent this number'} end
end
-- Need to insert x,y as 32bits respectively to front of image binary sequence
function output()
local x = 1, y = 3
local file, err = io.open("/storage/images/1.png", "rb")
if nil ~= file then
local d = file:read("*a") ------- type(d) is string, why?
if nil == d then
ngx.log(ngx.ERR, "read file error:", err)
else
-- WHAT WAY I'M THINKING -----------------
-- Convert x, y to binary table, then merge them to one binary table
data = table_merge(numToBits(x, 32), numToBits(y, 32))
-- Convert data from binary table to string
data = convert_binary_table_to_string(data) -- HOW TO DO THAT? --------
-- Insert x,y data to front of image data, is data .. d ineffective？
data = data .. d
-------------------------------------------
ngx.print(data)
ngx.flush(true)
end
file:close()
end
end
I've got date and time in separate fields, in yyyyMMdd and HHmmss formats respectively. To parse them I think to construct a yyyy-MM-ddTHH:mm:ss string and feed this to joda-time constructor. So I am looking to get 1-st 4 digits, then 2 digits starting from the index 5, etc. How to achieve this? List.fromString(String) (which I found here) seems to be broken.
The substring method certainly can get you there but String in Scala 2.8 also supports all other methods on sequences. The ScalaDoc for class StringOps gives a complete list.
In particular, the splitAt method comes in handly. Here's a REPL interaction which shows how.
scala> val ymd = "yyyyMMdd"
ymd: java.lang.String = yyyyMMdd
scala> val (y, md) = ymd splitAt 4
y: String = yyyy
md: String = MMdd
scala> val (m, d) = md splitAt 2
m: String = MM
d: String = dd
scala> y+"-"+m+"-"+d
res3: java.lang.String = yyyy-MM-dd
Just use the substring() method on the string. Note that Scala strings behave like Java strings (with some extra methods), so anything that's in java.lang.String can also be used on Scala strings.
val s = "20100903"
val t = s.substring(0, 4) // t will contain "2010"
(Note that the arguments are not length and starting index, but starting index (inclusive) and ending index (exclusive)).
But if this is about parsing dates, why don't you just use java.text.SimpleDateFormat, like you would in Java?
val s = "20100903"
val fmt = new SimpleDateFormat("yyyyMMdd")
val date = fmt.parse(s) // will give you a java.util.Date object
If you're using Joda Time, you should be able to use
val date = DateTimeFormat.forPattern("yyyyMMdd, HHmmss")
.parseDateTime(field1 + ", " + field2)
For the more general problem of parsing Strings like this, it can be helpful to use a Regex (although I wouldn't recommend it in this case):
scala> val Date = "(\\d\\d\\d\\d)(\\d\\d)(\\d\\d)".r
Date: scala.util.matching.Regex = (\d\d\d\d)(\d\d)(\d\d)
scala> "20100903" match {
| case Date(year, month, day) => year + "-" + month + "-" + day
| }
res1: java.lang.String = 2010-09-03
val field1="20100903"
val field2="100925"
val year = field1.substring(1,5)
val month = field1.substring(5,7)
val day = ...
...
val toYodaTime = year + "-" + month+"-"+day+ ...