I am receiving some values in my R process and I want to compute them asynchronously. I am using promises and future package.
This is how my current code looks like:
arr = list()
i=0
while(i < 10)
{
a = read messages from KAFKA topic
arr[[i]] = future(DoSomething(a))
i = i + 1
}
Now, arr contains a list of promises
How do I get value() of the promise that has resolved first (and so on)?
Something like promise.race in javascript.
Edit: I just re-read your question and saw that you were asking about getting the first result, not just all results. Below is the code for getting that. It's a while loop that waits for any result to be ready and then moves forward when there's a result ready.
There is also a function called promise_race in the promises package, but the issue with the promises package is that it can only output results. You can't get the value produced back into a variable for further computations in the main thread.
require(future)
plan(multiprocess)
longRunningFunction <- function(value) {
random1<- runif(min= 5 ,max = 30,n = 1)
Sys.sleep(random1)
return(value)
}
arr = list()
#changed starting number to 1 since R lists start at 1, not 0
i=1
#If the number of futures generated is more than the number of cores available, then the main thread will block until the first future completes and allows more futures to be started
while(i < 6)
{
arr[[i]] = future(longRunningFunction(i))
i = i + 1
}
while(all(!resolved(arr))){ }
raceresults_from_future<-lapply(arr[resolved(arr)], value)
print(paste("raceresults_from_future: ",raceresults_from_future) )
Is there any options in doing date comparison in Go? I have to sort data based on date and time - independently. So I might allow an object that occurs within a range of dates so long as it also occurs within a range of times. In this model, I could not simply just select the oldest date, youngest time/latest date, latest time and Unix() seconds compare them. I'd really appreciate any suggestions.
Ultimately, I wrote a time parsing string compare module to check if a time is within a range. However, this is not faring to well; I've got some gaping issues. I'll post that here just for fun, but I'm hoping there's a better way to time compare.
package main
import (
"strconv"
"strings"
)
func tryIndex(arr []string, index int, def string) string {
if index <= len(arr)-1 {
return arr[index]
}
return def
}
/*
* Takes two strings of format "hh:mm:ss" and compares them.
* Takes a function to compare individual sections (split by ":").
* Note: strings can actually be formatted like "h", "hh", "hh:m",
* "hh:mm", etc. Any missing parts will be added lazily.
*/
func timeCompare(a, b string, compare func(int, int) (bool, bool)) bool {
aArr := strings.Split(a, ":")
bArr := strings.Split(b, ":")
// Catches margins.
if (b == a) {
return true
}
for i := range aArr {
aI, _ := strconv.Atoi(tryIndex(aArr, i, "00"))
bI, _ := strconv.Atoi(tryIndex(bArr, i, "00"))
res, flag := compare(aI, bI)
if res {
return true
} else if flag { // Needed to catch case where a > b and a is the lower limit
return false
}
}
return false
}
func timeGreaterEqual(a, b int) (bool, bool) {return a > b, a < b}
func timeLesserEqual(a, b int) (bool, bool) {return a < b, a > b}
/*
* Returns true for two strings formmated "hh:mm:ss".
* Note: strings can actually be formatted like "h", "hh", "hh:m",
* "hh:mm", etc. Any missing parts will be added lazily.
*/
func withinTime(timeRange, time string) bool {
rArr := strings.Split(timeRange, "-")
if timeCompare(rArr[0], rArr[1], timeLesserEqual) {
afterStart := timeCompare(rArr[0], time, timeLesserEqual)
beforeEnd := timeCompare(rArr[1], time, timeGreaterEqual)
return afterStart && beforeEnd
}
// Catch things like `timeRange := "22:00:00-04:59:59"` which will happen
// with UTC conversions from local time.
// THIS IS THE BROKEN PART I BELIEVE
afterStart := timeCompare(rArr[0], time, timeLesserEqual)
beforeEnd := timeCompare(rArr[1], time, timeGreaterEqual)
return afterStart || beforeEnd
}
So TLDR, I wrote a withinTimeRange(range, time) function but it's not working totally correctly. (In fact, mostly just the second case, where a time range crosses over days is broken. The original part worked, I just realized I'd need to account for that when making conversions to UTC from local.)
If there's a better (preferably built in) way, I'd love to hear about it!
NOTE:
Just as an example, I solved this issue in Javascript with this function:
function withinTime(start, end, time) {
var s = Date.parse("01/01/2011 "+start);
var e = Date.parse("01/0"+(end=="24:00:00"?"2":"1")+"/2011 "+(end=="24:00:00"?"00:00:00":end));
var t = Date.parse("01/01/2011 "+time);
return s <= t && e >= t;
}
However I really want to do this filter server-side.
Use the time package to work with time information in Go.
Time instants can be compared using the Before, After, and Equal
methods. The Sub method subtracts two instants, producing a Duration.
The Add method adds a Time and a Duration, producing a Time.
Play example:
package main
import (
"fmt"
"time"
)
func inTimeSpan(start, end, check time.Time) bool {
return check.After(start) && check.Before(end)
}
func main() {
start, _ := time.Parse(time.RFC822, "01 Jan 15 10:00 UTC")
end, _ := time.Parse(time.RFC822, "01 Jan 16 10:00 UTC")
in, _ := time.Parse(time.RFC822, "01 Jan 15 20:00 UTC")
out, _ := time.Parse(time.RFC822, "01 Jan 17 10:00 UTC")
if inTimeSpan(start, end, in) {
fmt.Println(in, "is between", start, "and", end, ".")
}
if !inTimeSpan(start, end, out) {
fmt.Println(out, "is not between", start, "and", end, ".")
}
}
For comparison between two times use time.Sub()
// utc life
loc, _ := time.LoadLocation("UTC")
// setup a start and end time
createdAt := time.Now().In(loc).Add(1 * time.Hour)
expiresAt := time.Now().In(loc).Add(4 * time.Hour)
// get the diff
diff := expiresAt.Sub(createdAt)
fmt.Printf("Lifespan is %+v", diff)
The program outputs:
Lifespan is 3h0m0s
http://play.golang.org/p/bbxeTtd4L6
For case when your interval's end date doesn't contains hours like
"from 2017-01-01 to whole day of 2017-01-16" it's better to adjust interval's end to midnight of the next day to include all milliseconds like this:
if now.After(start) && now.Before(end.Add(24 * time.Hour).Truncate(24 * time.Hour)) {
...
}
It's possible to compare date using int64 of Unix epoch with seconds granularity. If you need more exact comparison like milisecons or microseconds etc. I guess that
#Oleg Neumyvakin's answer is perfect.
if expirationDate.Unix() > time.Now().Unix() {
...
}
If you're interested in comparing whether a time is close to another for test purposes, you can use testify assert.WithinDuration for this. For example:
expectedTime := time.Now()
actualTime := expectedTime.Add(100*time.Millisecond)
assert.WithinDuration(t, expectedTime, actualTime, 1*time.Second) // pass
assert.WithinDuration(t, expectedTime, actualTime, 1*time.Millisecond) // fail
Otherwise the implementation of assert.WithinDuration can be re-used in your code to determine how close two times are (subtracting one date from the other gives the time difference):
func WithinDuration(expected, actual time.Time, delta time.Duration) bool {
dt := expected.Sub(actual)
return dt >= -delta && dt <= delta
}
Recent protocols prefer usage of RFC3339 per golang time package documentation.
In general RFC1123Z should be used instead of RFC1123 for servers that insist on that format, and RFC3339 should be preferred for new protocols. RFC822, RFC822Z, RFC1123, and RFC1123Z are useful for formatting; when used with time.Parse they do not accept all the time formats permitted by the RFCs.
cutOffTime, _ := time.Parse(time.RFC3339, "2017-08-30T13:35:00Z")
// POSTDATE is a date time field in DB (datastore)
query := datastore.NewQuery("db").Filter("POSTDATE >=", cutOffTime).
As explained in the theread we could use github.com/google/go-cmp/cmp package for dates comparison in tests.
func TestDates(t *testing.T) {
date, _ := time.Parse(time.RFC3339, "2021-11-05T12:00:00+02:00")
dateEqual, _ := time.Parse(time.RFC3339, "2021-11-05T11:00:00+01:00")
dateNotEqual, _ := time.Parse(time.RFC3339, "2021-11-05T12:00:01+02:00")
assertDates(t, date, dateEqual) //pass
assertDates(t, date, dateNotEqual) //fail
}
func assertDates(t *testing.T, expected, actual time.Time) {
t.Helper()
if diff := cmp.Diff(expected, actual); diff != "" {
t.Errorf("mismatch (-expected +actual):\n%s", diff)
}
}
package main
import (
"fmt"
"time"
)
func main() {
fmt.Println("Hello World")
maxRep := 5
repPeroid := 6
expiry := maxRep * repPeroid
fmt.Println("Expiry: ", expiry)
fmt.Println(time.Now())
CorrIdtime := time.Now().Add(time.Second * time.Duration(expiry)).Format(time.RFC3339)
Notifytime := time.Now().Add(2 * time.Second * time.Duration(expiry)).Format(time.RFC3339)
fmt.Println(CorrIdtime)
fmt.Println(Notifytime)
if CorrIdtime < Notifytime {
fmt.Println("Discarded")
} else {
fmt.Println("Accepted")
}
}
Per proposal time: add Time.Compare and related commit, time.Compare will be added in the new release (Go 1.20)
// Compare compares the time instant t with u. If t is before u, it returns -1;
// if t is after u, it returns +1; if they're the same, it returns 0.
func (t Time) Compare(u Time) int {
Sample
var t1, t2 Time
result := t1.Compare(t2)
I am using MySQLdb to talk to mysql database and I am able to retrieve dynamically all the result sets.
My problem is that once I get the result set, there are a couple columns which are declared as timestamps in mysql but when it is retrieved, it turns to None.
I have two columns, both are declared timestamps but one returns correct data while other returns None. Both utime and enddate are declared timestamps but utime does not return correctly while enddate does.
['utime', 'userstr', 'vstr_client', 'enddate']
((None, '000102030ff43260gg0809000000000004', '7.7.0', '1970-01-01 12:00:00.000000'))
def parse_data_and_description(cursor, data):
res = []
cols = [d[0] for d in cursor.description]
print cols
print data
for i in data:
res.append(OrderedDict(zip(cols, i)))
return res
def call_multi_rs(sp, args):
rs_id=0;
conn = connect()
cursor = conn.cursor()
try:
conn.autocommit(True)
cursor.execute ("CALL %s%s" % (sp, args))
while True:
rs_id+=1
data = cursor.fetchone( )
listout = parse_data_and_description(cursor, data)
print listout
if cursor.nextset( )==None:
# This means no more recordsets available
break
Finally after nobody answered or tried finding more information, I went ahead and looked for more solutions and found that the MySQLdb library converts the datatypes from sql to python and there is a bug which does not convert the timestamp.
I still do not know why one of them is converted and the other is not. If somebody can figure that out, please update this.
But here is the modification that needs to be done when connecting to the mysql database.
MySQLdb can't serialize a python datetime object
try:
import MySQLdb.converters
except ImportError:
_connarg('conv')
def connect(host='abc.dev.local', user='abc', passwd='def', db='myabc', port=3306):
try:
orig_conv = MySQLdb.converters.conversions
conv_iter = iter(orig_conv)
convert = dict(zip(conv_iter, [str,] * len(orig_conv.keys())))
print "Connecting host=%s user=%s db=%s port=%d" % (host, user, db, port)
conn = MySQLdb.connect(host, user, passwd, db, port, conv=convert)
except MySQLdb.Error, e:
print "Error connecting %d: %s" % (e.args[0], e.args[1])
return conn
I stumbled upon the same problem: Retrieving data of the DATETIME(1)-type returns None.
Some research brought up MySQLdb-Bug #325. According to that bug tracker, the issue is still open (has been for over 2 years now), but the comments provide a working solution:
In times.py of the MySQLdb package, you need to insert some lines to handle microseconds like this:
def DateTime_or_None(s):
if ' ' in s:
sep = ' '
elif 'T' in s:
sep = 'T'
else:
return Date_or_None(s)
try:
d, t = s.split(sep, 1)
if '.' in t:
t, ms = t.split('.',1)
ms = ms.ljust(6, '0')
else:
ms = 0
return datetime(*[ int(x) for x in d.split('-')+t.split(':')+[ms] ])
except (SystemExit, KeyboardInterrupt):
raise
except:
return Date_or_None(s)
def TimeDelta_or_None(s):
try:
h, m, s = s.split(':')
if '.' in s:
s, ms = s.split('.')
ms = ms.ljust(6, '0')
else:
ms = 0
h, m, s, ms = int(h), int(m), int(s), int(ms)
td = timedelta(hours=abs(h), minutes=m, seconds=s,
microseconds=ms)
if h < 0:
return -td
else:
return td
except ValueError:
# unpacking or int/float conversion failed
return None
def Time_or_None(s):
try:
h, m, s = s.split(':')
if '.' in s:
s, ms = s.split('.')
ms = ms.ljust(6, '0')
else:
ms = 0
h, m, s, ms = int(h), int(m), int(s), int(ms)
return time(hour=h, minute=m, second=s, microsecond=ms)
except ValueError:
return None
What I cannot explain, though, is your original query working on one column and not on the other.. Maybe, the second does not have any microsecond-information in it?
Let's say I have a closure:
def increment = {value, step ->
value + step
}
Now I want to loop over every item of my integers collection, increment it with 5, and save new elements to a new collection:
def numbers = [1..10]
def biggerNumbers = numbers.collect {
it + 5
}
And now I want to achieve the same result but by means of using increment closure. How can I do this?
Should be something like this (wrong code below):
def biggerNumbers = numbers.collect increment(it, 5) //what's the correct name of 'it'??
The solution to your problem would be nesting your call of increment in a closure:
def biggerNumbers = numbers.collect {increment(it, 5)}
If you wanted to pass a premade closure to the collect you should have made it compatible with collect - accepting a single parameter that is:
def incrementByFive = {it + 5}
def biggerNumbers = numbers.collect incrementByFive
mojojojo has the right answer, but just thought I'd add that this looks like a good candidate for currying (specifically using rcurry)
If you have:
def increment = {value, step ->
value + step
}
You can then curry the right-hand parameter of this function with:
def incrementByFive = increment.rcurry 5
And then, you can do:
def numbers = 1..10
def biggerNumbers = numbers.collect incrementByFive
Just thought it might be of interest ;-)
The main issue is that [1..10] creates a List<IntRange> which you are trying to increment. You should collect on the IntRange directly (note the lack of brackets):
(1..10).collect { it + 5 }
Or with curry:
def sum = { a, b -> a + b }
(1..10).collect(sum.curry(5))
I have two fields in my text file which are
timestamp number
The format of timestamp is hh:mm:ss.mmm
some sample records are
18:31:48.345 0.00345
18:31:49.153 0.00123
18.32:23.399 0.33456
I want to print out averages of records which are no more than 30 second apart. what is a good and fast way of doing it
Here is a starting point in awk. I know you can optimize code better.
count == 0 { startTime = timeToSeconds($1) }
{ currentTime = timeToSeconds($1)
elapsedTime = currentTime - startTime
if (elapsedTime > 30.0) {
calculateAverage()
startTime = timeToSeconds($1)
}
print
sum += $2
count++
}
END { calculateAverage() }
function timeToSeconds(timeString) {
# Convert a time string to number of seconds
split(timeString, tokens, ":")
seconds = tokens[1]*3600.0 + tokens[2]*60.0 + tokens[3]
return seconds
}
function calculateAverage() {
# Use & modify global vars: count, sum
average = sum / count
printf "Average: %.4g\n\n", average
sum = 0.0; count = 0
}
I would start by using some scripting language that has built-in date/time 'operations'. For instance, in Ruby you could easily do:
require 'time'
t,n = gets.chomp.split(/\s+/)
ts1 = Time.parse(t)
# ...
t,n = gets.chomp.split(/\s+/)
ts2 = Time.parse(t)
Which now allows you to do things like:
diff = ts2 - ts1
if diff > 30
# difference is greater than 30 seconds
end
Ruby Time objects can be used in context (float, int, String, etc) so it is trivial to start doing calculations as if the parsed dates are actually numeric values.