I want to run an external process in Julia for a given amount of time (say 10 seconds) and capture its output.
I tried something like readchomp(`timeout 5 sleep 10`) (substitute sleep 10 by an arbitrary external process). However, a ProcessFailedException is thrown if the process was killed by timeout and I can not read the output of the program it produced until it was killed.
run only throws an exception if wait = true (which it is by default). You'll want to set that kwarg to false and then manually wait on the process, while redirecting stdout to e.g. an IOBuffer via pipeline:
shell> cat loop.sh
#!/bin/bash
for i in {1..10}; do
echo $i
sleep 1
done
julia> b = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> p = run(pipeline(`timeout 5 ./loop.sh`, stdout = b), wait = false); wait(p)
julia> p
Process(`timeout 5 ./loop.sh`, ProcessExited(124))
julia> String(take!(b))
"1\n2\n3\n4\n5\n"
As I'm trying to get two different outputs by running same function with different parameters asynchronously using asyncio in python.
Example:
import asyncio
import time
async def async_generate(sleep_time):
s = time.perf_counter()
await asyncio.sleep(0)
result = generate(sleep_time)
elapsed = time.perf_counter() - s
print(f"segment executed in {elapsed:0.2f} seconds.")
return result
def generate(sleep_time):
time.sleep(sleep_time)
return sleep_time
async def main():
task1 = asyncio.ensure_future(async_generate(1))
task2 = asyncio.ensure_future(async_generate(2))
await asyncio.gather(task1,task2)
return task1.result(), task2.result()
loop = asyncio.get_event_loop()
s = time.perf_counter()
tasks = loop.run_until_complete(main())
print(tasks)
elapsed = time.perf_counter() - s
print(f"final executed in {elapsed:0.2f} seconds.")
loop.close()
Output:
segment executed in 1.00 seconds.
segment executed in 2.00 seconds.
(1, 2)
final executed in 3.00 seconds.
In above example, expecting 2 secs to complete the code execution but taking 3 secs, can you suggest me any changes ?
Thanks in advance
The documentation for redirect_stdouton version 1.1.0, that I am currently using, does not seem to give an example of how to use that function. Maybe I missed it?
I want to capture the output of println and get it back as a string.
Here is an example:
julia> VERSION
v"1.1.0"
julia> (rd, wr) = redirect_stdout();
julia> println("This is a test.")
julia> # Get back the string "This is a test."
julia> # s = do_something_with_rd(rd)
julia> # s == "This is a test."
julia> # true
Any suggestions?
Edit
Based on the accepted answer below, here is a complete solution to my question:
julia> original_stdout = stdout;
julia> (rd, wr) = redirect_stdout();
julia> println("This is a test.")
julia> s = readline(rd)
"This is a test."
julia> s == "This is a test."
true
julia> redirect_stdout(original_stdout);
julia> println("Test of orig. stdout.")
Test of orig. stdout.
Edit 2: A More Complete Example
Here is an example of testing a variety of print and println function outputs using redirection of stdout. Thanks to #Bogumił Kamiński for his answer and edit that made this more clear to me:
using Test
# Test redirect_stdout.
#testset "Example tests using redirect_stdout" begin
original_stdout = stdout;
(read_pipe, write_pipe) = redirect_stdout();
print("Using print function.")
println("Using println function.")
println("Second use of println function.")
println("Line 1.\nLine 2.\nLine 3.\nEND")
println("""
This is new line 1.
This is new line 2. Next a Char = """)
print('A')
redirect_stdout(original_stdout);
close(write_pipe)
#test readline(read_pipe) == "Using print function.Using println function."
#test readline(read_pipe) == "Second use of println function."
#test read(read_pipe, String) == "Line 1.\nLine 2.\nLine 3.\nEND\n" *
"This is new line 1.\nThis is new line 2. Next a Char = \nA"
end
# Suppress unnecessary output when this file.
return nothing
Here is the output:
julia> include("test_redirect_stdout.jl")
Test Summary: | Pass Total
Example tests using redirect_stdout | 3 3
Just run readline on rd (or any other reading operation).
You have just to be careful that read operations on rd are blocking, i.e. the terminal will seem to hang when the operation cannot be completed. One solution is to use #async for this. For instance:
julia> (rd, wr) = redirect_stdout();
julia> #async global x = readline(rd) # if we did not put #async here the terminal would be blocked
Task (runnable) #0x0000000004e46e10
julia> x # x is yet undefined as readline is waiting for an input
ERROR: UndefVarError: x not defined
julia> println("something") # we feed data to stdout
julia> x # and readline has finished its work and bound the value to variable x
"something"
Of course if you know exactly that the data you want to read in is there just run readline or some other function and all will work without #async.
EDIT
Given the comments from SalchiPapa I think it is also add this pattern of possible usage as it is simplest to think of IMO:
original_stdout = stdout
(rd, wr) = redirect_stdout();
println("This is a test 1.")
println("This is a test 2.")
println("This is a test 3.")
redirect_stdout(original_stdout)
# you can still write to wr
println(wr, "This is a test 4.")
# you have to close it to make the read non-blocking
close(wr)
# the pipe is redirected to original stdout and wr is closed so this is non-blocking
s = read(rd, String)
You could use sprint AKA "string print":
https://docs.julialang.org/en/v1/base/io-network/index.html#Base.sprint
julia> sprint(println, "This is a test")
"This is a test\n"
I'm working on a mobile game backend in Erlang. For each HTTP request, it might need to query different data sources, such as PostgreSQL, MongoDB and Redis. I want to make independent calls to these data sources in parallel but cannot find a clear Erlang way to do it.
For example,
handle_request(?POST, <<"login">>, UserId, Token) ->
% Verify token from PostgreSQL
AuthResult = auth_service:login(UserId, Token),
% Get user data such as outfits and level from MongoDB
UserDataResult = userdata_service:get(UserId),
% Get messages sent to the user from Redis
MessageResult = message_service:get(UserId),
% How to run the above 3 calls in parallel?
% Then wait for all their results here?
% Combine the result and send back to client
build_response(AuthResult, UserDataResult, MessageResult).
Each service will eventually call into the corresponding data driver (epgsql, eredis, mongo_erlang) that end up with some pooboy:transaction and gen_server:call there. How to design these services module are not decided yet either.
I want to make sure that the 3 data calls above could run in parallel, and then the handle_request function waits for all of those 3 calls finish, and then call build_response. How could I do that properly?
As a reference, in NodeJS, I might do this
var authPromise = AuthService.login(user_id, token);
var userDataPromise = UserdataService.get(user_id);
var messagePromise = MessageService.get(user_id);
Promise.all(authPromise, userDataPromise, messagePromise).then( function(values) {
buildResponse(values);
}
In Scala I might do this
val authFuture = AuthService.login(userId, token)
val userDataFuture = UserdataService.get(userId)
val messageFuture = MessageService.get(userId)
for {
auth <- authFuture
userData <- userDataFuture
message <- messageFuture
} yield ( buildResponse(auth, userData, message )
Apparently, I'm thinking the problem as a promise/future/yield problem. But I was told that if I'm looking for a Promise in Erlang I might be going in the wrong direction. What would be the best practice in Erlang to achieve this?
How to make parallel calls in Erlang and wait for all of the results?
You can employ stacked receive clauses. Erlang will wait forever in a receive clause until a message arrives from a process (or you can specify a timeout with after)--which is similar to awaiting a promise in nodejs:
-module(my).
-compile(export_all).
all_results() ->
Pid1 = spawn(?MODULE, getdata1, [self(), {10, 20}]),
Pid2 = spawn(?MODULE, getdata2, [self(), 30]),
Pid3 = spawn(?MODULE, getdata3, [self()]),
[receive {Pid1, Result1} -> Result1 end,
receive {Pid2, Result2} -> Result2 end,
receive {Pid3, Result3} -> Result3 end].
getdata1(From, {X, Y}) ->
%% mimic the time it takes to retrieve the data:
SleepTime = rand:uniform(100),
io:format("Sleeping for ~w milliseconds~n", [SleepTime]),
timer:sleep(SleepTime),
From ! {self(), X+Y}. %% send the data back to the main process
getdata2(From, Z) ->
SleepTime = rand:uniform(100),
io:format("Sleeping for ~w milliseconds~n", [SleepTime]),
timer:sleep(SleepTime),
From ! {self(), Z+1}.
getdata3(From) ->
SleepTime = rand:uniform(100),
io:format("Sleeping for ~w milliseconds~n", [SleepTime]),
timer:sleep(SleepTime),
From ! {self(), 16}.
Note that this code:
[receive {Pid1, Result1} -> Result1 end,
receive {Pid2, Result2} -> Result2 end,
receive {Pid3, Result3} -> Result3 end].
is equivalent to:
R1 = receive {Pid1, Result1} ->
Result1
end,
R2 = receive {Pid2, Result2} ->
Result2
end,
R3 = receive {Pid3, Result3} ->
Result3
end,
[R1, R2, R3].
In the shell:
~/erlang_programs$ erl
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V9.3 (abort with ^G)
1> c(my).
my.erl:2: Warning: export_all flag enabled - all functions will be exported
{ok,my}
2> timer:tc(my, all_results, []).
Sleeping for 66 milliseconds
Sleeping for 16 milliseconds
Sleeping for 93 milliseconds
{96356,[30,31,16]}
3> timer:tc(my, all_results, []).
Sleeping for 57 milliseconds
Sleeping for 30 milliseconds
Sleeping for 99 milliseconds
{100153,[30,31,16]}
4> timer:tc(my, all_results, []).
Sleeping for 66 milliseconds
Sleeping for 31 milliseconds
Sleeping for 24 milliseconds
{66426,[30,31,16]}
timer:tc() returns the time that a function takes to execute in microseconds (1,000 microseconds = 1 millisecond) along with the function's return value. For instance, the first time that all_results() was called it took 96.4 milliseconds to complete, while the individual processes would have taken 66+16+93=175+ milliseconds to finish if executed sequentially.
In Elixir I can run code asynchronously like this
defmodule Async do
def example do
process_id = Task.async(fn ->
#some code you want its result later
end)
#do some stuff
async_result = Task.await(process_id)
end
end
and, if i don't need any results I can do like this
defmodule Async do
def example do
process_id = Task.start_link(fn ->
#some code you want its result later
end)
#do some stuff
:ok
end
end
What's the equivalent of above in Julia lang?
if you do not care about the result, you can use #async:
function foo()
sleep(100)
sum(1:100)
end
julia> #async foo()
Task (runnable) #0x00007f983b9d5f90
julia>
in the above example you get back the control of the terminal, without having to wait until the end of the execution of foo()
if you want to know the result, and keep an asynchronous behavior, you can use Task, schedule and fetch together:
julia> a() = sum(1:10000)
a (generic function with 1 method)
julia> b = Task(a)
Task (runnable) #0x00007f983b9d5cf0
julia> schedule(b)
Task (done) #0x00007f983b9d5cf0
julia> fetch(b)
50005000
Here is a small piece of code I use to illustrate the different behaviors:
module async
function example1()
a() = #async sleep(2)
b = Task(a)
schedule(b)
println(fetch(b))
sleep(4)
fetch(b)
end
function example2()
a() = sleep(2)
b = Task(a)
schedule(b)
fetch(b)
end
function example3()
a() = sum(1:10000)
b = Task(a)
schedule(b)
fetch(b)
end
end;
when I run this code, I get:
julia> async.example1()
Task (runnable) #0x00007f983b9d5510
Task (done) #0x00007f983b9d5510
julia> async.example2()
julia> async.example3()
50005000
async.example2() does not return any result, but keep the terminal busy around 2 seconds since fetch waits for the task to complete before giving back the hand.