I am trying to understand the usage of #sync and #async macros in Julia. I am trying to get this MWE to work and the program does not terminate. Any help is appreciated.
function process_node(nodes, id)
#show id
sleep(1.0)
nodes[id] = true
return
end
function main()
nodes = Dict( i => false for i in 1:10 )
jobs = Channel{Int}(15)
for i in 1:10
put!(jobs, i)
end
#sync for id in jobs
#async process_node(nodes, id)
end
println("done")
end
main()
The program never gets to the line println("done"). I do not know why.
Thanks in advance.
There is nothing wrong about the use of #sync and #async in this example.
the loop for id in jobs never return because it blocks forever waiting endlessy for values no more inserted into the Channel.
Directly from the docs:
The returned Channel can be used as an iterable object in a for loop, in which case the loop variable takes on all the produced > values. The loop is terminated when the channel is closed.
One solution is to signal the end of the streams of values with a special Int value, for example -1 or if this is not possible with a nothing value, declaring jobs as Channel{Union{Int, Nothing}}.
function main()
nodes = Dict( i => false for i in 1:10 )
jobs = Channel{Int}(15)
for i in 1:10
put!(jobs, i)
end
put!(jobs, -1)
#sync for id in jobs
if id == -1
close(jobs)
else
#async process_node(nodes, id)
end
end
println("done")
end
Related
This is my idea
task=#task begin
while true
sleep(1)
foo()
if should_end
# ?
end
end
end
But there are some problems
are there any simple ways to contact other than using global should_end?
how to end the task from inside the expression?
While implementing something like this can be a good exercise, note that Timers are in the standard library and they are similar to the JS functions you may be used to because Julia and Node.js both use libuv internally. Whether you use Node.js setInterval or Timeout, or Julia Timer eventually uv_timer_start is called (although the creation and low-level management of timers is different in the respective runtimes).
To answer your 2nd question, you can just break in the if expression:
julia> should_end = false
false
julia> task = #task begin
while true
sleep(1)
if should_end
break
end
end
"Tasks can return a value"
end
Task (runnable) #0x00007fca1c9c3990
julia> schedule(task)
Task (runnable) #0x00007fca1c9c3990
julia> task
Task (runnable) #0x00007fca1c9c3990
julia> should_end = true;sleep(1)
julia> task
Task (done) #0x00007fca1c9c3990
julia> fetch(task)
"Tasks can return a value"
As for the 1st question, there is a lot of information in the Julia docs, e.g. the Asynchronous Programming section. As is described there, Channels can be used for communication between tasks (when suitable). Note that should_end doesn't have to be global. A task wraps a function and that function can capture variables in enclosing scopes (#task begin a = 1 end is really just Task(() -> begin a = 1 end).
Here is a short example using a Timer and a Channel to send data to a task:
function pinger(n)
ping() = parse(Float64, match(r"time=(.*) ms", read(`ping -c 1 8.8.8.8`, String))[1])
ch = Channel{Float64}(0)
pinger = Timer(0.0; interval=1.0) do timer
put!(ch, ping())
end
summer = #task begin
total = 0.0
count = 0
while count < n
total += take!(ch)
count += 1
end
total / count
end
schedule(summer)
fetch(summer)
end
julia> pinger(3)
19.5
I am working with a program which includes many function calls inside a for loop. For short, it is something like this:
function something()
....
....
timer = zeros(NSTEP);
for it = 1:NSTEP # time steps
tic = time_ns();
Threads.#threads for p in 1:2 # Star parallel of two sigma functions
Threads.lock(l);
Threads.unlock(l);
arg_in_sig[p] = func_sig[p](arg_in_sig[p]);
end
.....
.....
Threads.#threads for p in 1:2
Threads.lock(l)
Threads.unlock(l)
arg_in_vel[p] = func_vel[p](arg_in_vel[p])
end
toc=time_ns();
timer[i] = toc-tic;
end # time loop
writedlm("timer.txt",timer)
return
end
What I am trying to do, is to meassure the time that takes to perform on each loop iteration, saving the result in an output file called "timer.txt". The thing is that it doesn't work.
It saves a file with all zeros on it (except two or three values, which is more confusing).
I made a toy example like:
using DelimitedFiles;
function test()
a=zeros(1000)
for i=1:1000
tic = time_ns();
C = rand(20,20)*rand(20,20);
toc = time_ns();
a[i] = toc-tic;
end
writedlm("aaa.txt",a);
return a;
end
and these actually works (it saves fine!). Is there something to do with the fact that I am implementing Threads.#threads?. What can be happening between writedlm() and time_ns() in my program?
Any help would be much apreciated!
You are iterating over it but try to save by:
timer[i] = toc-tic;
while it should be
timer[it] = toc-tic;
Perhaps you have some i in global scope and hence the code still works.
Additionally locking the thread and immediately unlocking does not seem to make much sense. Moreover, when you iterate over p which happens to be also index of the Vector cell where you save the results there is no need to use the locking mechanism at all (unless you are calling some functions that depend on a global state).
I'm coming to Elixir from primarily a Javascript background. in JS, it's possible to write a higher order function "once" which returns a function that will invoke the passed in function only once, and returns the previous result on subsequent calls- the trick is manipulating variables that were captured via closure:
var once = (func) => {
var wasCalled = false, prevResult;
return (...args) => {
if (wasCalled) return prevResult;
wasCalled = true;
return prevResult = func(...args);
}
}
It seems to me that it's not possible to create this function in Elixir, due to its different variable rebinding behavior. Is there some other clever way to do it via pattern matching or recursion, or is it just not possible? Without macros that is, I'd imagine those might enable it. Thanks
Using the current process dictionary:
defmodule A do
def once(f) do
key = make_ref()
fn ->
case Process.get(key) do
{^key, val} -> val
nil ->
val = f.()
Process.put(key, {key, val})
val
end
end
end
end
Or if the function will be passed across processes, an ets table can be used:
# ... during application initialization
:ets.new(:cache, [:set, :public, :named_table])
defmodule A do
def once(f) do
key = make_ref()
fn ->
case :ets.lookup(:cache, key) do
[{^key, val}] -> val
[] ->
val = f.()
:ets.insert(:cache, {key, val})
val
end
end
end
end
Application.put_env / Application.get_env can also be used to hold global state, though usually is used for configuration settings.
It's not considered idiomatic in most cases, but you can do this with Agent:
defmodule A do
def once(fun) do
{:ok, agent} = Agent.start_link(fn -> nil end)
fn args ->
case Agent.get(agent, & &1) do
nil ->
result = apply(fun, args)
:ok = Agent.update(agent, fn _ -> {:ok, result} end)
result
{:ok, result} ->
result
end
end
end
end
Now if you run this:
once = A.once(fn sleep ->
:timer.sleep(sleep)
1 + 1
end)
IO.inspect once.([1000])
IO.inspect once.([1000])
IO.inspect once.([1000])
IO.inspect once.([1000])
You'll see that the first line is printed after 1 second, but the next 3 are printed instantly, because the result is fetched from the agent.
While both already given answers are perfectly valid, the most precise translation from your javascript is shown below:
defmodule M do
use GenServer
def start_link(_opts \\ []) do
GenServer.start_link(__MODULE__, nil, name: __MODULE__)
end
def init(_args) do
Process.sleep(1_000)
{:ok, 42}
end
def value() do
start_link()
GenServer.call(__MODULE__, :value)
end
def handle_call(:value, _from, state) do
{:reply, state, state}
end
end
(1..5) |> Enum.each(&IO.inspect(M.value(), label: to_string(&1)))
Use the same metric as in #Dogbert’s answer: the first value is printed with a delay, all subsequent are printed immediately.
This is an exact analog of your memoized function using GenServer stage. GenServer.start_link/3 returns one of the following:
{:ok, #PID<0.80.0>}
{:error, {:already_started, #PID<0.80.0>}}
That said, it is not restarted if it’s already started. I do not bother to check the returned value since we are all set in any case: if it’s the initial start, we call the heavy function, if we were already started, the vaklue is already at fingers in the state.
Here's some Julia code:
addprocs()
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
this.addValue = function(v)
push!(this.values,v)
end
return this
end
end
end
#everywhere t = Test()
#everywhere t.addValue(myid())
#spawnat 2 whos()
r = #spawnat 2 t
I believe this code defines a Test type on all processes and then creates an instance of that type stored in a variable, called t, on each process. This variable is local to that process. I then use one of the Test methods run in parallel on each process to mutate the local instance of Test. In the line
#spawnat 2 whos()
we can see that the local t has indeed been updated. However, I get a HUGE error when trying to fetch any of the remote variables t into a RemoteRef. The error message is quite large, but it leads me to believe that the Julia serialization process cannot handle user-defined types. Can anybody lend some insight? Thank You!
Upadate:
A simpler example yielding the same error:
addprocs()
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
this.addValue = function(v)
push!(this.values,v)
end
return this
end
end
end
t = Test()
R = RemoteRef(2)
put!(R,t)
I don't know why trying to fetch that composite object with a function field inside, causes the Error..., anyway I encourage you to use Julia way for creating the same functionality:
First of all you don't need to include addValue function inside the Test type, try to separate data storage from functionality:
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
return this
end
end
end
#everywhere function addValue(v,t::test)
push!(t.values,v)
end
then initialize t everywhere:
#everywhere t = Test()
and mutate t everywhere:
#everywhere addValue(myid(),t)
Secondly run getfield() command on desired process to get local variable:
r = #spawnat 3 getfield(Main,:t)
check the result:
assert(fetch(r).values==[3])
for more details about how to pass variable between processes check this question.
Is there a way to get the file/name/line info for the caller of a Julia function?
I found this way to get some stacktrace info, and if the caller is another function (but not the main context) I get the file:line info:
module pd
global g_bTraceOn = true
export callerOfTraceIt
function callerOfTraceIt()
traceit( "hi" ) ;
end
function traceit( msg )
global g_bTraceOn
if ( g_bTraceOn )
bt = backtrace() ;
s = sprint(io->Base.show_backtrace(io, bt))
println( "debug: $s: $msg" )
end
end
end
using pd
callerOfTraceIt( )
This shows:
$ julia bt.jl
debug:
in traceit at C:\cygwin64\home\Peeter\julia\HarmonicBalance\bt.jl:15
in callerOfTraceIt at C:\cygwin64\home\Peeter\julia\HarmonicBalance\bt.jl:8
in include at boot.jl:245
in include_from_node1 at loading.jl:128
in process_options at client.jl:285
in _start at client.jl:354: hi
I'd really like just the second frame (caller of traceit()), and would also like the function name if it's available.
If you do #show bt inside traceit, you'll discover it's just a list of pointers, each corresponding to a single stack frame. Those stack frames that come from julia code (rather than C) are displayed by show_backtrace.
You can call Profile.lookup(uint(bt[1])) to extract file/function/line information from each element:
julia> Profile.lookup(uint(bt[1]))
LineInfo("rec_backtrace","/home/tim/src/julia-old/usr/bin/../lib/libjulia.so",-1,true,140293228378757)
julia> names(Profile.LineInfo)
5-element Array{Symbol,1}:
:func
:file
:line
:fromC
:ip
You likely want to ignore all elements with fromC == true.