I have been headbutting on a wall for a few days around this code:
using Distributed
using SharedArrays
# Dimension size
M=10;
N=100;
z_ijw = zeros(Float64,M,N,M)
z_ijw_tmp = SharedArray{Float64}(M*M*N)
i2s = CartesianIndices(z_ijw)
#distributed for iall=1:(M*M*N)
# get index
i=i2s[iall][1]
j=i2s[iall][2]
w=i2s[iall][3]
# Assign function value
z_ijw_tmp[iall]=sqrt(i+j+w) # Any random function would do
end
# Print the last element of the array
println(z_ijw_tmp[end])
println(z_ijw_tmp[end])
println(z_ijw_tmp[end])
The first printed out number is always 0, the second number is either 0 or 10.95... (sqrt of 120, which is correct). The 3rd is either 0 or 10.95 (if the 2nd is 0)
So it appears that the print code (#mainthread?) is allowed to run before all the workers finish. Is there anyway for the print code to run properly the first time (without a wait command)
Without multiple println, I thought it was a problem with scope and spend a few days reading about it #.#
#distributed with a reducer function, i.e. #distributed (+), will be synced, whereas #distributed without a reducer function will be started asynchronously.
Putting a #sync in front of your #distributed should make the code behave the way you want it to.
This is also noted in the documentation here:
Note that without a reducer function, #distributed executes asynchronously, i.e. it spawns independent tasks on all available workers and returns immediately without waiting for completion. To wait for completion, prefix the call with #sync
Related
My code is, simplified, something like:
import concurrent.futures
from IPython.display import display, display_markdown
def f(parameter):
display_markdown("## %s" % (column), raw=True)
# Do some processing
display(parameter)
# Do some more processing
display(parameter)
# Do even more processing
display(parameter)
with concurrent.futures.ThreadPoolExecutor() as executor:
for result in executor.map(f, range(5)):
pass # Intentionally ignore results
The problem with this is that, because the function f gets intentionally executed multiple times in parallel and the processing takes an individual amount of time, that the display_markdown and display calls are executed interleaved.
How can I ensure that the subsections/ output of each invocation of f are outputted together/ without interleaving with the others?
And, because the processing is taking some time, to see the intermediate results/ outputs while they are executed?
Logically somehow Jupyter has to maintain some cursor/ pointer for each invocation of f and insert its output at the memorized point, while further output already happened after it, instead of just outputting it at the end.
I apologize beforehand for the length of this question. I have tried to make it as succinct as possible, but it's just a rather complicated beast.
In chapter 24 of Ierusalimschy's Programming in Lua (4th ed.), the author presents a toy ("ugly") implementation of any asynchronous I/O library, like this one1:
-- filename: async.lua
-- Based (with several modifications) on Listing 24.3 (p. 246) of *Programming
-- in Lua*, 4th edition.
local async = {}
local queue = {}
local function enqueue (command) table.insert(queue, command) end
function async.readline (stream, callback)
enqueue(function () callback(stream:read()) end)
end
function async.writeline (stream, line, callback)
enqueue(function () callback(stream:write(line)) end)
end
function async.stop () enqueue("stop") end
function async.runloop ()
while true do
local next_command = table.remove(queue, 1)
if next_command == "stop" then break end
next_command()
end
end
return async
The author uses this toy library to illustrate some applications of coroutines, such as the scheme shown below for running "synchronous code on top of the asynchronous library"2:
-- Based (with several modifications) on Listing 24.5 (p. 248) of *Programming
-- in Lua*, 4th edition.
local async = require "async"
function run (synchronous_code)
local co = coroutine.create(function ()
synchronous_code()
async.stop()
end)
local wrapper = function ()
local status, result = assert(coroutine.resume(co))
return result
end
wrapper()
async.runloop()
end
function getline (stream)
local co = coroutine.running()
local callback = function (line) assert(coroutine.resume(co, line)) end
async.readline(stream, callback)
local line = coroutine.yield()
return line
end
function putline (stream, line)
local co = coroutine.running()
local callback = function () assert(coroutine.resume(co)) end
async.writeline(stream, line, callback)
coroutine.yield()
end
The author uses this technique to implement a function that prints to stdout in reverse order the lines it read from stdin:
function synchronous_code ()
local lines = {}
local input = io.input()
local output = io.output()
while true do
local line = getline(input)
if not line then break end
table.insert(lines, line)
end
for i = #lines, 1, -1 do putline(output, lines[i] .. "\n") end
end
run(synchronous_code)
The general idea is that the run function creates a coroutine that "registers" itself (through the callbacks created by getline and putline) into the asynchronous library's main loop. Whenever the asynchronous library's main loop executes one of these callbacks, it resumes the coroutine, which can do a bit more of its work, including registering the next callback with the main loop.
The run function gets the ball rolling by invoking the wrapper function, which, in turn, "resumes" (actually starts) the coroutine. The coroutine then runs until it encounters the first yield statement, which, in this example, happens within getline, right after getline has registered a callback into the async library's queue. Then the wrapper function regains control and returns. Finally, run invokes async.runloop. As async.runloop starts processing its queue, it resumes the coroutine, and off we go. The "synchronous code" (running within the coroutine) continues until the next getline or putline yields (after registering a callback), and async's main loop takes over again.
So far so good. But then, in Exercise 24.4 (p. 249), the author asks:
Exercise 24.4: Write a line iterator for the coroutine-based library (Listing 24.5), so that you can read the file with a for loop.
("Listing 24.5" refers to the code in the second code fragment above, where run, getline, and putline are defined.)
I am completely stumped with this one. In the example above, the coroutine "delivers" the lines it reads by writing them to stdout, which it can do all by itself. In contrast, the iterator requested by Exercise 24.4 would have to deliver its lines to a different coroutine, the one that is doing the iteration.
The only way that I can imagine this could happen is if the two coroutines could reciprocally resume each other. Is that even possible? I have not been able to construct a simple example of this, and would appreciate to see code that does it3.
Also, it seems to me that for this to work at all, one would need to implement an object with a write method (so that it can be passed to putline) that is ultimately responsible for delivering lines (somehow) to the iterator's coroutine.
1I have changed some superficial details, such as the names of variables, indentation, etc. The overall structure and function are unchanged.
2Again, I have changed some inessential details, to make the code easier for me to follow.
3 It is worth noting that the remaining two exercises for this chapter (24.5 and 24.6) are both about implementing systems involving multiple concurrent coroutines. Therefore, it is not farfetched to imagine that Exercise 24.4 is also about having two coroutines talking to each other.
I believe you're completely overthinking this exercise. The way I understand it, you're only meant to write a synchronous-style for iterator that runs within the synchronous code given to the run function. Taking the third code block as a base:
function for_file(file)
return function(file)
return getline(file)
end, file, nil
end
function synchronous_code ()
local lines = {}
local input = io.input()
local output = io.output()
for line in for_line(input) do
table.insert(lines, line)
end
for i = #lines, 1, -1 do putline(output, lines[i] .. "\n") end
end
run(synchronous_code)
As you can see, you don't really need to be aware of the coroutines at all for this to work, which is kind of the point of the library.
This particular example function is in Lua, but I think the main concepts are true for any language with lexical scoping, first-class functions, and iterators.
Code Description (TL;DR -- see code):
The code below defines an iterator constructor which merely defines a local value and returns an iterator function.
This iterator function, when run, uses the local value from the constructor and increases that value by 1. It then runs itself recursively until the value reaches 5, then adds 1 to the value and returns the number 5. If it's run again, it will run itself recursively until the value reaches 20 or higher, then it returns nil, which is the signal for the loop to stop.
I then run an outer loop using the iterator provided by the constructor, which will run the body of the loop when the iterator returns a value (5). In the body of the outer loop I put an inner loop that also uses an iterator provided by the constructor.
The program should run the body of the loop once (when the value in the outer loop iterator hits 5) and then run the inner loop completely (letting the inner loop value hit 20), then return back to the outer loop and run it until it finishes.
function iterConstr()
local indexes = {0}
function iter()
print(indexes[1])
indexes[1] = indexes[1] + 1
if indexes[1] == 5 then
indexes[1] = indexes[1] + 1
return 5
elseif indexes[1]>=21 then
print("finished!")
return nil
else
print("returning next one")
return iter()
end
end
return iter
end
for val in iterConstr() do
for newVal in iterConstr() do
end
print("big switch")
end
The behavior I did not expect is that the value from the inner loop and outer loop seem to be linked together. When focus is returned to the outer loop after running through the inner loop, it runs with the expected next value for the outer loop (6), but then instead of iterating and incrementing up to 20, it instead jumps immediately to 21, which is where the inner loop ended!
Can anyone help explain why this bizarre behavior occurs?
I believe your problem lies on line 3 - you declare iter as a global function instead of a local one, which makes it accessible from any Lua chunk. Changing that line to local function iter() corrects this behaviour. I think therefore that this is more of an issue with the developer applying lexical scoping than with lexical scoping itself :)
When generating a not explicitly generated version of a function, #ngenerate runs
eval(quote
local _F_
$localfunc # Definition of _F_ for the requested value of N
_F_
end)
Since eval runs in the scope of the current module, not the function, I wonder what is the effect of local in this context. As far as I know, the languange documentation only mentions the use of local inside function definitions.
To give some background why this question arose: I frequently need to code loops of the form
function foo(n::Int)
s::Int = 0
for i in 1:1000000000
for j in 1:n
s += 1
end
end
return s
end
where n <= 10 (of course, in my actual code the loops are such that they cannot just be reduced to O(1)). Because this code is very simple for the compiler but demanding at runtime, it turns out to be beneficial to simply recompile the loops with the required value of n each time foo is called.
function clever_foo(n::Int)
eval(quote
function clever_foo_impl()
s::Int = 0
for i in 1:1000000000
s += $(Expr(:call,:+,[1 for j in 1:n]...))
end
return s
end
end)
return clever_foo_impl()
end
However, I am not sure whether I am doing this the right way.
It's to prevent _F_ from being visible in the global method cache.
If you'll call clever_foo with the same n repeatedly, you can do even better by saving the compiled function in a Dict. That way you don't have to recompile it each time.
I have a piece of code in Julia in which a solver iterates many, many times as it seeks a solution to a very complex problem. At present, I have to provide a number of iterations for the code to do, set low enough that I don't have to wait hours for the code to halt in order to save the current state, but high enough that I don't have to keep activating the code every 5 minutes.
Is there a way, with the current state of Julia (0.2), to detect a keystroke instructing the code to either end without saving (in case of problems) or end with saving? I require a method such that the code will continue unimpeded unless such a keystroke event has happened, and that will interrupt on any iteration.
Essentially, I'm looking for a command that will read in a keystroke if a keystroke has occurred (while the terminal that Julia is running in has focus), and run certain code if the keystroke was a specific key. Is this possible?
Note: I'm running julia via xfce4-terminal on Xubuntu, in case that affects the required command.
You can you an asynchronous task to read from STDIN, blocking until something is available to read. In your main computation task, when you are ready to check for input, you can call yield() to lend a few cycles to the read task, and check a global to see if anything was read. For example:
input = ""
#async while true
global input = readavailable(STDIN)
end
for i = 1:10^6 # some long-running computation
if isempty(input)
yield()
else
println("GOT INPUT: ", input)
global input = ""
end
# do some other work here
end
Note that, since this is cooperative multithreading, there are no race conditions.
You may be able to achieve this by sending an interrupt (Ctrl+C). This should work from the REPL without any changes to your code – if you want to implement saving you'll have to handle the resulting InterruptException and prompt the user.
I had some trouble with the answer from steven-g-johnson, and ended up using a Channel to communicate between tasks:
function kbtest()
# allow 'q' pressed on the keyboard to break the loop
quitChannel = Channel(10)
#async while true
kb_input = readline(stdin)
if contains(lowercase(kb_input), "q")
put!(quitChannel, 1)
break
end
end
start_time = time()
while (time() - start_time) < 10
if isready(quitChannel)
break
end
println("in loop # $(time() - start_time)")
sleep(1)
end
println("out of loop # $(time() - start_time)")
end
This requires pressing and then , which works well for my needs.