Python crashes when I put process.join in script when using multiprocessing - python-3.4

Question

I have been researching multiprocessing and came upon an example of it on a website. However, when I try to run that example on my MacBook retina, nothing happens. The following was the example:
import random
import multiprocessing
def list_append(count, id, out_list):
"""
Creates an empty list and then appends a
random number to the list 'count' number
of times. A CPU-heavy operation!
"""
for i in range(count):
out_list.append(random.random())
if __name__ == "__main__":
size = 10000000 # Number of random numbers to add
procs = 2 # Number of processes to create
# Create a list of jobs and then iterate through
# the number of processes appending each process to
# the job list
jobs = []
for i in range(0, procs):
out_list = list()
process = multiprocessing.Process(target=list_append,
args=(size, i, out_list))
jobs.append(process)
# Start the processes (i.e. calculate the random number lists)
for j in jobs:
j.start()
# Ensure all of the processes have finished
for j in jobs:
j.join()
print ("List processing complete.")
As it turns out after I put a print statement in the 'list_append' function, nothing printed, so the problem is actually not the j.join() but rather the j.start() bit.

Answer 1

When you create a process with multiprocessing.Process, you prepare a sub-function to be run in a different process asynchronously. The computation starts when you call the start method. The join method waits for the computation to be done. So if you just start the process and do not wait for the to complete (or join) nothing will happen as the process will be killed when your program exit.
Here, one issue is that you are not using an object sharable in multiprocessing. When you use a common list(), each process will use a different list in memory. The local process will be cleared when the processes exit and the main list will be empty. If you want to be able to exchange processes between data you should use a multiprocessing.Queue:
import random
import multiprocessing
def list_append(count, id, out_queue):
"""
Creates an empty list and then appends a
random number to the list 'count' number
of times. A CPU-heavy operation!
"""
for i in range(count):
out_queue.put((id, random.random()))
if __name__ == "__main__":
size = 10000 # Number of random numbers to add
procs = 2 # Number of processes to create
# Create a list of jobs and then iterate through
# the number of processes appending each process to
# the job list
jobs = []
q = multiprocessing.Queue()
for i in range(0, procs):
process = multiprocessing.Process(target=list_append,
args=(size, i, q))
process.start()
jobs.append(process)
result = []
for k in range(procs*size):
result += [q.get()]
# Wait for all the processes to finish
for j in jobs:
j.join()
print("List processing complete. {}".format(result))
Note that this code can hang quite easily if you do not compute correctly the number of results sent back in out_queue.
If you try to retrieve too many results, q.get will wait for an extra result that will never come. If you do not retrieve all the result from q, your processes will freeze as the out_queue will be full, and out_queue.put will not return. Your processes will thus never exit and you will not be able to join them.
If your computation are independent, I strongly advise to look at higher level tools like Pool or even more robust third party library like joblib as it will take care of these aspects for you. (see this answer for some insights on Process vs Pool/joblib)
I actually reduced the number size as the program become to slow if you try to put to many objects in a Queue. If you need to pass a lot of small object, try passing all of them in one batch:
import random
import multiprocessing
def list_append(count, id, out_queue):
a = [random.random() for i in range(count)]
out_queue.put((id, a))
if __name__ == "__main__":
size = 10000 # Number of random numbers to add
procs = 2 # Number of processes to create
jobs = []
q = multiprocessing.Queue()
for i in range(0, procs):
process = multiprocessing.Process(target=list_append,
args=(size, i, q))
process.start()
jobs.append(process)
result += [q.get() for _ in range(procs)]
for j in jobs:
j.join()
print("List processing complete.")