After first run ,Jupyter notebook with python 3.6.1, using asyncio basic example gives: RuntimeError: Event loop is closed - jupyter-notebook

In Jupyter Notebook (python 3.6.1) I went to run the basic python docs Hello_World in (18.5.3.1.1. Example: Hello World coroutine) and noticed that it was giving me a RuntimeError. After trying a long time to find the problem with the program(my understanding is that the docs may not be totally up to date), I finally noticed that it only does this after the second run and tested in a restarted Kernel. I've since then copied the same small python program in two successive cells(In 1 and 2) and found that it gives the error on the second not the first and gives the error to both there after. This repeats this after restarting the Kernel.
import asyncio
def hello_world(loop):
print('Hello World')
loop.stop()
loop = asyncio.get_event_loop()
# Schedule a call to hello_world()
loop.call_soon(hello_world, loop)
# Blocking call interrupted by loop.stop()
loop.run_forever()
loop.close()
The traceback:
RuntimeError Traceback (most recent call last)
<ipython-input-2-0930271bd896> in <module>()
6 loop = asyncio.get_event_loop()
7 # Blocking call which returns when the hello_world() coroutine
----> 8 loop.run_until_complete(hello_world())
9 loop.close()
/home/pontiac/anaconda3/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
441 Return the Future's result, or raise its exception.
442 """
--> 443 self._check_closed()
444
445 new_task = not futures.isfuture(future)
/home/pontiac/anaconda3/lib/python3.6/asyncio/base_events.py in _check_closed(self)
355 def _check_closed(self):
356 if self._closed:
--> 357 raise RuntimeError('Event loop is closed')
358
359 def _asyncgen_finalizer_hook(self, agen):
RuntimeError: Event loop is closed
I don't get this error when running a file in the interpreter with all the Debug settings set. I am running this Notebook in my recently reinstalled Anaconda set up which only has the 3.6.1 python version installed.

the issue is that loop.close() makes the loop unavailable for future use. That is, you can never use a loop again after calling close. The loop stays around as an object, but almost all methods on th eloop will raise an exception once the loop is closed. However, asyncio.get_event_loop() returns the same loop if you call it more than once. You often want this, so that multiple parts of an application get the same event loop.
However if you plan on closing a loop, you are better off calling asyncio.new_event_loop rather than asyncio.get_event_loop. That will give you a fresh event loop. If you call new_event_loop rather than get_event_loop, you're responsible for making sure that the right loop gets used in all parts of the application that run in this thread. If you want to be able to run multiple times to test you could do something like:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
After that, you'll find that asyncio.get_event_loop returns the same thing as loop. So if you do that near the top of your program, you will have a new fresh event loop each run of the code.

Related

Is there a way to make a Denodo 8 VDP scheduler job WAIT() for a certain amount of time?

I want to make a VDP scheduler job in Denodo 8 wait for a certain amount of time. The wait function in the job creation process is not working as expected so I figured I'd write it into the VQL. However when i try the suggested function from the documentation (https://community.denodo.com/docs/html/browse/8.0/en/vdp/vql/stored_procedures/predefined_stored_procedures/wait) the Denodo 8 VQL shell doesn't recognize the function.
--Not working
SELECT WAIT('10000');
Returns the following error:
Function 'wait' with arity 1 not found
--Not working
WAIT('10000');
Returns the following error:
Error parsing command 'WAIT('10000')'
Any suggestions would be much appreciated.
There are two ways of invoking WAIT:
Option #1
-- Wait for one minute
CALL WAIT(60000);
Option #2:
-- Wait for ten seconds
SELECT timeinmillis
FROM WAIT()
WHERE timeinmillis = 10000;

quit 1000 of QThread - pyqt5

I have python gui app using Qt
I'm using pyqt5
the app should creating 1000 or more Qthreads, each thread will use pycurl to open external URL
I'm using this code to open the threads:
self.__threads = []
# workerClass thread
for i in range(1000):
workerInstance = workerClass()
workerInstance.sig.connect(self.ui.log.append)
thread = QThread()
self.__threads.append((thread, workerInstance))
workerInstance.moveToThread(thread)
thread.started.connect(workerInstance.worker_func)
thread.start()
WorkerClass will use a simple pycurl to visit an external URL and emit signal to add some info on the log, here is the code:
class workerClass(QObject):
x = 1
sig = pyqtSignal(str)
def __init__(self, linksInstance, timeOut):
super().__init__()
self.linksInstance = linksInstance
self.timeOut = timeOut
#pyqtSlot()
def worker_func(self):
while self.x:
# run the pycurl code
self.sig.emit('success!') ## emit success message
Now the problem is with stopping all this, I used the below code for stop button:
def stop_func(self):
workerClass.x = 0
if self.__threads:
for thread, worker in self.__threads:
thread.quit()
thread.wait()
changing the workerClass.x to 0 will stop the while self.x in the class workerClass and quit and wait should close all threads
all this is working properly, BUT only in the low amount of threads, 10 or may be 100
but if I rung 1000 threads, it takes much time, I wait for 10 minutes but it didn't stopped (mean threads not terminated), however the pycurl time out is only 15 seconds
I also used: self.thread.exit() self.thread.quit() self.thread.terminate() but it gives no difference
if any one have any idea about this, this will be really great :)
after some tries, I removed the pycurl code and test to start 10k QThreads and stopped it, and this working perfect without any errors. so I define that the error is with the pycurl code.
as per the advice of ekhumoro in the question comments, I may try to use pycurl multi
but for now I added this line to the pycurl code:
c.setopt(c.TIMEOUT, self.timeOut)
I was using only this:
c.setopt (c.CONNECTTIMEOUT, self.timeOut)
so now with the both 2 parameters all is properly working, starting and stopping bulk qthreads

Multiprocessing with worker.run() does work in serie instead of parallel?

I'm trying to create a program which in its essence works like this:
import multiprocessing
import time
def worker(numbers):
print(numbers)
time.sleep(2)
return
if __name__ =='__main__':
multiprocessing.set_start_method("spawn")
p1 = multiprocessing.Process(target=worker, args=([0,1,2,3,4],))
p2 = multiprocessing.Process(target=worker, args=([5,6,7,8],))
p1.start()
p2.start()
p1.join()
p2.join()
while(1):
p1.run()
p2.run()
p1.join()
p2.join()
print('Done!')
The first time the processes are called via p#.start(), they are executed in parallel. The second time they are called via the p#.run() method, they are executed in series.
How can I make sure the subsequent method calls are also performed in parallel?
Edit: It is important that the processes start together. It cannot happen that process 1 gets executed twice while process 2 only gets executed once.
Edit: I should also note that this code is running on a raspberry pi v3 model B.
As far as I know, a thread can only be started once. After that when you call the run method, it's just a simple function. That's why it isn't run in parallel.

Callback from "multiprocessing" with CFFI segfaults after ~100 iterations

A PyPy callback, that works perfectly (in an infinite loop) when implemented (straightforwardly) as method of a Python object, segfaults after approximately 100 iterations when I move the Python object into a separate multiprocessing process.
In the main code I have:
import multiprocessing as mp
class Task(object):
def __init__(self, com, lib):
self.com = com # communication queue
self.lib = lib # ffi library
self.proc = mp.Process(target=self.spawn, args=(self.com,))
self.register_callback()
def spawn(self, com):
print('%s spawned.'%self.name)
# loop (keeping 'self' alive) until BREAK:
while True:
cmd = com.get()
if cmd == self.BREAK:
break
print("%s stopped."%self.name)
#ffi.calback("int(void*, Data*"): # old cffi (ABI mode)
def callback(self, data):
# <work on data>
return 1
def register_callback(self):
s = ffi.new_handle(self)
self.lib.register_callback(s, self.callback) # C-call
The idea is that multiple tasks should serve an equal number of callbacks concurrently. I have no clue what may cause the segfault, especially since it runs fine for the first ~100 iterations or so. Help much appreciated!
Solution
Handle 's' is garbage collected when returning from 'register_callback()'. Making the handle an attribute of 'self' and passing keeps it alive.
Standard CPython (cffi 1.6.0) segfaulted at the first iteration (i.e. gc was immediate) and provided me a crucial informative error message. PyPy on the other hand segfaulted after approximately 100 iterations without providing a message... Both run fine now.

How can I configure my IPython notebook so it always shows the execution time as part of the output?

Sometimes I execute a method that takes long to compute
In [1]:
long_running_invocation()
Out[1]:
Often I am interested in knowing how much time it took, so I have to write this:
In[2]:
import time
start = time.time()
long_running_invocation()
end = time.time()
print end - start
Out[2]: 1024
Is there a way to configure my IPython notebook so that it automatically prints the execution time of every call I am making like in the following example?
In [1]:
long_running_invocation()
Out[1] (1.59s):
This ipython extension does what you want: https://github.com/cpcloud/ipython-autotime
load it by putting this at the top of your notebook:
%install_ext https://raw.github.com/cpcloud/ipython-autotime/master/autotime.py
%load_ext autotime
Once loaded, every subsequent cell execution will include the time it took to execute as part of its output.
i haven't found a way to have every cell output the time it takes to execute the code, but instead of what you have, you can use cell magics: %time or %timeit
ipython cell magics
You can now just use the %%time magic at the beginning of the cell like this:
%%time
data = sc.textFile(sample_location)
doc_count = data.count()
doc_objs = data.map(lambda x: json.loads(x))
which when executed will print out an output like:
CPU times: user 372 ms, sys: 48 ms, total: 420 ms
Wall time: 37.7 s
The Simplest way to configure your ipython notebook in a way it automatically shows the execution time without running any %%time or %%timelit or time.time() in each cell, is by using ipython-autotime package.
Install the package in the begining of the notebook
pip install ipython-autotime
and then load the extension by running below
%load_ext autotime
Once you have loaded it, any cell run after this ,will give you the execution time of the cell.
And dont worry if you want to turn it off, just unload the extension by running below
%unload_ext autotime
It is pretty simple and easy to use it whenever you want.
And if you want to check out more, can refer to ipython-autime documentation or its github source

Resources