I am running the simplest of examples on asyncio:
import asyncio
async def main():
print("A")
await asyncio.sleep.sleep(1)
print("B")
asyncio.run(main())
and I get a runtime error:
RuntimeError: asyncio.run() cannot be called from a running event loop
I am using Spyder (Python 3.9) on an M1 Mac (...if that matters).
the outcome expected is:
A
B
Process finished with exit code 0
But for the ".sleep.sleep" this code is fine - "event loop already running" is certainly not an issue for a standalone script with this code.
Maybe you are running it in as a notebook cell, with some asyncio state already set-up?
In a bash terminal, I pasted your code as is, and just replaced the incorrect function name:
[gwidion#fedora tmp01]$ cat >bla42.py
import asyncio
async def main():
print("A")
await asyncio.sleep.sleep(1)
print("B")
asyncio.run(main())
[gwidion#fedora tmp01]$ python bla42.py
A
Traceback (most recent call last):
[...]
File "/home/gwidion/tmp01/bla42.py", line 5, in main
await asyncio.sleep.sleep(1)
AttributeError: 'function' object has no attribute 'sleep'
[gwidion#fedora tmp01]$ python -c 'open("bla43.py", "w").write(open("bla42.py").read().replace(".sleep.sleep", ".sleep"))'
[gwidion#fedora tmp01]$ python bla43.py
A
B
[gwidion#fedora tmp01]$
Related
I have a couple of DAGs running, which execute python scripts which are basically copying data from A to B. But one of the dags throws an error - but data is still getting copied, so somehow it does not have influence on the execution of the python script.
The only "special" what is within this dag is, that the python script builds up a connecton to a postgres database using psycopg2~=2.8.5 but not sure if this is somehow the root cause.
I also checked the permissions for the user, which seem to be fine at least in the dags folder.
Is there any specific timeout value I have to adjust in the config?
[2021-05-19 12:53:42,036] {taskinstance.py:1145} ERROR - Bash command failed 255
Traceback (most recent call last):
File "/hereisthepath/venv/lib64/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/hereisthepath/venv/lib64/python3.6/site-packages/airflow/operators/bash_operator.py", line 134, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
Update: This is the passage of the operator, which fails. I copied the entire function, however the error throws at line 134 ("raise AirflowException("Bash command failed"))
def execute(self, context):
"""
Execute the bash command in a temporary directory
which will be cleaned afterwards
"""
self.log.info("Tmp dir root location: \n %s", gettempdir())
# Prepare env for child process.
env = self.env
if env is None:
env = os.environ.copy()
airflow_context_vars = context_to_airflow_vars(context, in_env_var_format=True)
self.log.debug('Exporting the following env vars:\n%s',
'\n'.join(["{}={}".format(k, v)
for k, v in airflow_context_vars.items()]))
env.update(airflow_context_vars)
self.lineage_data = self.bash_command
with TemporaryDirectory(prefix='airflowtmp') as tmp_dir:
with NamedTemporaryFile(dir=tmp_dir, prefix=self.task_id) as f:
f.write(bytes(self.bash_command, 'utf_8'))
f.flush()
fname = f.name
script_location = os.path.abspath(fname)
self.log.info(
"Temporary script location: %s",
script_location
)
def pre_exec():
# Restore default signal disposition and invoke setsid
for sig in ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ'):
if hasattr(signal, sig):
signal.signal(getattr(signal, sig), signal.SIG_DFL)
os.setsid()
self.log.info("Running command: %s", self.bash_command)
self.sub_process = Popen(
['bash', fname],
stdout=PIPE, stderr=STDOUT,
cwd=tmp_dir, env=env,
preexec_fn=pre_exec)
self.log.info("Output:")
line = ''
for line in iter(self.sub_process.stdout.readline, b''):
line = line.decode(self.output_encoding).rstrip()
self.log.info(line)
self.sub_process.wait()
self.log.info(
"Command exited with return code %s",
self.sub_process.returncode
)
if self.sub_process.returncode:
raise AirflowException("Bash command failed")
if self.xcom_push_flag:
return line
Update2: It really seems, that this behavior is related to the psycopg2: I now tested all other possible error sources and only when I test with the postgres datasource using psycopg2 package, the error occurs. Meanwhile I also upgraded to the most recent version of psycopg2 (2.8.6) but without success.
Maybe this helps for further investigation
I have created a custom python sync block for use in a gnuradio flowgraph. The block tests for invalid input and, if found, raises a ValueError exception. I would like to create a unit test to verify that the exception is raised when the block indeed receives invalid input data.
As part of the python-based qa test for this block, I created a flowgraph such that the block receives invalid data. When I run the test, the block does appear to raise the exception but then hangs.
What is the appropriate way to test for this? Here is a minimal working example:
#!/usr/bin/env python
import numpy as np
from gnuradio import gr, gr_unittest, blocks
class validate_input(gr.sync_block):
def __init__(self):
gr.sync_block.__init__(self,
name="validate_input",
in_sig=[np.float32],
out_sig=[np.float32])
self.max_input = 100
def work(self, input_items, output_items):
in0 = input_items[0]
if (np.max(in0) > self.max_input):
raise ValueError('input exceeds max.')
validated_in = output_items[0]
validated_in[:] = in0
return len(output_items[0])
class qa_validate_input (gr_unittest.TestCase):
def setUp (self):
self.tb = gr.top_block ()
def tearDown (self):
self.tb = None
def test_check_valid_data(self):
src_data = (0, 201, 92)
src = blocks.vector_source_f(src_data)
validate = validate_input()
snk = blocks.vector_sink_f()
self.tb.connect (src, validate)
self.tb.connect (validate, snk)
self.assertRaises(ValueError, self.tb.run)
if __name__ == '__main__':
gr_unittest.run(qa_validate_input, "qa_validate_input.xml")
which produces:
DEPRECATED: Using filename with gr_unittest does no longer have any effect.
handler caught exception: input exceeds max.
Traceback (most recent call last):
File "/home/xxx/devel/gnuradio3_8/lib/python3.6/dist-packages/gnuradio/gr/gateway.py", line 60, in eval
try: self._callback()
File "/home/xxx/devel/gnuradio3_8/lib/python3.6/dist-packages/gnuradio/gr/gateway.py", line 230, in __gr_block_handle
) for i in range(noutputs)],
File "qa_validate_input.py", line 21, in work
raise ValueError('input exceeds max.')
ValueError: input exceeds max.
thread[thread-per-block[1]: <block validate_input(2)>]: SWIG director method error. Error detected when calling 'feval_ll.eval'
^CF
======================================================================
FAIL: test_check_valid_data (__main__.qa_validate_input)
----------------------------------------------------------------------
Traceback (most recent call last):
File "qa_validate_input.py", line 47, in test_check_valid_data
self.assertRaises(ValueError, self.tb.run)
AssertionError: ValueError not raised by run
----------------------------------------------------------------------
Ran 1 test in 1.634s
FAILED (failures=1)
The top_block's run() function does not call the block's work() function directly but starts the internal task scheduler and its threads and waits them to finish.
One way to unit test the error handling in your block is to call the work() function directly
def test_check_valid_data(self):
src_data = [[0, 201, 92]]
output_items = [[]]
validate = validate_input()
self.assertRaises(ValueError, lambda: validate.work(src_data, output_items))
How can I use Asynchronous Widgets on jupyter lab?
I'm trying to reproduce the official Asynchronous Widgets-Example on jupyter lab, but the await never continues.
Setup / reproduction
docker run --rm -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes jupyter/datascience-notebook start-notebook.sh --NotebookApp.token=''
firefox 0.0.0.0:8888
create a new python3 notebook
create a cell and enter the code below
run cell
move slider
code for the cell
%gui asyncio
import asyncio
def wait_for_change(widget, value):
future = asyncio.Future()
def getvalue(change):
# make the new value available
future.set_result(change.new)
widget.unobserve(getvalue, value)
widget.observe(getvalue, value)
return future
from ipywidgets import IntSlider
slider = IntSlider()
async def f():
for i in range(10):
print('did work %s'%i)
#x = await asyncio.sleep(1)
x = await wait_for_change(slider, 'value')
print('async function continued with value %s'%x)
asyncio.ensure_future(f())
#task = asyncio.create_task(f())
slider
Expected result
The cell outputs
did work 0
async function continued with value 1
did work 1
async function continued with value 2
[...]
Actual output
nothing after the first did work 0
Notes
I'm specifically talking about jupyter lab and not about regular jupyter notebooks
There is no error-message or anything. The expected output just doesn't happen
The minimal asyncio-example does work in jupyter lab:
import asyncio
async def main():
print('hello')
await asyncio.sleep(1)
print('world')
await main()
when you leave out the -e JUPYTER_ENABLE_LAB=yes, then you get a regular jupyter notebook without jupyter lab and the expected result happens.
this is not a duplicate of ipywidgets widgets values not changing or Jupyter Interactive Widget not executing properly, because these questions nether include jupyter lab nor asyncio
Actually it works, but jupyter lose print output.
Try this code:
from IPython.display import display
import ipywidgets as widgets
out = widgets.Output()
import asyncio
def wait_for_change(widget, value):
future = asyncio.Future()
def getvalue(change):
# make the new value available
future.set_result(change.new)
widget.unobserve(getvalue, value)
widget.observe(getvalue, value)
return future
from ipywidgets import IntSlider
slider = IntSlider()
# Now the key: the container is displayed (while empty) in the main thread
async def f():
for i in range(10):
out.append_stdout('did work %s'%i)
x = await wait_for_change(slider, 'value')
out.append_stdout('async function continued with value %s'%x)
asyncio.ensure_future(f())
display(slider)
display(out)
You can find more details here: https://github.com/jupyter-widgets/ipywidgets/issues/2567#issuecomment-535971252
I've had luck with jupyter-ui-poll to synchronize widget activity with the Jupyter Python kernal:
https://github.com/Kirill888/jupyter-ui-poll
In particular I used it here:
https://github.com/AaronWatters/jp_doodle/blob/master/jp_doodle/auto_capture.py
Works for me.
Hope that helps!
below is my source:
class Get_Salt_Handler(tornado.web.RequestHandler):
#tornado.web.asynchronous
#tornado.gen.coroutine
def get(self):
#yield tornado.gen.Task(tornado.ioloop.IOLoop.instance().add_timeout, time.time() + 5)
yield tornado.gen.Task(self.get_salt_from_db, 123)
self.write("when i sleep 5s")
def get_salt_from_db(self, params):
print params
and I run it; the console reported that:
TypeError: get_salt_from_db() got an unexpected keyword argument 'callback'
and I don't know why?
gen.Task is used to adapt a callback-based function to the coroutine style; it cannot be used to call synchronous functions. What you probably want is a ThreadPoolExecutor (standard in Python 3.2+, available with pip install futures on Python 2):
# global
executor = concurrent.futures.ThreadPoolExecutor(NUM_THREADS)
#gen.coroutine
def get(self):
salt = yield executor.submit(self.get_salt_from_db)
I am trying to use Tornado's sync-style 'gen' tool to run a simple echo function, in a non-blocking style:
import tornado.web
import tornado.gen
import logging
def echo(message):
return message
#tornado.gen.engine
def runme():
response = yield tornado.gen.Task(echo, 'this is a message')
logging.warn(response)
runme()
As far as I can tell this code isn't significantly different to the demo code in the docs, minus the unnecessary request handler stuff - I'm not handling any HTTP requests, AFAICT that's orthagonal to running something asynchronously. Yet this always fails with:
Traceback (most recent call last):
File "./server.py", line 46, in <module>
runme()
TypeError: wrapper() takes at least 1 argument (0 given)
Exactly where am I missing the argument? How can I make Tornado run this function asynchronously?
Task doesn't actually make a callback for the function being run, and start the callback when the function returns, as I originally thought.
I need to create a callback in the task being run myself, and invoke it, i.e.:
import tornado.web
import tornado.gen
import logging
def echo(message, callback=None):
callback(message)
#tornado.gen.engine
def runme():
response = yield tornado.gen.Task(echo, 'this is a message')
logging.warn(response)
runme()