I want to call an async method from other libraries in tornado, like so:
class Database:
async def find_info(user_id):
pass
class TestClass(tornado.web.RequestHandler):
def get(self, id):
db = Database()
user = yield db.find_info(user_id=id)
return self.write(user.username)
But it goes to something like sleeping mode and I'll never get any result.
Which other libraries? Most async functions are written for a particular event loop (Tornado, asyncio, Twisted, etc). Different event loops don't cooperate unless you ask them to. You probably want to enable Tornado/asyncio interoperability with tornado.platform.asyncio.AsyncIOMainLoop
Related
I use Django 3.0.6 and Jupyter notebook running with shell_plus --notebook.
I try run queryset:
User.objects.all()
But return this error SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
I try this command
from asgiref.sync import sync_to_async
users = sync_to_async(User.objects.all())
for user in users:
print(user)
TypeError: 'SyncToAsync' object is not iterable
The solution of Django documentation
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true" in settings.py is the unique solution?
The error occurs because Jupyter notebooks has a running event loop.
From documentation
If you try to run any of these parts from a thread where there is a running event loop, you will get a SynchronousOnlyOperation error. Note that you don’t have to be inside an async function directly to have this error occur. If you have called a synchronous function directly from an asynchronous function without going through something like sync_to_async() or a threadpool, then it can also occur, as your code is still running in an asynchronous context.
If you encounter this error, you should fix your code to not call the offending code from an async context; instead, write your code that talks to async-unsafe in its own, synchronous function, and call that using asgiref.sync.sync_to_async(), or any other preferred way of running synchronous code in its own thread.
If you are absolutely in dire need to run this code from an asynchronous context - for example, it is being forced on you by an external environment, and you are sure there is no chance of it being run concurrently (e.g. you are in a Jupyter notebook), then you can disable the warning with the DJANGO_ALLOW_ASYNC_UNSAFE environment variable.
As the question is specific to jupyter notebook environment following is a valid solution.
import os
import django
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'rest.settings')
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"
django.setup()
from users.models import User
User.objects.all()
One need to however by wary and not use it in production environment as per documentation warnings.
sync_to_async takes a callable, not the result. Instead, you want this:
from asgiref.sync import sync_to_async
users = sync_to_async(User.objects.all)()
for user in users:
print(user)
You can also put the call(s) you want to wrap in a decorated function:
from asgiref.sync import sync_to_async
#sync_to_async
def get_all_users():
return User.objects.all()
for user in await get_all_users():
print(user)
Note that this must be used from an async context, so a full example would look like:
from asgiref.sync import sync_to_async
#sync_to_async
def get_all_users():
return User.objects.all()
async def foo(request):
for user in await get_all_users():
print(user)
Full documentation
You cant pass around a Queryset between sync and async functions since it is lazy.
You need to evaluate it inside a async context.
Like this:
from django.contrib.auth.models import User
from asgiref.sync import sync_to_async
import asyncio
#sync_to_async
def get_users():
return list(
User.objects.all()
)
async def user_loop():
results = await get_users()
for r in results:
print(r.username)
loop = asyncio.get_event_loop()
loop.run_until_complete(user_loop())
From documentation
If you're trying to run any none-async parts of Django like ORM, you will get a SynchronousOnlyOperation error. Note that you don’t have to be inside an async function directly to have this error occur. If you have called a sync function directly from an async function, without using sync_to_async() or similar, then it can also occur. This is because your code is still running in a thread with an active event loop, even though it may not be declared as an async code.
You should use:
from asgiref.sync import sync_to_async
def _get_blog(pk):
return Blog.objects.get(pk=pk)
get_blog = await sync_to_async(_get_blog, thread_sensitive=True)(10)
....
Or
from asgiref.sync import sync_to_async
get_blog = await sync_to_async(Blog.objects.get, thread_sensitive=True)(10)
You can use database_sync_to_async of Django channel which is an improved sync_to_async function of asgiref.sync
for example:
from channels.db import database_sync_to_async
get_blog = await database_sync_to_async(Blog.objects.get, thread_sensitive=True)(10)
I just used this way to get all the user as normal sync method.
What is i done is
First i create a method the get all the query and then i iterate by for loop.
The for loop is only for put a time loop like time.sleep(seconds). The loop itself choose a time. If the queryset completely got in the users variable. The timer is started (i.e) the for loop.
from asgiref.sync import sync_to_async
#sync_to_async
def get_all_user():
users = None
users = User.objects.all()
for user in users:
'''
timming loop. itself set a time by the time taken for iterate over the all User.objects.all()
'''
pass
return users
You can find the answer from here: https://docs.djangoproject.com/en/3.2/topics/async/#async-safety
If you try to run any of these parts from a thread where there is a running event loop, you will get a SynchronousOnlyOperation error. Note that you don’t have to be inside an async function directly to have this error occur. If you have called a sync function directly from an async function, without using sync_to_async() or similar, then it can also occur. This is because your code is still running in a thread with an active event loop, even though it may not be declared as async code.
override method on_finish().
Decorate this method with .gen.coroutine to make it asynchronous, it worked.
#gen.coroutine
def on_finish(self):
print("==========on_finish==============")
but use async def to make it asynchronous, it failed.
async def on_finish(self):
print("==========on_finish==============")
# RuntimeWarning: coroutine 'BaseHandler.on_finish' was never awaited
self.on_finish()
Why can't I use async def to make it asynchronous？
Coroutines have a different interface than regular functions. When overriding a method, you should make it a coroutine (or not) according to its documentation. All overridable methods in Tornado that may be coroutines say so in their documentation, and on_finish is not one of them.
What happens if you use a coroutine when the superclass doesn't expect it? As you see, this depends on the kind of coroutine. #gen.coroutine is "self-starting", so the coroutine will start, but nothing will wait for it to finish, and if it raises an exception, there is nowhere for that exception to be caught (so it will just be logged by the IOLoop). async def coroutines do not start automatically, something must await them, so if you use one where the caller is not expecting a coroutine, nothing will happen.
If you need to use a coroutine from a method where this is not allowed, you need to decide who's going to wait on the coroutine and handle its exceptions. If you don't need to catch exceptions and can just leave them for the logs, you can use IOLoop.add_callback to explicitly give this responsibility to the IOLoop:
def on_finish(self):
IOLoop.current().add_callback(self.on_finish_async)
async def on_finish_async(self):
...
I'm trying to get my head around Tornado. I'm writing a chat application backed by mongodb and I'm using motor for non-blocking access to it.
What I'm trying to achieve is:
Create a decorator that uses motor to asynchronously pull the user's record from mongo
Validate their credentials (username & token)
Create another decorator that checks that the user_id retrieved in 1. above is permitted access to the chat room. This requires another asynchronous call to mongo with motor to retrieve the 'ChatRoom' record.
Subscribe to the chat room if all is OK
I have decorator 1. working (basically taken from http://tornadogists.org/5251927/):
def authenticated_async(f):
#functools.wraps(f)
#gen.engine
def wrapper(self, *args, **kwargs):
self.current_user = yield gen.Task(self.get_current_user_async)
if self.current_user:
logging.info("User '%s' (%s) successfully authenticated" %
(self.current_user['username'],
self.current_user['_id']))
f(self, *args, **kwargs)
else:
raise tornado.web.HTTPError(401, "User not authenticated, "
"aborting")
return wrapper
The trouble is that for the second decorator, I need to access self.current_user. Because this is set in an asynchronous callback, it's not available when I get into my validation decorator (i.e the validation decorator is called before the auth decorator completes).
Is it just not possible for me to use decorators in this way with asynchronous functions? Do I just need to call the validation method inside the above method after making sure that self.current_user is True so it's more like a callback?
I'd ideally like to have my methods in my Handler wrapped with both of these decorators so I can reuse them elsewhere, i.e.:
class ChatSocketHandler(tornado.websocket.WebSocketHandler):
#gen.coroutine
#validate_invitation_access_async
#authenticated_async
def open(self, invitation_id):
# do stuff here...
Update
In fact, there is no dependency. user_id is provided as a parameter, and that could be used to run both decorators in parallel - one to confirm authentication, the other to see whether the user with that ID is allowed access to the room. The open() method would then only proceed if self.auth_check == True and self.room_check == True.
Could open() ever be called before the async decorators complete though?
You need to switch the order of the decorators so your validate_invitation_access_async wrapper has access to current_user:
#authenticated_async
#validate_invitation_access_async
def open(self, invitation_id):
# do stuff here...
Then the wrapper inside validate_invitation_access_async is the "f" in authenticated_async: it's called after self.current_user is set. Note that you don't need an additional gen.engine decorator, all the wrappers are already engines. Your wrapper could be like:
def validate_invitation_access_async(f):
#gen.engine
def wrapper(self, *args, **kwargs):
# ... use Motor to see if user is authorized ...
# if not, log and redirect. otherwise:
f(self, *args, **kwargs)
You should also update your code to use Tornado 3's gen.coroutine instead of gen.engine: it's much cleaner. But I leave that as an exercise.
I am attempting to implement get_current_user in the RequestHandler for Tornado, but I need the call to block while waiting on the asynchronous call to my database. Decorating the call with #tornado.web.asynchronous will not work because either way the get_current_user method returns before the async query completes and the query callback is executed.
For example:
class MyHandler(BaseHandler):
#tornado.web.asynchronous
#tornado.web.authenticated
def get(self):
self.write('example')
self.finish()
class BaseHandler(tornado.web.RequestHandler):
def get_current_user(self):
def query_cb(self, doc):
return doc or None
database.get(username='test', password='t3st', callback=query_cb)
#tornado.web.authenticated calls get_current_user, but always receives "None" because the BaseHandler does not have time to respond. Is there a way, using tornado, to temporarily block for a call such as the one above?
Do a blocking database operation instead of the non blocking described above (There is a blocking mysql lib shipped with tornado).
From the Tornado wiki page about threads and concurrency:
"Do it synchronously and block the IOLoop. This is most appropriate for things like memcache and database queries that are under your control and should always be fast. If it's not fast, make it fast by adding the appropriate indexes to the database, etc."
https://github.com/facebook/tornado/wiki/Threading-and-concurrency
How about having get_current_user return a Future that you signal when the asynchronous response from your database is returned?
class BaseHandler(tornado.web.RequestHandler):
def get_current_user(self):
future = Future()
def query_cb(user):
future.set_result(user or None)
database.get(username='test', password='t3st', callback=query_cb)
return future
class MainHandler(BaseHandler):
#gen.coroutine
def get(self):
user = yield self.get_current_user()
self.write('user: ' + user)
# ... actual request processing
I thought Tornado allowed you to make either blocking or non-blocking requests.
Here is Tornado being used for both: https://bitbucket.org/nephics/tornado-couchdb/src/147579581b47/couch.py
Disclaimer: I know very little of Python and Tornado.
I have a function that loads a user object from a web service asynchronously.
I wrap this function call in another function and make it synchronous.
For example:
private function getUser():User{
var newUser:User;
var f:UserFactory = new UserFactory();
f.GetCurrent(function(u:User):void{
newUser = u;
});
return newUser;
}
UserFactory.GetCurrent looks like this:
public function GetCurrent(callback:Function):void{ }
But my understanding is there is no guarantee that when this function gets called, newUser will actually be the new user??
How do you accomplish this type of return function in Flex?
This way madness lies.
Seriously, you're better off not trying to force an asynchronous call into some kind of synchronous architecture. Learn how the event handling system works in your favour and add a handler for the result event. In fact, here's the advice straight from the flexcoders FAQ :
Q: How do I make synchronous data calls?
A: You CANNOT do synchronous calls. You MUST use the result event. No,
you can't use a loop, or setInterval or even callLater. This paradigm is
quite aggravating at first. Take a deep breath, surrender to the
inevitable, resistance is futile.
There is a generic way to handle the asynchronous nature of data service
calls called ACT (Asynchronous Call Token). Search for this in
Developing Flex Apps doc for a full description.
See my answer here:
DDD and Asynchronous Repositories
Flex and Flash Remoting is inherently asynchronous so fighting against that paradigm is going to give you a ton of trouble. Our service delegates return AsyncToken from every method and we've never had a problem with it.
If you want to ensure that the application doesn't render a new view or perform some other logic until the result/fault comes back, you could do the following:
Attach an event listener for a custom event that will invoke your "post result/fault code"
Make the async call
Handle the result/fault
Dispatch the custom event to trigger your listener from #1
Bear in mind this going to lead to a lot of annoying boilterplate code every time you make an async call. I would consider very carefully whether you really need a synchronous execution path.
You can't convert async call into sync one without something like "sleep()" function and as far as I know it is missing in AS3. And yes, it is not guaranteed that newUser would contain user name before return statement.
The AS3 port of the PureMVC framework has mechanisms for implementing synchronous operations in a Model-View-Controller context. It doesn't try to synchronize asynchronous calls, but it lets you add a synchronous application pattern for controlling them.
Here's an example implementation: PureMVC AS3 Sequential Demo.
In this example, five subcommands are run sequentially, together composing a whole command. In your example, you would implement getUser() as a command, which would call commandComplete() in the getURL() (or whatever) callback. This means the next command would be certain that the getUser() operation is finished.