I might have misunderstood or be missing something but I have to clear this issue:
Asynchronous code means code that gets executed with multiple operations at the same time, without blocking.
Synchronous code means code that gets executed one operation at a time.
But the definition of the word synchronous is occurring at the same time, so that's the other way around isn't it? Why the confusing naming? Is the definition referring to something I am not aware of?
We use the term "Async" to allow the code stream to continue without blocking the main stream. This is OK. We want the mainstream to continue without waiting for that process to complete. But usually, "Async and Await" are used together.
My question is; When we add "await", we expect "Async" status in the code stream. In this case, I do not understand what is the benefit of using "Async"? Can someone explain this, thank you, good work.
It's not clear which technology you're referring to. I'll assume you're referring to the async and await keywords that have been added to many languages the last few years.
In that case, this statement is actually incorrect:
We use the term "Async" to allow the code stream to continue without blocking the main stream.
In C#, JavaScript, Python, F#, and Visual Basic, the async keyword does not act asynchronously. It has two effects:
Enable the await keyword within that method/lambda.
Transform that method/lambda into a state machine.
So async does not mean "run this code on a background thread" or anything like that, which is a common but incorrect assumption.
The first point above is important for backwards compatibility. Each of these languages use await as a contextual keyword, so the text "await" in a program is only a keyword if it is in a method marked with async. This allows the languages to introduce the new keywords without breaking any existing code (e.g., someone's code that used await as a variable name).
See this post of mine which collected a lot of the discussion around the keywords as they were originally designed for C#/VB. Those same design decisions carried over almost exactly to other languages as they adopted the keywords, too. One of the resources linked from that post is Eric Lippert's post on why we need an async keyword in addition to an await keyword.
I wanted to ask about this in general, but I had concerns switching my frontend automation suite from a Java framework to a JavaScript one. Mainly around an individual test running asynchronously.
Could cases potentially happen where tests take steps out of order or do a false positive of passing a test before the last expect argument is resolved?
If they can, in general, how do I resolve this issue?
You have two options to handle asynchronous nature of JS with Protractor:
1) Use async/await
or
2) User control flow (this basically means that if you are not using async/await you are using the Control Flow. It can not be combined as far as I know)
If you are not familiar with the async/await you can simply write your tests in synchronous manner e.g:
browser.get("https://myurl.com")
element(by.id("login").sendKeys("admin")
element(by.id("password").sendKeys("secretpassword")
element(by.buttonText("Login!").click()
expect(element(by.cssContainingText("header", "Welcome Admin")).toBePresent()
Protractor will execute this code in synchronous manner and the biggest plus is that it will actually wait for your Angular application to be ready (nevertheless you will have to use smart waits (Protractor expect condition) from time to time)
Before executing your tests protractor will build a queue of your steps and will execute it one by one since most of this steps are build upon promises. All none-protractor/webdriver method will be added to the call stack in common asynchronous manner. E.g. if you will add console.log("foo") at the end of the previous code snippet it will print the console.log before executing the steps.
This is pretty poor explanation but hope it will help
Problem:
I am trying to update some old code (which I didn't write), which uses an outdated version of Tornado and gen.Task, to use the current version of Tornado and asyncio. This is mostly straightforward except for this one expression which (1) I do not fully understand, and which (2) I cannot figure out how to replace with an equivalent asyncio expression.
The single line of code I want to replace is of the form:
response = yield gen.Task(fn, request)
where the signature of the function fn is fn(request, callback), and then later in the code (which is the method definition of a gen.coroutine) we run callback(response). And I think fn may itself be asynchronous, although I'm not sure, and don't understand what the implications of that would be if it is true.
EDIT: Following the advice of another answer, I was able to rewrite this as
fn(request=request, callback=(yield gen.Callback("key")))
response = yield gen.Wait("key")
Of course though the release notes for Tornado 6 say that both gen.Wait and gen.Callback have been removed. Earlier versions of the documentation say that the gen.Wait is deprecated and should be replaced by tornado.concurrent.Futures, unfortunately it doesn't specify how to do so, especially given how gen.Wait requires a key argument, whereas concurrent.futures.Futures (apparently an alias for asyncio.Future) explicitly has no way to support a key argument. So I don't understand the claim that this is somehow replaceable.
Also add_done_callback seems to be inadequate for this purpose, since the documentation explicitly states that the callback can only take one argument, but fn has two.
Although so far what has worked best (and may actually work, provided I can make the gen.coroutine to async def transition correctly elsewhere) seems to be:
response = await asyncio.Future().add_done_callback(partial(fn, request=request))
This only produces unexpected behavior (endless blocking, seemingly probably because of the insufficient gen.coroutine to async def conversions mentioned above) and no errors. This gives the error TypeError: Can't await NoneType. So I have no clue.
Background: I have tried to figure out what recommendations Tornado gave when gen.Task was updated and finally removed. In the changelog for version 6, however, it does not say how to update our code using gen.Task, only that it has been removed. I have found at least one question on StackOverflow as well as a Tornado GitHub issue where it is said (without giving specific examples or implementation details) that any instance of gen.Task can be replaced with a gen.coroutine. However, since I do not understand the general concepts of asynchronous programming very well, nor the particulars of tornado.gen.Task, it is very difficult for me to figure out how this could be done. It would be great though since it seems to be easy to replace gen.coroutine's with asyncio equivalents -- just async def and await everything.
The result of yield gen.Task is supposed to be, according to the documentation:
Takes a function (and optional additional arguments) and runs it with those arguments plus a callback keyword argument. The argument passed to the callback is returned as the result of the yield expression.
Changed in version 4.0: gen.Task is now a function that returns a Future...
However this seems more complicated than something that can be replaced with gen.coroutine, since it directly creates a Future, rather than awaiting the result of an asynchronous function, and there are numerous ways to create and work with futures in asyncio, and I vaguely remember reading somewhere that Tornado futures and asyncio futures aren't actually equivalent.
The fact that this involves both asynchronous and functional programming makes the problem even more difficult to understand -- I vaguely grasp the functional part, but my understanding of asynchronous programming is very poor, to the extent that it suddenly also makes the functional aspect difficult to understand now too.
What I've tried so far:
response = yield asyncio.add_done_callback(functools.partial(fn, request=request))
giving the error AttributeError: module 'asyncio' has no attribute 'add_done_callback', which, fine, I get that add_done_callback is supposed to be an attribute of an asyncio.Future object, but then what do I make/choose to be the asyncio.Future?
response = yield asyncio.Task(partial(fn, request=request).func)
which gave the error TypeError: a coroutine was expected, got <bound method Class.fn of <SubClass object at 0x7f5df254b748>>.
The reason I tried to use the .func attribute of the partial object is because when I tried:
response = yield asyncio.Task(partial(fn, request=request))
I got the error TypeError: a coroutine was expected, got functools.partial(<bound method Class.fn of <SubClass object at 0x7ffaad59b710>>, request=<tornado.httpclient._RequestProxy object at 0x7ffaad4c8080>). But I only tried to do that because more straightforward attempts at solutions led to complaints about the number of arguments being incorrect.
In particular, trying one of the most naive things,
response = yield asyncio.Task(fn, request)
led to the in hindsight predictable error TypeError: Task() takes at most 1 positional arguments (2 given). The release notes for Tornado 5.0 say that internally all gen.Task's were replaced with asyncio.Task's, but this makes it difficult for me to understand how, since it looks like asyncio.Task is inadequate by itself to handle callbacks.
I had originally been more optimistic and hoped that asyncio.Task would notice that the call signature of fn was fn(request, callback), and would then understand fn(request) to be the partially applied function. But of course that
response = yield asyncio.Task(fn(request))
gave the error TypeError: fn() missing 1 required positional argument: 'callback'.
What's even more confusing is that fn itself is possibly asynchronous, so I thought that using asyncio I might just be able to partially apply it and get back an asynchronous function that takes a callback as an option
response = yield fn(request)
but that just led to the error TypeError: fn() missing 1 required positional argument: 'callback'.
I also tried creating a task or future in asyncio (I'm not sure which of the two I need to create) using the recommended ensure_future and create_task functions, since using Task directly is strongly discouraged according to the asyncio docs. This did not work out well:
response = yield asyncio.create_task(fn, request)
giving the error TypeError: create_task() takes 1 positional argument but 2 were given.
Using ensure_future led to no better result:
response = asyncio.ensure_future(functools.partial(fn, request))
gave the result TypeError: An asyncio.Future, a coroutine or an awaitable is required, and not using partial
response = asyncio.ensure_future(super().fetch_impl, request=request)
gave the error TypeError: ensure_future() got an unexpected keyword argument 'request'.
In case it's relevant, fn is the fetch_impl method of Tornado's CurlAsyncHTTPClient.
Similar questions: These two questions seem similar, but I don't understand how to use their answers for my problem. They probably are applicable, but again my understanding of asynchronous programming in general and asyncio in particular is very bad and I am very stupid. So an answer explaining the answers to these other two questions like I'm a five year old would also be appreciated. Any patience you can muster for my stupidity+ignorance will be appreciated.
How does 'yield' work in tornado when making an asynchronous call?
Extending tornado.gen.Task
I have found at least one question on StackOverflow as well as a Tornado GitHub issue where it is said (without giving specific examples or implementation details) that any instance of gen.Task can be replaced with a gen.coroutine. However, since I do not understand the general concepts of asynchronous programming very well, nor the particulars of tornado.gen.Task, it is very difficult for me to figure out how this could be done. It would be great though since it seems to be easy to replace gen.coroutine's with asyncio equivalents -- just async def and await everything.
You're focusing on "how do I call this thing that takes a callback". The problem is that the entire concept of callbacks has been deprecated and removed from Tornado, so there is no elegant way to call something that takes a callback any more. The intended path forward is to change the thing that takes a callback (i.e., fn) to use gen.coroutine and/or return a Future, and then you can call it directly from your other coroutines.
If fn was using #gen.engine (the first version of coroutines in Tornado), this is fairly easy: just replace #gen.engine with #gen.coroutine and remove any references to the callback argument. The function probably ends with callback(response); replace this with raise gen.Return(response).
If fn was just using raw callbacks without #gen.engine, then updating it to work in the modern way will be harder, and it's something that needs to be handled on a case-by-case basis so I can't give you useful guidance here.
If you're stuck with something that takes a callback and you can't change it, this sequence is almost equivalent to response = yield gen.Task(fn, request):
future = tornado.concurrent.Future()
fn(request, callback=future.set_result)
response = yield future
The difference between this and gen.Task has to do with error handling. If fn raises an exception, gen.Task had some expensive magic to ensure that it could catch that exception and re-raise it in the calling function. Maintaining that magic had some performance cost even for apps that didn't use gen.Task, which is why it was eventually deprecated and removed (along with everything related to callbacks). So you may need to change fn to ensure that any possible exceptions are caught and reported appropriately (again, the recommended way to do this is to move to coroutines where exception handling works more as you'd expect).
If you can update your function to an async def (and therefore use await), then what you need can be expressed as:
future = asyncio.get_event_loop().create_future()
fn(request=request, callback=future.set_result)
response = await future
The "future" object can be awaited, and its set_result method resumes the awaitee. fn doesn't need to know about the future, though, it only sees a callback function.
As this question is huge, I will give my view on this question so that you can simply tell me whether I am right or not. If not, where to correct. If my view is superficial, please present an overview of F# async usage. In mu understanding, to write async program, you need to put async code into "async" block like async{expression}, and use "let!" or "use!" to bind names to primitives, then you need to use method to run this async expression like "Async.Run". In addition, you can use exception handling to deal with exception, and cancellation to cancel when necessary. I also know there are several primitives that defined in F# core libraries, and F# extension of I/O operation. I just need to make sure the relation between these things. If you think my view on async workflows is superficial, please give an overview usage like what I have mentioned above. Thank you very much!
This question is huge, so at best, I can highlight some ideas and point you to learning resources and examples.
The description in the question isn't wrong (though there is no Async.Run function). But the main point about Asyncs is how they execute and why the way they execute is useful.
An async block defines a piece of code that becomes an Async<'T> object, which can be seen as a computation that can be executed at a later time. The Async returns an object of type 'T when its execution has completed -- if it has neither failed nor been cancelled.
let!, do! and use! are used inside of an async block to run another Async and, in the cases of let! and use!, bind its result to a name inside the current async. Unlike for example normal let, which simply binds any value to a name, the versions with an exclamation mark explicitly "import" the result of another async.
When an Async depends on another and waits for its result, such as with a let! binding, it normally does not block a thread. Asyncs utilize the .NET thread pool for easy parallel execution, and after an Async completes that another Async depends on, a continuation runs the remainder of the dependent Async.
The Async functions offer many ready-made ways to run Asyncs, such as Async.Start, which is a simple dispatch of an Async with no result, Async.RunSynchronously, which runs the Async and returns its result as if it were a normal function, Async.Parallel, which combines a sequence of Asyncs into one that executes them in parallel, or Async.StartAsTask, which runs an Async as an independent task. Further methods allow composition of Asyncs in terms of cancellation, or explicit control over continuation after an exception or cancellation.
Asyncs are very useful where waiting times are included: otherwise blocking calls can use Asyncs to not block execution, for example in I/O bound functions.
The best introductions to F# Asyncs I know are written, or co-written, by Don Syme, the lead designer of F#:
The chapter Reactive, Asynchronous, and Parallel Programming in the book Expert F#
A blog post with examples for asyncronous agents
The blog post introducing Asyncs in late 2007