How to extract values from async functions to a non-async one? [duplicate] - asynchronous

I am trying to use hyper to grab the content of an HTML page and would like to synchronously return the output of a future. I realized I could have picked a better example since synchronous HTTP requests already exist, but I am more interested in understanding whether we could return a value from an async calculation.
extern crate futures;
extern crate hyper;
extern crate hyper_tls;
extern crate tokio;
use futures::{future, Future, Stream};
use hyper::Client;
use hyper::Uri;
use hyper_tls::HttpsConnector;
use std::str;
fn scrap() -> Result<String, String> {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
futures::future::ok(s_body)
})
}).map_err(|err| format!("Error scraping web page: {:?}", &err))
});
scraped_content.wait()
}
fn read() {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
println!("Reading body: {}", s_body);
Ok(())
})
}).map_err(|err| {
println!("Error reading webpage: {:?}", &err);
})
});
tokio::run(scraped_content);
}
fn main() {
read();
let content = scrap();
println!("Content = {:?}", &content);
}
The example compiles and the call to read() succeeds, but the call to scrap() panics with the following error message:
Content = Err("Error scraping web page: Error { kind: Execute, cause: None }")
I understand that I failed to launch the task properly before calling .wait() on the future but I couldn't find how to properly do it, assuming it's even possible.

Standard library futures
Let's use this as our minimal, reproducible example:
async fn example() -> i32 {
42
}
Call executor::block_on:
use futures::executor; // 0.3.1
fn main() {
let v = executor::block_on(example());
println!("{}", v);
}
Tokio
Use the tokio::main attribute on any function (not just main!) to convert it from an asynchronous function to a synchronous one:
use tokio; // 0.3.5
#[tokio::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
tokio::main is a macro that transforms this
#[tokio::main]
async fn main() {}
Into this:
fn main() {
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap()
.block_on(async { {} })
}
This uses Runtime::block_on under the hood, so you can also write this as:
use tokio::runtime::Runtime; // 0.3.5
fn main() {
let v = Runtime::new().unwrap().block_on(example());
println!("{}", v);
}
For tests, you can use tokio::test.
async-std
Use the async_std::main attribute on the main function to convert it from an asynchronous function to a synchronous one:
use async_std; // 1.6.5, features = ["attributes"]
#[async_std::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
For tests, you can use async_std::test.
Futures 0.1
Let's use this as our minimal, reproducible example:
use futures::{future, Future}; // 0.1.27
fn example() -> impl Future<Item = i32, Error = ()> {
future::ok(42)
}
For simple cases, you only need to call wait:
fn main() {
let s = example().wait();
println!("{:?}", s);
}
However, this comes with a pretty severe warning:
This method is not appropriate to call on event loops or similar I/O situations because it will prevent the event loop from making progress (this blocks the thread). This method should only be called when it's guaranteed that the blocking work associated with this future will be completed by another thread.
Tokio
If you are using Tokio 0.1, you should use Tokio's Runtime::block_on:
use tokio; // 0.1.21
fn main() {
let mut runtime = tokio::runtime::Runtime::new().expect("Unable to create a runtime");
let s = runtime.block_on(example());
println!("{:?}", s);
}
If you peek in the implementation of block_on, it actually sends the future's result down a channel and then calls wait on that channel! This is fine because Tokio guarantees to run the future to completion.
See also:
How can I efficiently extract the first element of a futures::Stream in a blocking manner?

As this is the top result that come up in search engines by the query "How to call async from sync in Rust", I decided to share my solution here. I think it might be useful.
As #Shepmaster mentioned, back in version 0.1 futures crate had beautiful method .wait() that could be used to call an async function from a sync one. This must-have method, however, was removed from later versions of the crate.
Luckily, it's not that hard to re-implement it:
trait Block {
fn wait(self) -> <Self as futures::Future>::Output
where Self: Sized, Self: futures::Future
{
futures::executor::block_on(self)
}
}
impl<F,T> Block for F
where F: futures::Future<Output = T>
{}
After that, you can just do following:
async fn example() -> i32 {
42
}
fn main() {
let s = example().wait();
println!("{:?}", s);
}
Beware that this comes with all the caveats of original .wait() explained in the #Shepmaster's answer.

This works for me using tokio:
tokio::runtime::Runtime::new()?.block_on(fooAsyncFunction())?;

Related

Async move closure vs. fold

Async closures are still unstable in Rust, as pointed out in the related question What is the difference between |_| async move {} and async move |_| {}, the answer to which I do not really understand.
As far as I do understand, the following is not an async closure:
let mut sum: i32 = 0;
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { sum+=result; })
.await;
println!("->{}", sum);
I was baffled by this, initially: sum is used in the for_each, but it is not moved, otherwise the println! would produce a compiler error. The compiler gives a warning, though, that the "value assigned to sum is never read". But, in fact, sum is copied.
Here is the complete example code
use futures::{stream, StreamExt};
use rand::{thread_rng, Rng};
use std::time::Duration;
async fn compute(i: i32) -> i32 {
let mut rng = thread_rng();
let sleep_ms: u64 = rng.gen_range(0..1000);
tokio::time::sleep(Duration::from_millis(sleep_ms)).await;
println!("#{} done", i);
i * i
}
async fn sum_with_fold() {
let sum = stream::iter(1..25)
.map(compute)
.buffered(12)
.fold(0, |sum,x| async move {sum+x} )
.await;
println!("->{}", sum);
}
async fn sum_with_closure() {
let mut sum: i32 = 0;
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { sum+=result; })
.await;
println!("->{}", sum);
}
#[tokio::main]
async fn main() {
sum_with_fold().await;
sum_with_closure().await;
}
// Cargo.toml:
// [dependencies]
// futures = "0.3"
// rand = "0.8"
// tokio = { version = "1", features = ["full"] }
The fold works correctly, whereas sum_with_closure works on a copied sum and this sum cannot be retrieved.
Am I getting this right, and can it be fixed? I.e. is there a way to do the fold with a closure like this? Or am I indeed running into the unstable async closure feature?
This can be done with for_each, but current Rust can't check at compile-time that the lifetimes are correct so you need to use a RefCell to enable run-time checking (with a small performance cost):
async fn sum_with_closure() {
use std::cell::RefCell;
let sum = RefCell::new (0);
let sumref = ∑
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { *sumref.borrow_mut() +=result; })
.await;
println!("->{}", sum.into_inner());
}
Playground

How can I use futures::stream::StreamExt::for_each with error handling in the callback?

I have this piece of code which is supposed to serialize a futures::stream::Stream to a Write. I want this code to return Err if write_all fails, but I don't see a way to get that out of the callback to for_each. I found How to send data through a futures Stream by writing through the io::Write trait?, but I don't understand how to make it work in my situation.
use std::io::Write;
use std::error::Error;
use futures::stream::StreamExt;
pub async fn download(url: &str, mut dest: impl Write) -> Result<(), Box<dyn Error>> {
let byte_stream = reqwest::get(url).await?.bytes_stream();
byte_stream.for_each(|bytes| {
if let Ok(bytes) = bytes {
dest.write_all(&bytes).expect("failed to write");
}
futures::future::ready(())
}).await;
Ok(())
}
Cargo.toml
[dependencies]
reqwest = { version = "0.11", features = ["json", "stream"] }
tokio = { version = "1", features = ["full"] }
futures = "0.3"
bytes = "1"
I'd probably use StreamExt::next and write the loop out a bit more manually:
use futures::stream::StreamExt;
use std::{error::Error, io::Write};
pub async fn download(url: &str, mut dest: impl Write) -> Result<(), Box<dyn Error>> {
let mut byte_stream = reqwest::get(url).await?.bytes_stream();
while let Some(bytes) = byte_stream.next().await {
let bytes = bytes?;
dest.write_all(&bytes)?;
}
Ok(())
}
Notes:
These is TryStreamExt::try_for_each which can be used here, but it wants to take ownership of dest so I went in a different direction.
Mixing synchronous IO (std::io::Write) and asynchronous IO (reqwest) is a bad idea without taking special care, which is not demonstrated in this answer.
See also:
What is the best approach to encapsulate blocking I/O in future-rs?

Is there a way to do recursive async calls on Rust without Box?

Recursive calls with a retry count embedded into the argument is good for the mental model of the project because you don't have to keep thinking about the state of the object. The retry_count is passed in every call. Here's a simple implementation:
use std::future::Future;
use futures::future::{BoxFuture, FutureExt};
fn do_or_fail() -> std::result::Result<(),()> {
Ok(())
}
fn do_something<'a>(
retry_count: u32
) -> BoxFuture<'a, std::result::Result<(), ()>> {
async move {
match do_or_fail() {
Ok(_) => Ok(()),
Err(_) => do_something(retry_count -1).await
}
}.boxed()
}
fn main() {
do_something(3);
}
Playground
The problem is that this requires a dynamic allocation at every call so it can return the BoxFuture. This is because of how async is implemented in Rust. It generates a state machine for every await call so the return type is not Sized.
What would be good ways to overcome dynamic allocation in recursive async calls?
While it technically doesn't overcome the problem (it just hides it), I can recommend the crate async_recursion.
https://crates.io/crates/async-recursion
use async_recursion::async_recursion;
use tokio;
fn do_or_fail() -> std::result::Result<(), ()> {
Ok(())
}
#[async_recursion]
async fn do_something(retry_count: u32) -> std::result::Result<(), ()> {
match do_or_fail() {
Ok(_) => Ok(()),
Err(_) => do_something(retry_count - 1).await,
}
}
#[tokio::main]
async fn main() {
do_something(3).await.unwrap()
}
I don't think you actually can overcome dynamic allocation in async calls, that's just how async works in general.

How to convert a Future into a Stream?

I'm trying to use async_std to receive UDP datagrams from the network.
There is a UdpSocket that implements async recv_from, this method returns a future but I need a async_std::stream::Stream that gives a stream of UDP datagrams because it is a better abstraction.
I've found tokio::net::UdpFramed that does exactly what I need but it is not available in current versions of tokio.
Generally speaking the question is how do I convert Futures from a given async function into a Stream?
For a single item, use FutureExt::into_stream:
use futures::prelude::*; // 0.3.1
fn outer() -> impl Stream<Item = i32> {
inner().into_stream()
}
async fn inner() -> i32 {
42
}
For a stream from a number of futures generated by a closure, use stream::unfold:
use futures::prelude::*; // 0.3.1
fn outer() -> impl Stream<Item = i32> {
stream::unfold((), |()| async { Some((inner().await, ())) })
}
async fn inner() -> i32 {
42
}
In your case, you can use stream::unfold:
use async_std::{io, net::UdpSocket}; // 1.4.0, features = ["attributes"]
use futures::prelude::*; // 0.3.1
fn read_many(s: UdpSocket) -> impl Stream<Item = io::Result<Vec<u8>>> {
stream::unfold(s, |s| {
async {
let data = read_one(&s).await;
Some((data, s))
}
})
}
async fn read_one(s: &UdpSocket) -> io::Result<Vec<u8>> {
let mut data = vec![0; 1024];
let (len, _) = s.recv_from(&mut data).await?;
data.truncate(len);
Ok(data)
}
#[async_std::main]
async fn main() -> io::Result<()> {
let s = UdpSocket::bind("0.0.0.0:9876").await?;
read_many(s)
.for_each(|d| {
async {
match d {
Ok(d) => match std::str::from_utf8(&d) {
Ok(s) => println!("{}", s),
Err(_) => println!("{:x?}", d),
},
Err(e) => eprintln!("Error: {}", e),
}
}
})
.await;
Ok(())
}
Generally speaking the question is how do I convert Futures from a given async function into a Stream?
There is FutureExt::into_stream, but don't let the name fool you; it is not a good fit for your situation.
There is a UdpSocket that implements async recv_from, this method returns a future but I need a async_std::stream::Stream that gives a stream of UDP datagrams because it is a better abstraction.
It is not necessarily a better abstraction here.
Specifically, async-std's UdpSocket::recv_from returns a future that has output type of (usize, SocketAddr) — the size of the data received and the peer address. If you were to use into_stream to convert it to a stream, it would give you just that, not the data received.
I've found tokio::net::UdpFramed that does exactly what I need but it is not available in current versions of tokio.
It has been moved to tokio-util crate. Unfortunately, you can't (easily) use that either. It requires a tokio::net::UdpSocket, which is not the same as async_std::net::UdpSocket.
You can, of course, use futures utility functions such as futures::stream::poll_fn or futures::stream::unfold to give UdpSocket::recv_from a futures::stream::Stream facade, but then what will you do with that? If you end up calling StreamExt::next to poll a value out of it, you could have used recv_from directly.
It is only necessary to reach for Stream if some API you are using requires a Stream input, such as rusoto:
Is it possible to create a Stream from a File rather than loading the file contents into memory?

What happens to an async task when it is aborted?

Rust has async methods that can be tied to Abortable futures. The documentation says that, when aborted:
the future will complete immediately without making any further progress.
Will the variables owned by the task bound to the future be dropped? If those variables implement drop, will drop be called? If the future has spawned other futures, will all of them be aborted in a chain?
E.g.: In the following snippet, I don't see the destructor happening for the aborted task, but I don't know if it is not called or happens in a separate thread where the print is not shown.
use futures::executor::block_on;
use futures::future::{AbortHandle, Abortable};
struct S {
i: i32,
}
impl Drop for S {
fn drop(&mut self) {
println!("dropping S");
}
}
async fn f() -> i32 {
let s = S { i: 42 };
std::thread::sleep(std::time::Duration::from_secs(2));
s.i
}
fn main() {
println!("first test...");
let (abort_handle, abort_registration) = AbortHandle::new_pair();
let _ = Abortable::new(f(), abort_registration);
abort_handle.abort();
std::thread::sleep(std::time::Duration::from_secs(1));
println!("second test...");
let (_, abort_registration) = AbortHandle::new_pair();
let task = Abortable::new(f(), abort_registration);
block_on(task).unwrap();
std::thread::sleep(std::time::Duration::from_secs(1));
}
playground
Yes, values that have been created will be dropped.
In your first example, the future returned by f is never started, so the S is never created. This means that it cannot be dropped.
In the second example, the value is dropped.
This is more obvious if you both run the future and abort it. Here, I spawn two concurrent futures:
create an S and waits 200ms
wait 100ms and abort future #1
use futures::future::{self, AbortHandle, Abortable};
use std::time::Duration;
use tokio::time;
struct S {
i: i32,
}
impl S {
fn new(i: i32) -> Self {
println!("Creating S {}", i);
S { i }
}
}
impl Drop for S {
fn drop(&mut self) {
println!("Dropping S {}", self.i);
}
}
#[tokio::main]
async fn main() {
let create_s = async {
let s = S::new(42);
time::delay_for(Duration::from_millis(200)).await;
println!("Creating {} done", s.i);
};
let (abort_handle, abort_registration) = AbortHandle::new_pair();
let create_s = Abortable::new(create_s, abort_registration);
let abort_s = async move {
time::delay_for(Duration::from_millis(100)).await;
abort_handle.abort();
};
let c = tokio::spawn(create_s);
let a = tokio::spawn(abort_s);
let (c, a) = future::join(c, a).await;
println!("{:?}, {:?}", c, a);
}
Creating S 42
Dropping S 42
Ok(Err(Aborted)), Ok(())
Note that I've switched to Tokio to be able to use time::delay_for, as you should never use blocking operations in an async function.
See also:
Why does Future::select choose the future with a longer sleep period first?
What is the best approach to encapsulate blocking I/O in future-rs?
If the future has spawned other futures, will all of them be aborted in a chain?
No, when you spawn a future, it is disconnected from where it was spawned.
See also:
What is the purpose of async/await in Rust?

Resources