I'm trying to kick off some async tasks in rust, then await them later in the code. Here's a simplified version of my code:
async fn my_async_fn() -> i64 {
return 0;
}
async fn main() {
let mut futures = HashMap::new();
futures.insert("a", my_async_fn());
// this is where I would do other work not blocked by these futures
let res_a = futures.get("a").expect("unreachable").await;
println!("my result: {}", res_a);
}
But when I try to run this, I get this self-contradictory message:
error[E0277]: `&impl futures::Future` is not a future
--> my/code:a:b
|
111 | let res_a = futures.get("a").expect("unreachable").await;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `&impl futures::Future` is not a future
|
= help: the trait `futures::Future` is not implemented for `&impl futures::Future`
= note: required by `futures::Future::poll`
How can I await futures that I've put into a HashMap? Or is there another way altogether?
Using await requires the Future is pinned and mutable.
use std::collections::HashMap;
async fn my_async_fn() -> i64 {
return 0;
}
#[tokio::main]
async fn main() {
let mut futures = HashMap::new();
futures.insert("a", Box::pin(my_async_fn()));
// this is where I would do other work not blocked by these futures
let res_a = futures.get_mut("a").expect("unreachable").await;
println!("my result: {}", res_a);
}
Related
Async closures are still unstable in Rust, as pointed out in the related question What is the difference between |_| async move {} and async move |_| {}, the answer to which I do not really understand.
As far as I do understand, the following is not an async closure:
let mut sum: i32 = 0;
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { sum+=result; })
.await;
println!("->{}", sum);
I was baffled by this, initially: sum is used in the for_each, but it is not moved, otherwise the println! would produce a compiler error. The compiler gives a warning, though, that the "value assigned to sum is never read". But, in fact, sum is copied.
Here is the complete example code
use futures::{stream, StreamExt};
use rand::{thread_rng, Rng};
use std::time::Duration;
async fn compute(i: i32) -> i32 {
let mut rng = thread_rng();
let sleep_ms: u64 = rng.gen_range(0..1000);
tokio::time::sleep(Duration::from_millis(sleep_ms)).await;
println!("#{} done", i);
i * i
}
async fn sum_with_fold() {
let sum = stream::iter(1..25)
.map(compute)
.buffered(12)
.fold(0, |sum,x| async move {sum+x} )
.await;
println!("->{}", sum);
}
async fn sum_with_closure() {
let mut sum: i32 = 0;
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { sum+=result; })
.await;
println!("->{}", sum);
}
#[tokio::main]
async fn main() {
sum_with_fold().await;
sum_with_closure().await;
}
// Cargo.toml:
// [dependencies]
// futures = "0.3"
// rand = "0.8"
// tokio = { version = "1", features = ["full"] }
The fold works correctly, whereas sum_with_closure works on a copied sum and this sum cannot be retrieved.
Am I getting this right, and can it be fixed? I.e. is there a way to do the fold with a closure like this? Or am I indeed running into the unstable async closure feature?
This can be done with for_each, but current Rust can't check at compile-time that the lifetimes are correct so you need to use a RefCell to enable run-time checking (with a small performance cost):
async fn sum_with_closure() {
use std::cell::RefCell;
let sum = RefCell::new (0);
let sumref = ∑
stream::iter(1..25)
.map(compute)
.buffered(12)
.for_each(|result| async move { *sumref.borrow_mut() +=result; })
.await;
println!("->{}", sum.into_inner());
}
Playground
I am trying to use hyper to grab the content of an HTML page and would like to synchronously return the output of a future. I realized I could have picked a better example since synchronous HTTP requests already exist, but I am more interested in understanding whether we could return a value from an async calculation.
extern crate futures;
extern crate hyper;
extern crate hyper_tls;
extern crate tokio;
use futures::{future, Future, Stream};
use hyper::Client;
use hyper::Uri;
use hyper_tls::HttpsConnector;
use std::str;
fn scrap() -> Result<String, String> {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
futures::future::ok(s_body)
})
}).map_err(|err| format!("Error scraping web page: {:?}", &err))
});
scraped_content.wait()
}
fn read() {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
println!("Reading body: {}", s_body);
Ok(())
})
}).map_err(|err| {
println!("Error reading webpage: {:?}", &err);
})
});
tokio::run(scraped_content);
}
fn main() {
read();
let content = scrap();
println!("Content = {:?}", &content);
}
The example compiles and the call to read() succeeds, but the call to scrap() panics with the following error message:
Content = Err("Error scraping web page: Error { kind: Execute, cause: None }")
I understand that I failed to launch the task properly before calling .wait() on the future but I couldn't find how to properly do it, assuming it's even possible.
Standard library futures
Let's use this as our minimal, reproducible example:
async fn example() -> i32 {
42
}
Call executor::block_on:
use futures::executor; // 0.3.1
fn main() {
let v = executor::block_on(example());
println!("{}", v);
}
Tokio
Use the tokio::main attribute on any function (not just main!) to convert it from an asynchronous function to a synchronous one:
use tokio; // 0.3.5
#[tokio::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
tokio::main is a macro that transforms this
#[tokio::main]
async fn main() {}
Into this:
fn main() {
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap()
.block_on(async { {} })
}
This uses Runtime::block_on under the hood, so you can also write this as:
use tokio::runtime::Runtime; // 0.3.5
fn main() {
let v = Runtime::new().unwrap().block_on(example());
println!("{}", v);
}
For tests, you can use tokio::test.
async-std
Use the async_std::main attribute on the main function to convert it from an asynchronous function to a synchronous one:
use async_std; // 1.6.5, features = ["attributes"]
#[async_std::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
For tests, you can use async_std::test.
Futures 0.1
Let's use this as our minimal, reproducible example:
use futures::{future, Future}; // 0.1.27
fn example() -> impl Future<Item = i32, Error = ()> {
future::ok(42)
}
For simple cases, you only need to call wait:
fn main() {
let s = example().wait();
println!("{:?}", s);
}
However, this comes with a pretty severe warning:
This method is not appropriate to call on event loops or similar I/O situations because it will prevent the event loop from making progress (this blocks the thread). This method should only be called when it's guaranteed that the blocking work associated with this future will be completed by another thread.
Tokio
If you are using Tokio 0.1, you should use Tokio's Runtime::block_on:
use tokio; // 0.1.21
fn main() {
let mut runtime = tokio::runtime::Runtime::new().expect("Unable to create a runtime");
let s = runtime.block_on(example());
println!("{:?}", s);
}
If you peek in the implementation of block_on, it actually sends the future's result down a channel and then calls wait on that channel! This is fine because Tokio guarantees to run the future to completion.
See also:
How can I efficiently extract the first element of a futures::Stream in a blocking manner?
As this is the top result that come up in search engines by the query "How to call async from sync in Rust", I decided to share my solution here. I think it might be useful.
As #Shepmaster mentioned, back in version 0.1 futures crate had beautiful method .wait() that could be used to call an async function from a sync one. This must-have method, however, was removed from later versions of the crate.
Luckily, it's not that hard to re-implement it:
trait Block {
fn wait(self) -> <Self as futures::Future>::Output
where Self: Sized, Self: futures::Future
{
futures::executor::block_on(self)
}
}
impl<F,T> Block for F
where F: futures::Future<Output = T>
{}
After that, you can just do following:
async fn example() -> i32 {
42
}
fn main() {
let s = example().wait();
println!("{:?}", s);
}
Beware that this comes with all the caveats of original .wait() explained in the #Shepmaster's answer.
This works for me using tokio:
tokio::runtime::Runtime::new()?.block_on(fooAsyncFunction())?;
I have a simple struct and an implementation that looks like this.
#[derive(Debug)]
struct MyStruct {
data: u64,
}
impl MyStruct {
async fn something_async(&mut self) -> Result<(), Box<dyn Error>> {
self.data += 1;
Ok(())
}
}
I want to use MyStruct on the heap using a smart pointer and a mutex so that I can use it from multiple threads.
However, when I try to call the async function called something_async()...
tokio::spawn(async move {
let ptr = Arc::new(Mutex::new(MyStruct { data: 1 }));
let mut s = ptr.lock().unwrap();
s.something_async().await.unwrap();
println!("{:?}", s);
});
...I get the following error:
error: future cannot be sent between threads safely
--> src/main.rs:18:5
|
18 | tokio::spawn(async move {
| ^^^^^^^^^^^^ future created by async block is not `Send`
|
= help: within `impl Future<Output = [async output]>`, the trait `Send` is not implemented for `std::sync::MutexGuard<'_, MyStruct>`
note: future is not `Send` as this value is used across an await
--> src/main.rs:21:9
|
20 | let mut s = ptr.lock().unwrap();
| ----- has type `std::sync::MutexGuard<'_, MyStruct>` which is not `Send`
21 | s.something_async().await.unwrap();
| ^^^^^^^^^^^^^^^^^^^^^^^^ await occurs here, with `mut s` maybe used later
22 | println!("{:?}", s);
23 | });
| - `mut s` is later dropped here
note: required by a bound in `tokio::spawn`
--> /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/spawn.rs:127:21
|
127 | T: Future + Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
I am assuming that the compiler does not like me keeping a MutexGuard held across an await, so I tried this instead:
let fut = {
let mut s = ptr.lock().unwrap();
s.something_async()
};
fut.await.unwrap();
But of course then it complains about the Future outliving the mutable reference not living long enough:
error[E0597]: `s` does not live long enough
--> src/main.rs:22:13
|
20 | let fut = {
| --- borrow later stored here
21 | let mut s = ptr.lock().unwrap();
22 | s.something_async()
| ^^^^^^^^^^^^^^^^^^ borrowed value does not live long enough
23 | };
| - `s` dropped here while still borrowed
How do I call an async method on an object wrapped in an Arc<Mutex<T>>?
Make MyStruct copyable.
#[derive(Debug, Copy, Clone)]
struct MyStruct {
data: u64,
}
impl MyStruct {
async fn something_async(&mut self) -> Result<(), Box<dyn Error>> {
self.data += 1;
Ok(())
}
}
#[tokio::main]
async fn main() {
tokio::spawn(async move {
let ptr = Arc::new(Mutex::new(MyStruct { data: 1 }));
let mut s = *ptr.lock().unwrap();
s.something_async().await.unwrap();
println!("{:?}", s);
});
}
I have a struct Test I want to implement std::future::Future that would poll function:
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
struct Test;
impl Test {
async fn function(&mut self) {}
}
impl Future for Test {
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
match self.function() {
Poll::Pending => Poll::Pending,
Poll::Ready(_) => Poll::Ready(()),
}
}
}
That didn't work:
error[E0308]: mismatched types
--> src/lib.rs:17:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
17 | Poll::Pending => Poll::Pending,
| ^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
error[E0308]: mismatched types
--> src/lib.rs:18:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
18 | Poll::Ready(_) => Poll::Ready(()),
| ^^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
I understand that function must be called once, the returned Future must be stored somewhere in the struct, and then the saved future must be polled. I tried this:
struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
impl Test {
async fn function(&mut self) {}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(Box::pin(s.function()));
s
}
}
That also didn't work:
error[E0277]: the size for values of type `(dyn Future<Output = ()> + 'static)` cannot be known at compilation time
--> src/lib.rs:7:13
|
7 | struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `Sized` is not implemented for `(dyn Future<Output = ()> + 'static)`
After I call function() I have taken a &mut reference of Test, because of that I can't change the Test variable, and therefore can't store the returned Future inside the Test.
I did get an unsafe solution (inspired by this)
struct Test<'a>(Option<BoxFuture<'a, ()>>);
impl Test<'_> {
async fn function(&mut self) {
println!("I'm alive!");
}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(unsafe { &mut *(&mut s as *mut Self) }.function().boxed());
s
}
}
impl Future for Test<'_> {
type Output = ();
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
self.0.as_mut().unwrap().poll_unpin(cx)
}
}
I hope that there is another way.
Though there are times when you may want to do things similar to what you're trying to accomplish here, they are a rarity. So most people reading this, maybe even OP, may wish to restructure such that struct state and data used for a single async execution are different objects.
To answer your question, yes it is somewhat possible. Unless you want to absolutely resort to unsafe code you will need to use Mutex and Arc. All fields you wish to manipulate inside the async fn will have to be wrapped inside a Mutex and the function itself will accept an Arc<Self>.
I must stress, however, that this is not a beautiful solution and you probably don't want to do this. Depending on your specific case your solution may vary, but my guess of what OP is trying to accomplish while using Streams would be better solved by something similar to this gist that I wrote.
use std::{
future::Future,
pin::Pin,
sync::{Arc, Mutex},
};
struct Test {
state: Mutex<Option<Pin<Box<dyn Future<Output = ()>>>>>,
// if available use your async library's Mutex to `.await` locks on `buffer` instead
buffer: Mutex<Vec<u8>>,
}
impl Test {
async fn function(self: Arc<Self>) {
for i in 0..16u8 {
let data: Vec<u8> = vec![i]; // = fs::read(&format("file-{}.txt", i)).await.unwrap();
let mut buflock = self.buffer.lock().unwrap();
buflock.extend_from_slice(&data);
}
}
pub fn new() -> Arc<Self> {
let s = Arc::new(Self {
state: Default::default(),
buffer: Default::default(),
});
{
// start by trying to aquire a lock to the Mutex of the Box
let mut lock = s.state.lock().unwrap();
// create boxed future
let b = Box::pin(s.clone().function());
// insert value into the mutex
*lock = Some(b);
} // block causes the lock to be released
s
}
}
impl Future for Test {
type Output = ();
fn poll(
self: std::pin::Pin<&mut Self>,
ctx: &mut std::task::Context<'_>,
) -> std::task::Poll<<Self as std::future::Future>::Output> {
let mut lock = self.state.lock().unwrap();
let fut: &mut Pin<Box<dyn Future<Output = ()>>> = lock.as_mut().unwrap();
Future::poll(fut.as_mut(), ctx)
}
}
I'm not sure what you want to achieve and why, but I suspect that you're trying to implement Future for Test based on some ancient tutorial or misunderstanding and just overcomplicating things.
You don't have to implement Future manually. An async function
async fn function(...) {...}
is really just syntax sugar translated behind the scenes into something like
fn function(...) -> Future<()> {...}
All you have to do is to use the result of the function the same way as any future, e.g. use await on it or call block a reactor until it's finished. E.g. based on your first version, you can simply call:
let mut test = Test{};
test.function().await;
UPDATE1
Based on your descriptions I still think you're trying to overcomplicate this minimal working snippet without the need to manually implement Future for anything:
async fn asyncio() { println!("Doing async IO"); }
struct Test {
count: u32,
}
impl Test {
async fn function(&mut self) {
asyncio().await;
self.count += 1;
}
}
#[tokio::main]
async fn main() {
let mut test = Test{count: 0};
test.function().await;
println!("Count: {}", test.count);
}
I'm trying to find a simple way to implement the Future trait from the future crate, version 0.2.1:
extern crate futures;
use futures::executor::ThreadPool;
use futures::prelude::*;
use futures::task::Context;
use std::{thread, time::Duration};
struct SendThree {
firstTime: bool,
}
impl Future for SendThree {
type Item = u32;
type Error = Never;
fn poll(&mut self, ctx: &mut Context) -> Result<Async<Self::Item>, Never> {
if self.firstTime {
self.firstTime = false;
thread::spawn(move || {
thread::sleep(Duration::from_millis(10));
ctx.waker().wake();
});
Ok(Async::Pending)
} else {
Ok(Async::Ready(3))
}
}
}
fn main() {
let mut fut = SendThree { firstTime: true };
let mut executor: ThreadPool = ThreadPool::new().unwrap();
let result = executor.run(fut).unwrap();
println!("{}", result);
}
playground
My problem is that the Context variable is not Send so I can't call wake from another thread:
error[E0277]: the trait bound `futures::executor::Executor: std::marker::Send` is not satisfied
--> src/main.rs:19:13
|
19 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `futures::executor::Executor` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `futures::executor::Executor`
= note: required because of the requirements on the impl of `std::marker::Send` for `&mut futures::executor::Executor`
= note: required because it appears within the type `std::option::Option<&mut futures::executor::Executor>`
= note: required because it appears within the type `futures::task::Context<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `&mut futures::task::Context<'_>`
= note: required because it appears within the type `[closure#src/main.rs:19:27: 22:14 ctx:&mut futures::task::Context<'_>]`
= note: required by `std::thread::spawn`
If I change the code to this it works, but I can't do a thread::sleep without blocking:
if self.firstTime {
self.firstTime = false;
ctx.waker().wake();
Ok(Async::Pending)
}
Is there an idiomatic way to implement this?
While Context isn't able to be sent across threads, a Waker is:
fn poll(&mut self, ctx: &mut Context) -> Result<Async<Self::Item>, Never> {
if self.first_time {
self.first_time = false;
let waker = ctx.waker().clone();
thread::spawn(move || {
thread::sleep(Duration::from_millis(10));
waker.wake();
});
Ok(Async::Pending)
} else {
Ok(Async::Ready(3))
}
}
Note that this is a really inefficient way to implement timeouts. There are Futures-native methods that should be used instead.