I am trying to add a return type of a future in a closure. But the compiler is telling me that
`impl Trait` only allowed in function and inherent method return types, not in closure return
I have have also tried wrapping it in Box but that didn't work. I tried type aliases but they are nighly only features. I am unable to understand how else can I solve it.
pub async fn execute_event_handler(
event_handler: Option<Arc<PyFunction>>,
event_loop: Arc<Py<PyAny>>,
) -> Result<(), Box<dyn std::error::Error>> {
if let Some(handler) = event_handler {
match &(*handler) {
PyFunction::SyncFunction(function) => {
println!("Startup event handler");
Python::with_gil(|py| -> Result<(), Box<dyn std::error::Error>> {
function.call0(py)?;
Ok(())
})?;
}
PyFunction::CoRoutine(function) => {
let future = Python::with_gil(
|py| -> impl Future<Output = Result<Py<PyAny>, PyErr>> + Send { // this is giving an error
println!("Startup event handler async");
let coroutine = function.as_ref(py).call0().unwrap();
pyo3_asyncio::into_future_with_loop((*event_loop).as_ref(py), coroutine)
.unwrap()
},
);
future.await?;
}
}
}
Ok(())
}
PS: I want to wrap the closure output in an Result type and hence a return type is necessary.
Don't annotate the future.
If you need to annotate the enclosing type (e.g. Result), you can either annotate it when returning it (Result::<_, ErrorType>) or as the return type, but let inference find the future type itself with _: || -> Result<_, ErrorType>.
Related
Tokio's spawn can only work with a Send future. This makes the following code invalid:
async fn async_foo(v: i32) {}
async fn async_computation() -> Result<i32, Box<dyn std::error::Error>> {
Ok(1)
}
async fn async_start() {
match async_computation().await {
Ok(v) => async_foo(v).await,
_ => unimplemented!(),
};
}
#[tokio::main]
async fn main() {
tokio::spawn(async move {
async_start().await;
});
}
The error is:
future cannot be sent between threads safely
the trait `Send` is not implemented for `dyn std::error::Error`
If I understand correctly: because async_foo(v).await might yield, Rust internally have to save all the context which might be on a different thread - hence the result of async_computation().await must be Send - which dyn std::error::Error is not.
This could be mitigated if the non-Send type can be dropped before the .await, such as:
async fn async_start() {
let result;
match async_computation().await {
Ok(v) => result = v,
_ => return,
};
async_foo(result).await;
}
However once another .await is needed before the non-Send type is dropped, the workarounds are more and more awkward.
What a is a good practice for this - especially when the non-Send type is for generic error handling (the Box<dyn std::error::Error>)? Is there a better type for errors that common IO ops implements (async or not) in Rust? Or there is a better way to group nested async calls?
Most errors are Send so you can just change the return type to:
Box<dyn std::error::Error + Send>
It's also common to have + Sync.
I am getting data from the open weather map API. Currently the data is being retrieved synchronously which is slow. However, the function has to be synchronous as it is part of a library, but it can call an async function. How might I still make concurrent requests to increase performance? A solution that does not use reqwests works, but reqwests is preferred.
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let mut data: Vec<HashMap<String, String>> = Vec::new();
for url in urls {
let request = reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected");
data.push(request);
}
return data;
}
If your program isn't already async, probably the easiest way might be to use rayon.
use reqwest;
use std::collections::HashMap;
use rayon::prelude::*;
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let data : Vec<_>= urls
.par_iter()
.map(|&url| reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected"))
.collect();
return data;
}
The easiest is probably to use tokios new_current_thread runtime and blocking on the data retreival.
use std::collections::HashMap;
use tokio::runtime;
pub fn collect_data() -> Vec<HashMap<String, String>> {
let rt = runtime::Builder::new_current_thread()
.build()
.expect("couldn't start runtime");
let urls = vec!["https://example.com/a", "https://example.com/b"];
rt.block_on(async move {
let mut data = vec![];
for url in urls {
data.push(async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
});
}
futures::future::join_all(data).await
})
}
You need an asynchronous runtime in order to call asynchronous functions. The easiest way to get one is to use the #[tokio::main] attribute (which despite the name can be applied to any function):
#[tokio::main]
fn get_combined_data(
open_weather_map_api_url: String,
open_weather_map_api_key: String,
coordinates: Vec<String>,
metric: bool,
) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(
open_weather_map_api_url,
open_weather_map_api_key,
coordinates
.get(0)
.expect("Improper coordinates")
.to_string()
+ ","
+ coordinates.get(1).expect("Improper coordinates"),
metric,
);
futures::future::join_all (urls.map (|u| {
async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
}
})).await
}
Recursive calls with a retry count embedded into the argument is good for the mental model of the project because you don't have to keep thinking about the state of the object. The retry_count is passed in every call. Here's a simple implementation:
use std::future::Future;
use futures::future::{BoxFuture, FutureExt};
fn do_or_fail() -> std::result::Result<(),()> {
Ok(())
}
fn do_something<'a>(
retry_count: u32
) -> BoxFuture<'a, std::result::Result<(), ()>> {
async move {
match do_or_fail() {
Ok(_) => Ok(()),
Err(_) => do_something(retry_count -1).await
}
}.boxed()
}
fn main() {
do_something(3);
}
Playground
The problem is that this requires a dynamic allocation at every call so it can return the BoxFuture. This is because of how async is implemented in Rust. It generates a state machine for every await call so the return type is not Sized.
What would be good ways to overcome dynamic allocation in recursive async calls?
While it technically doesn't overcome the problem (it just hides it), I can recommend the crate async_recursion.
https://crates.io/crates/async-recursion
use async_recursion::async_recursion;
use tokio;
fn do_or_fail() -> std::result::Result<(), ()> {
Ok(())
}
#[async_recursion]
async fn do_something(retry_count: u32) -> std::result::Result<(), ()> {
match do_or_fail() {
Ok(_) => Ok(()),
Err(_) => do_something(retry_count - 1).await,
}
}
#[tokio::main]
async fn main() {
do_something(3).await.unwrap()
}
I don't think you actually can overcome dynamic allocation in async calls, that's just how async works in general.
I have a struct Test I want to implement std::future::Future that would poll function:
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
struct Test;
impl Test {
async fn function(&mut self) {}
}
impl Future for Test {
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
match self.function() {
Poll::Pending => Poll::Pending,
Poll::Ready(_) => Poll::Ready(()),
}
}
}
That didn't work:
error[E0308]: mismatched types
--> src/lib.rs:17:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
17 | Poll::Pending => Poll::Pending,
| ^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
error[E0308]: mismatched types
--> src/lib.rs:18:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
18 | Poll::Ready(_) => Poll::Ready(()),
| ^^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
I understand that function must be called once, the returned Future must be stored somewhere in the struct, and then the saved future must be polled. I tried this:
struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
impl Test {
async fn function(&mut self) {}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(Box::pin(s.function()));
s
}
}
That also didn't work:
error[E0277]: the size for values of type `(dyn Future<Output = ()> + 'static)` cannot be known at compilation time
--> src/lib.rs:7:13
|
7 | struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `Sized` is not implemented for `(dyn Future<Output = ()> + 'static)`
After I call function() I have taken a &mut reference of Test, because of that I can't change the Test variable, and therefore can't store the returned Future inside the Test.
I did get an unsafe solution (inspired by this)
struct Test<'a>(Option<BoxFuture<'a, ()>>);
impl Test<'_> {
async fn function(&mut self) {
println!("I'm alive!");
}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(unsafe { &mut *(&mut s as *mut Self) }.function().boxed());
s
}
}
impl Future for Test<'_> {
type Output = ();
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
self.0.as_mut().unwrap().poll_unpin(cx)
}
}
I hope that there is another way.
Though there are times when you may want to do things similar to what you're trying to accomplish here, they are a rarity. So most people reading this, maybe even OP, may wish to restructure such that struct state and data used for a single async execution are different objects.
To answer your question, yes it is somewhat possible. Unless you want to absolutely resort to unsafe code you will need to use Mutex and Arc. All fields you wish to manipulate inside the async fn will have to be wrapped inside a Mutex and the function itself will accept an Arc<Self>.
I must stress, however, that this is not a beautiful solution and you probably don't want to do this. Depending on your specific case your solution may vary, but my guess of what OP is trying to accomplish while using Streams would be better solved by something similar to this gist that I wrote.
use std::{
future::Future,
pin::Pin,
sync::{Arc, Mutex},
};
struct Test {
state: Mutex<Option<Pin<Box<dyn Future<Output = ()>>>>>,
// if available use your async library's Mutex to `.await` locks on `buffer` instead
buffer: Mutex<Vec<u8>>,
}
impl Test {
async fn function(self: Arc<Self>) {
for i in 0..16u8 {
let data: Vec<u8> = vec![i]; // = fs::read(&format("file-{}.txt", i)).await.unwrap();
let mut buflock = self.buffer.lock().unwrap();
buflock.extend_from_slice(&data);
}
}
pub fn new() -> Arc<Self> {
let s = Arc::new(Self {
state: Default::default(),
buffer: Default::default(),
});
{
// start by trying to aquire a lock to the Mutex of the Box
let mut lock = s.state.lock().unwrap();
// create boxed future
let b = Box::pin(s.clone().function());
// insert value into the mutex
*lock = Some(b);
} // block causes the lock to be released
s
}
}
impl Future for Test {
type Output = ();
fn poll(
self: std::pin::Pin<&mut Self>,
ctx: &mut std::task::Context<'_>,
) -> std::task::Poll<<Self as std::future::Future>::Output> {
let mut lock = self.state.lock().unwrap();
let fut: &mut Pin<Box<dyn Future<Output = ()>>> = lock.as_mut().unwrap();
Future::poll(fut.as_mut(), ctx)
}
}
I'm not sure what you want to achieve and why, but I suspect that you're trying to implement Future for Test based on some ancient tutorial or misunderstanding and just overcomplicating things.
You don't have to implement Future manually. An async function
async fn function(...) {...}
is really just syntax sugar translated behind the scenes into something like
fn function(...) -> Future<()> {...}
All you have to do is to use the result of the function the same way as any future, e.g. use await on it or call block a reactor until it's finished. E.g. based on your first version, you can simply call:
let mut test = Test{};
test.function().await;
UPDATE1
Based on your descriptions I still think you're trying to overcomplicate this minimal working snippet without the need to manually implement Future for anything:
async fn asyncio() { println!("Doing async IO"); }
struct Test {
count: u32,
}
impl Test {
async fn function(&mut self) {
asyncio().await;
self.count += 1;
}
}
#[tokio::main]
async fn main() {
let mut test = Test{count: 0};
test.function().await;
println!("Count: {}", test.count);
}
My goal is to implement a function that returns another function, which returns some trait. To be more specific, the returned function should itself return a Future.
To return a function that returns a concrete type, we obviously can do this:
fn returns_closure() -> impl Fn(i32) -> i32 {
|x| x + 1
}
But what if instead of i32 we want to return a Future?
I tried the following:
use futures::Future;
fn factory() -> (impl Fn() -> impl Future) {
|| async {
// some async code
}
}
This does not work because the second impl keyword is not allowed:
error[E0562] `impl Trait` not allowed outside of function and inherent method return types
What is the best way to solve this issue?
I don't know of any way to do this on stable Rust. However, you can use a type alias for an opaque type (also known as existential type) on Rust nightly like this (playground):
#![feature(type_alias_impl_trait)]
use futures::Future;
type Fut<O> = impl Future<Output = O>;
fn factory<O>() -> impl Fn() -> Fut<O> {
|| async {
todo!()
}
}