Rust showing expected trait object `dyn Future`, found opaque type when passing function as a param - asynchronous

use std::io::prelude::*;
use std::net::TcpListener;
use std::net::TcpStream;
use std::time::Duration;
// pyO3 module
use pyo3::prelude::*;
use pyo3::wrap_pyfunction;
use std::future::Future;
#[pyfunction]
pub fn start_server() {
let listener = TcpListener::bind("127.0.0.1:7878").unwrap();
let pool = ThreadPool::new(4);
for stream in listener.incoming() {
let stream = stream.unwrap();
pool.execute(|| {
let rt = tokio::runtime::Runtime::new().unwrap();
handle_connection(stream, rt, &test_helper);
});
}
}
#[pymodule]
pub fn roadrunner(_: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_wrapped(wrap_pyfunction!(start_server))?;
Ok(())
}
async fn read_file(filename: String) -> String {
let con = tokio::fs::read_to_string(filename).await;
con.unwrap()
}
async fn test_helper(contents: &mut String, filename: String) {
// this function will accept custom function and return
*contents = tokio::task::spawn(read_file(filename.clone()))
.await
.unwrap();
}
pub fn handle_connection(
mut stream: TcpStream,
runtime: tokio::runtime::Runtime,
test: &dyn Fn(&mut String, String) -> (dyn Future<Output = ()> + 'static),
) {
let mut buffer = [0; 1024];
stream.read(&mut buffer).unwrap();
let get = b"GET / HTTP/1.1\r\n";
let sleep = b"GET /sleep HTTP/1.1\r\n";
let (status_line, filename) = if buffer.starts_with(get) {
("HTTP/1.1 200 OK", "hello.html")
} else if buffer.starts_with(sleep) {
thread::sleep(Duration::from_secs(5));
("HTTP/1.1 200 OK", "hello.html")
} else {
("HTTP/1.1 404 NOT FOUND", "404.html")
};
let mut contents = String::new();
let future = test_helper(&mut contents, String::from(filename));
runtime.block_on(future);
let response = format!(
"{}\r\nContent-Length: {}\r\n\r\n{}",
status_line,
contents.len(),
contents
);
stream.write(response.as_bytes()).unwrap();
stream.flush().unwrap();
}
I am trying to create a module where I need to pass an async function as an argument. I have passed the element but I am unable to deduce what should I do from the error message. It is telling me that there is some mismatch in type inference.
Here is the error message I am getting on cargo check
error[E0271]: type mismatch resolving `for<'r> <for<'_> fn(&mut String, String) -> impl Future {test_helper} as FnOnce<(&'r mut String, String)>>::Output == (dyn Future<Output = ()> + 'static)`
--> src/lib.rs:124:43
|
124 | handle_connection(stream, rt, &test_helper);
| ^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
...
140 | async fn test_helper(contents: &mut String, filename: String) {
| - checked the `Output` of this `async fn`, found opaque type
|
= note: while checking the return type of the `async fn`
= note: expected trait object `(dyn Future<Output = ()> + 'static)`
found opaque type `impl Future`
= note: required for the cast to the object type `dyn for<'r> Fn(&'r mut String, String) -> (dyn Future<Output = ()> + 'static)`
error: aborting due to previous error
Please let me know what change should be made here. Thanks in advance.

You are writing a function type that returns a dyn type, not a reference to it, but the unsized type itself, that is not possible. Every time you want to write something like this, try using a generic instead:
pub fn handle_connection<F>(
mut stream: TcpStream,
runtime: tokio::runtime::Runtime,
test: &dyn Fn(&mut String, String) -> F,
)
where F: Future<Output = ()> + 'static
This now fails with this weird error:
error[E0308]: mismatched types
--> src/lib.rs:19:43
|
19 | handle_connection(stream, rt, &test_helper);
| ^^^^^^^^^^^^ one type is more general than the other
|
= note: expected associated type `<for<'_> fn(&mut String, String) -> impl Future {test_helper} as FnOnce<(&mut String, String)>>::Output`
found associated type `<for<'_> fn(&mut String, String) -> impl Future {test_helper} as FnOnce<(&mut String, String)>>::Output`
But this is expected too, your future is holding a reference to that &mut String you are passing, so it is not 'static anymore. The solution is just to add a lifetime generic parameter:
pub fn handle_connection<'a, F>(
mut stream: TcpStream,
runtime: tokio::runtime::Runtime,
test: &dyn Fn(&'a mut String, String) -> F,
)
where F: Future<Output = ()> + 'a
And now it should compile.

Related

Why am I getting a lifetime mismatch?

The following function compiles without issues:
async fn filter_con<T, O, F>(arr: Vec<T>, predicate: F) -> Vec<T>
where
O: Future<Output = bool>,
F: for Fn(&T) -> O,
{
join_all(arr.into_iter().map(|it| async {
if predicate(&it).await {
Some(it)
} else {
None
}
}))
.await
.into_iter()
.filter_map(|p| p)
.collect::<Vec<_>>()
}
But I can't find a way to call it properly:
let items_filtered = filter_con(items, filter).await;
Error:
error[E0308]: mismatched types
--> src/lifecycle/querier.rs:101:30
|
101 | let items_filtered = filter_con(items_response, filter).await;
| ^^^^^^^^^^ lifetime mismatch
|
= note: expected associated type `<for<'_> fn(&PairInfo) -> impl futures::Future<Output = bool> {lifecycle::querier::filter} as FnOnce<(&PairInfo,)>>::Output`
found associated type `<for<'_> fn(&PairInfo) -> impl futures::Future<Output = bool> {lifecycle::querier::filter} as FnOnce<(&PairInfo,)>>::Output`
= note: the required lifetime does not necessarily outlive the empty lifetime
note: the lifetime requirement is introduced here
--> src/lifecycle/querier.rs:246:26
|
246 | F: for<'a> Fn(&T) -> O,
| ^
Why is this happening?
Full playground here
You have essentially the same problem as in here
To make it easier to understand I'll rewrite/desugar some of your code. Let's start with the predicate:
async fn filter(x: &Test) -> bool{
x.0 >= 50
}
Is functionally equivalent to:
fn filter(x: &Test) -> impl Future<Output=bool> + '_{
async {
x.0 >= 50
}
}
Now it should be obvious that the lifetime of the returned Future is not 'static, but actually it depends on the lifetime of the parameter x: &Test.
This should make the error more understandable - your predicate is returning some future with some lifetime, which is not mentioned anywhere in the generic type definition - you can see that you don't mention it when you define neither O, nor F:
where
O: Future<Output = bool>,
F: for Fn(&T) -> O,
The problem is that, currently the language does not provide any means to say that O's lifetime depends on F's lifetime. The only easy solution I know is to use the escape hatch - box the future, which will allow you to use the same lifetime in F and O. the downside is that, F will be heap allocated:
use futures::future::{join_all, Future};
use futures::future::BoxFuture;
use futures::FutureExt;
#[derive(Debug)]
struct Test(usize);
#[tokio::main]
async fn main() {
let items = vec![Test(10), Test(100), Test(1000)];
let items_filtered = filter_con(items, filter).await;
println!("{:?}", items_filtered); // should print [100, 1000]
}
fn filter(x: &Test) -> BoxFuture<'_, bool>{
async {
x.0 >= 50
}.boxed()
}
async fn filter_con<T, F>(arr: Vec<T>, predicate: F) -> Vec<T>
where
F: for<'a> Fn(&'a T) -> BoxFuture<'a, bool>,
{
join_all(arr.into_iter().map(|it| async{
if predicate(&it).await {
Some(it)
} else {
None
}
}))
.await
.into_iter()
.filter_map(|p| p)
.collect::<Vec<_>>()
}
Here is a link to the Rust Plauground

Declaring Associated Type of Trait Object in Async Function Parameter

I'd like a function which asynchronously processes a variable amount of (Sink, Stream) tuples.
use futures::channel::mpsc;
use futures::{Sink, Stream, SinkExt, StreamExt};
async fn foo(v: Vec<(Box<dyn Sink<Error = std::io::Error>>, Box<dyn Stream<Item = u8>>)>) {
for (mut tx, mut rx) in v {
let _ = tx.send(0);
let _ = rx.next().await;
}
}
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
let (tx, mut rx) = mpsc::channel(32);
foo(vec![(Box::new(tx), Box::new(rx))]).await;
Ok(())
}
But I get this compilation error:
error[E0107]: wrong number of type arguments: expected 1, found 0
--> src/main.rs:4:30
|
4 | async fn foo(v: Vec<(Box<dyn Sink<Error = std::io::Error>>, Box<dyn Stream<Item = u8>>)>) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 1 type argument
I was prompted to declare the associated type for the trait object that way by the compiler itself. I'm unsure why it does not accept it.
The compiler wants you to specify the "type argument" of the Sink. This is not the error type, but the type of the item being sent down the sink, as in Sink<Foo>. You specify u8 as the type of the stream, and are sending the value unchanged between one and the other, so you probably want a Sink<u8>.
Once you do that, the compiler will next complain that you need to specify the Error associated type (this time for real). However if you specify std::io::Error, the call to foo() from main() won't compile because the implementation of Sink for mpsc::Sender specifies its own mpsc::SendError as the error type.
Finally, both the sink and the stream need to be pinned so they can live across await points. This is done by using Pin<Box<...>> instead of Box<...> and Box::pin(...) instead of Box::new(...).
With the above changes, a version that compiles looks like this:
use futures::channel::mpsc;
use futures::{Sink, SinkExt, Stream, StreamExt};
use std::pin::Pin;
async fn foo(
v: Vec<(
Pin<Box<dyn Sink<u8, Error = mpsc::SendError>>>,
Pin<Box<dyn Stream<Item = u8>>>,
)>,
) {
for (mut tx, mut rx) in v {
let _ = tx.send(0);
let _ = rx.next().await;
}
}
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
let (tx, rx) = mpsc::channel(32);
foo(vec![(Box::pin(tx), Box::pin(rx))]).await;
Ok(())
}

Cannot use `impl Future` to store async function in a vector

I am trying to store async functions in a vector, but it seems like impl cannot be used in the vector type definition:
use std::future::Future;
fn main() {
let mut v: Vec<fn() -> impl Future<Output = ()>> = vec![];
v.push(haha);
}
async fn haha() {
println!("haha");
}
error[E0562]: `impl Trait` not allowed outside of function and inherent method return types
--> src/main.rs:4:28
|
4 | let mut v: Vec<fn() -> impl Future<Output = ()>> = vec![];
| ^^^^^^^^^^^^^^^^^^^^^^^^
How do I write the type inside the vector?
I found that there may be a workaround by using a type alias, so I changed the code:
use std::future::Future;
type Haha = impl Future<Output = ()>;
fn main() {
let mut v: Vec<fn() -> Haha> = vec![];
v.push(haha);
}
async fn haha() {
println!("haha");
}
This doesn't work either; this time the error occurs in the type alias:
error[E0658]: `impl Trait` in type aliases is unstable
--> src/main.rs:3:1
|
3 | type Haha = impl Future<Output = ()>;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: for more information, see https://github.com/rust-lang/rust/issues/63063
error[E0308]: mismatched types
--> src/main.rs:8:12
|
8 | v.push(haha);
| ^^^^ expected opaque type, found a different opaque type
|
= note: expected type `fn() -> Haha`
found type `fn() -> impl std::future::Future {haha}`
= note: distinct uses of `impl Trait` result in different opaque types
error: could not find defining uses
--> src/main.rs:3:1
|
3 | type Haha = impl Future<Output = ()>;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
How do I fix it?
You cannot use the impl Trait this way. To be able to store different types that implement a trait into the same container you have to use dynamic dispatch, by storing something like Box<dyn Trait>.
In your particular case, you do not specify if you want to store the async functions themselves or the future generated by the async functions, the solution would be somewhat different.
To store just the futures, you write a container such as:
let mut v: Vec<Box<dyn Future<Output = ()>>> = vec![];
And then just call the function, box it and store it in the container:
v.push(Box::new(haha()));
If instead you want to store the async function itself, without calling it, you need a container with a double dyn:
let mut v2: Vec<Box<dyn Fn() -> Box<dyn Future<Output = ()>>>> = vec![];
Now, since your haha function does not implement this Fn trait you need an adaptor. A lambda function will do, but don't forget the double Box:
v2.push(Box::new(|| Box::new(haha())));
Unfortunately, with these solutions you will be able to create the vector, but not to .await for your futures. For that you need the futures to implement the Unpin marker. That guarantees to the compiler that the future will not move while it is running (if it did, the implementation would be totally unsafe). You could add the + Unpin requirement to the futures, but async fn are not Unpin so you could not fill the vector. The easiest way to fix it is to use this handy function from std:
pub fn into_pin(boxed: Box<T>) -> Pin<Box<T>>
for f in v2 {
f().into_pin().await;
}
Unfortunately, it is still unstable. Fortunately, there is a From impl that does exactly the same. So you can just write:
for f in v2 {
Pin::from(f()).await;
}
In your comment below you write this code to wait for the futures:
for f in v2 {
async { f().await }
}
Note that an async block itself will evaluate to another future, so here you are just wrapping each future into another future, but nobody is waiting for that one. Actually you'll get a warning about it:
warning: unused implementer of std::future::Future that must be used.
Remember that in order to properly wait for all the futures you will need an async runtime.
rodrigo's answer is correct, but I'd prefer to use Box::pin and bake the Pin type into the API of the collection. This makes using the Future trait object (or closure trait object producing a Future trait object) easier:
use std::{future::Future, pin::Pin};
type PinFutureObj<Output> = Pin<Box<dyn Future<Output = Output>>>;
async fn collection_of_pinned_future_trait_objects() {
let v: Vec<PinFutureObj<()>> = vec![
Box::pin(haha()),
Box::pin(hehe()),
Box::pin(haha()),
Box::pin(hehe()),
];
for f in v {
f.await
}
}
async fn collection_of_closure_trait_objects() {
let v: Vec<Box<dyn Fn() -> PinFutureObj<()>>> = vec![
Box::new(|| Box::pin(haha())),
Box::new(|| Box::pin(hehe())),
Box::new(|| Box::pin(haha())),
Box::new(|| Box::pin(hehe())),
];
for f in v {
f().await
}
}
async fn haha() {
println!("haha");
}
async fn hehe() {
println!("hehe");
}
I'd also start introducing type aliases for the longer types.
In fact, this type alias already exists in the futures crate as LocalBoxFuture and can be created via FutureExt::boxed_local. There's also BoxFuture produced by FutureExt::boxed which adds common trait bounds.
use futures::future::{FutureExt, LocalBoxFuture}; // 0.3.5
async fn collection_of_pinned_future_trait_objects() {
let v: Vec<LocalBoxFuture<'static, ()>> = vec![
haha().boxed_local(),
hehe().boxed_local(),
haha().boxed_local(),
hehe().boxed_local(),
];
for f in v {
f.await
}
}
async fn collection_of_closure_trait_objects() {
let v: Vec<Box<dyn Fn() -> LocalBoxFuture<'static, ()>>> = vec![
Box::new(|| haha().boxed_local()),
Box::new(|| hehe().boxed_local()),
Box::new(|| haha().boxed_local()),
Box::new(|| hehe().boxed_local()),
];
for f in v {
f().await
}
}
async fn haha() {
println!("haha");
}
async fn hehe() {
println!("hehe");
}
See also:
How can I put an async function into a map in Rust?
Why can impl trait not be used to return multiple / conditional types?

Hyper 0.12.x : Implementing Service for a struct

In hyper 0.12.33, how do I implement hyper::service::Service for a struct ?
I have tried the following but it is not sufficient as it seems that in 0.12 the Future trait is not provided automatically anymore for a struct that implements Service:
use futures::future::Future;
use hyper::{Body, Request, Response};
struct MyStruct;
impl MyStruct {
pub fn new() -> Self {
MyStruct
}
}
impl hyper::service::Service for MyStruct {
type ReqBody = Body;
type ResBody = Body;
type Error = hyper::Error;
type Future = Box<Future<Item = Response<Body>, Error = hyper::Error>>;
fn call(&mut self, req: Request<Body>) -> Self::Future {
unimplemented!()
}
}
fn main() {
let addr = "0.0.0.0:8080".parse().unwrap();
let server = hyper::Server::bind(&addr)
.serve(|| MyStruct::new())
.map_err(|e| eprintln!("server error: {}", e));
hyper::rt::run(server);
}
gives me the build error message:
Standard Error
Compiling playground v0.0.1 (/playground)
error[E0277]: the trait bound `MyStruct: futures::future::Future` is not satisfied
--> src/main.rs:26:10
|
26 | .serve(|| MyStruct::new())
| ^^^^^ the trait `futures::future::Future` is not implemented for `MyStruct`
|
= note: required because of the requirements on the impl of `hyper::service::make_service::MakeServiceRef<hyper::server::tcp::addr_stream::AddrStream>` for `[closure#src/main.rs:26:16: 26:34]`
error[E0599]: no method named `map_err` found for type `hyper::server::Server<hyper::server::tcp::AddrIncoming, [closure#src/main.rs:26:16: 26:34]>` in the current scope
--> src/main.rs:27:10
|
27 | .map_err(|e| eprintln!("server error: {}", e));
| ^^^^^^^
|
= note: the method `map_err` exists but the following trait bounds were not satisfied:
`&mut hyper::server::Server<hyper::server::tcp::AddrIncoming, [closure#src/main.rs:26:16: 26:34]> : futures::future::Future`
`hyper::server::Server<hyper::server::tcp::AddrIncoming, [closure#src/main.rs:26:16: 26:34]> : futures::future::Future`
This example gives one way. It compiles and runs with v0.14.12
#![deny(warnings)]
use std::task::{Context, Poll};
use futures_util::future;
use hyper::service::Service;
use hyper::{Body, Request, Response, Server};
const ROOT: &str = "/";
#[derive(Debug)]
pub struct Svc;
impl Service<Request<Body>> for Svc {
type Response = Response<Body>;
type Error = hyper::Error;
type Future = future::Ready<Result<Self::Response, Self::Error>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Ok(()).into()
}
fn call(&mut self, req: Request<Body>) -> Self::Future {
let rsp = Response::builder();
let uri = req.uri();
if uri.path() != ROOT {
let body = Body::from(Vec::new());
let rsp = rsp.status(404).body(body).unwrap();
return future::ok(rsp);
}
let body = Body::from(Vec::from(&b"heyo!"[..]));
let rsp = rsp.status(200).body(body).unwrap();
future::ok(rsp)
}
}
pub struct MakeSvc;
impl<T> Service<T> for MakeSvc {
type Response = Svc;
type Error = std::io::Error;
type Future = future::Ready<Result<Self::Response, Self::Error>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Ok(()).into()
}
fn call(&mut self, _: T) -> Self::Future {
future::ok(Svc)
}
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// pretty_env_logger::init();
let addr = "127.0.0.1:1337".parse().unwrap();
let server = Server::bind(&addr).serve(MakeSvc);
println!("Listening on http://{}", addr);
server.await?;
Ok(())
}
The indirection (MakeSvc -> Src) appears to follow from the architecture of Hyper, as described in this issue:
There's two steps involved here, and both make use of Service:
The MakeSvc is a Service that creates Svcs for each connection.
The Svc is a Service to handle requests on a single connection.

How to asynchronously explore a directory and its sub-directories?

I need to explore a directory and all its sub-directories. I can explore the directory easily with recursion in a synchronous way:
use failure::Error;
use std::fs;
use std::path::Path;
fn main() -> Result<(), Error> {
visit(Path::new("."))
}
fn visit(path: &Path) -> Result<(), Error> {
for e in fs::read_dir(path)? {
let e = e?;
let path = e.path();
if path.is_dir() {
visit(&path)?;
} else if path.is_file() {
println!("File: {:?}", path);
}
}
Ok(())
}
When I try to do the same in an asynchronous manner using tokio_fs:
use failure::Error; // 0.1.6
use futures::Future; // 0.1.29
use std::path::PathBuf;
use tokio::{fs, prelude::*}; // 0.1.22
fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
let task = fs::read_dir(path)
.flatten_stream()
.for_each(|entry| {
println!("{:?}", entry.path());
let path = entry.path();
if path.is_dir() {
let task = visit(entry.path());
tokio::spawn(task.map_err(drop));
}
future::ok(())
})
.map_err(Error::from);
task
}
Playground
I get the following error:
error[E0391]: cycle detected when processing `visit::{{opaque}}#0`
--> src/lib.rs:6:28
|
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: ...which requires processing `visit`...
--> src/lib.rs:6:1
|
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: ...which requires evaluating trait selection obligation `futures::future::map_err::MapErr<impl futures::future::Future, fn(failure::error::Error) {std::mem::drop::<failure::error::Error>}>: std::marker::Send`...
= note: ...which again requires processing `visit::{{opaque}}#0`, completing the cycle
note: cycle used when checking item types in top-level module
--> src/lib.rs:1:1
|
1 | / use failure::Error; // 0.1.6
2 | | use futures::Future; // 0.1.29
3 | | use std::path::PathBuf;
4 | | use tokio::{fs, prelude::*}; // 0.1.22
... |
20| | task
21| | }
| |_^
error[E0391]: cycle detected when processing `visit::{{opaque}}#0`
--> src/lib.rs:6:28
|
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: ...which requires processing `visit`...
--> src/lib.rs:6:1
|
6 | fn visit(path: PathBuf) -> impl Future<Item = (), Error = Error> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= note: ...which again requires processing `visit::{{opaque}}#0`, completing the cycle
note: cycle used when checking item types in top-level module
--> src/lib.rs:1:1
|
1 | / use failure::Error; // 0.1.6
2 | | use futures::Future; // 0.1.29
3 | | use std::path::PathBuf;
4 | | use tokio::{fs, prelude::*}; // 0.1.22
... |
20| | task
21| | }
| |_^
What is the correct way of exploring a directory and its sub-directories asynchronously while propagating all the errors?
I would make several modifications to rodrigo's existing answer:
Return a Stream from the function, allowing the caller to do what they need with a given file entry.
Return an impl Stream instead of a Box<dyn Stream>. This leaves room for more flexibility in implementation. For example, a custom type could be created that uses an internal stack instead of the less-efficient recursive types.
Return io::Error from the function to allow the user to deal with any errors.
Accept a impl Into<PathBuf> to allow a nicer API.
Create an inner hidden implementation function that uses concrete types in its API.
Futures 0.3 / Tokio 0.2
In this version, I avoided the deeply recursive calls, keeping a local stack of paths to visit (to_visit).
use futures::{stream, Stream, StreamExt}; // 0.3.1
use std::{io, path::PathBuf};
use tokio::fs::{self, DirEntry}; // 0.2.4
fn visit(path: impl Into<PathBuf>) -> impl Stream<Item = io::Result<DirEntry>> + Send + 'static {
async fn one_level(path: PathBuf, to_visit: &mut Vec<PathBuf>) -> io::Result<Vec<DirEntry>> {
let mut dir = fs::read_dir(path).await?;
let mut files = Vec::new();
while let Some(child) = dir.next_entry().await? {
if child.metadata().await?.is_dir() {
to_visit.push(child.path());
} else {
files.push(child)
}
}
Ok(files)
}
stream::unfold(vec![path.into()], |mut to_visit| {
async {
let path = to_visit.pop()?;
let file_stream = match one_level(path, &mut to_visit).await {
Ok(files) => stream::iter(files).map(Ok).left_stream(),
Err(e) => stream::once(async { Err(e) }).right_stream(),
};
Some((file_stream, to_visit))
}
})
.flatten()
}
#[tokio::main]
async fn main() {
let root_path = std::env::args().nth(1).expect("One argument required");
let paths = visit(root_path);
paths
.for_each(|entry| {
async {
match entry {
Ok(entry) => println!("visiting {:?}", entry),
Err(e) => eprintln!("encountered an error: {}", e),
}
}
})
.await;
}
Futures 0.1 / Tokio 0.1
use std::path::PathBuf;
use tokio::{fs, prelude::*}; // 0.1.22
use tokio_fs::DirEntry; // 1.0.6
fn visit(
path: impl Into<PathBuf>,
) -> impl Stream<Item = DirEntry, Error = std::io::Error> + Send + 'static {
fn visit_inner(
path: PathBuf,
) -> Box<dyn Stream<Item = DirEntry, Error = std::io::Error> + Send + 'static> {
Box::new({
fs::read_dir(path)
.flatten_stream()
.map(|entry| {
let path = entry.path();
if path.is_dir() {
// Optionally include `entry` if you want to
// include directories in the resulting
// stream.
visit_inner(path)
} else {
Box::new(stream::once(Ok(entry)))
}
})
.flatten()
})
}
visit_inner(path.into())
}
fn main() {
tokio::run({
let root_path = std::env::args().nth(1).expect("One argument required");
let paths = visit(root_path);
paths
.then(|entry| {
match entry {
Ok(entry) => println!("visiting {:?}", entry),
Err(e) => eprintln!("encountered an error: {}", e),
};
Ok(())
})
.for_each(|_| Ok(()))
});
}
See also:
How do I synchronously return a value calculated in an asynchronous Future in stable Rust?
Your code has two errors:
First, a function returning impl Trait cannot currently be recursive, because the actual type returned would depend on itself.
To make your example work, you need to return a sized type. The simple candidate is a trait object, that is, a Box<dyn Future<...>>:
fn visit(path: PathBuf) -> Box<dyn Future<Item = (), Error = Error>> {
// ...
let task = visit(entry.path());
tokio::spawn(task.map_err(drop));
// ...
Box::new(task)
}
There is still your second error:
error[E0277]: `dyn futures::future::Future<Item = (), Error = failure::error::Error>` cannot be sent between threads safely
--> src/lib.rs:14:30
|
14 | tokio::spawn(task.map_err(drop));
| ^^^^^^^^^^^^^^^^^^ `dyn futures::future::Future<Item = (), Error = failure::error::Error>` cannot be sent between threads safely
|
::: /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/executor/mod.rs:131:52
|
131 | where F: Future<Item = (), Error = ()> + 'static + Send
| ---- required by this bound in `tokio::executor::spawn`
|
= help: the trait `std::marker::Send` is not implemented for `dyn futures::future::Future<Item = (), Error = failure::error::Error>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<dyn futures::future::Future<Item = (), Error = failure::error::Error>>`
= note: required because it appears within the type `std::boxed::Box<dyn futures::future::Future<Item = (), Error = failure::error::Error>>`
= note: required because it appears within the type `futures::future::map_err::MapErr<std::boxed::Box<dyn futures::future::Future<Item = (), Error = failure::error::Error>>, fn(failure::error::Error) {std::mem::drop::<failure::error::Error>}>`
This means that your trait object is not Send so it cannot be scheduled for execution in another thread using tokio::spawn(). Fortunately, this is easy to fix: just add + Send to your trait object:
fn visit(path: PathBuf) -> Box<dyn Future<Item = (), Error = Error> + Send> {
//...
}
See the full code in the Playground.

Resources