Running asynchronous mutable operations with Rust futures - asynchronous

I'm building a service in Rust using tokio-rs and was happy with this tech stack so far. I'm now trying to chain up async operations that includes writes and having a hard time with the borrow checker.
My simplified minimal code sample is this:
extern crate futures; // 0.1.21
use futures::Future;
use std::{cell::RefCell, rc::Rc};
trait RequestProcessor {
fn prepare(&self) -> Box<Future<Item = (), Error = ()>>;
fn process(&mut self, request: String) -> Box<Future<Item = (), Error = ()>>;
}
struct Service {
processor: Rc<RefCell<RequestProcessor>>,
}
impl Service {
fn serve(&mut self, request: String) -> Box<Future<Item = (), Error = ()>> {
let processor_clone = self.processor.clone();
let result_fut = self
.processor
.borrow()
.prepare()
.and_then(move |_| processor_clone.borrow_mut().process(request));
Box::new(result_fut)
}
}
fn main() {}
As a short summary, after an async preparation step I'm trying to run another async operation that writes a field of self. Without mutability this works easily with a plain Rc member, but mutability breaks it, producing the following error:
error[E0597]: `processor_clone` does not live long enough
--> src/main.rs:22:32
|
22 | .and_then(move |_| processor_clone.borrow_mut().process(request));
| ^^^^^^^^^^^^^^^ - `processor_clone` dropped here while still borrowed
| |
| borrowed value does not live long enough
|
= note: values in a scope are dropped in the opposite order they are created
I'd expect this should work, I don't see where the mutable reference is still borrowed. I think process() should release the processor's &mut self after the future is returned, so the compile error shouldn't occur.
Could you please explain the underlying reasons? How should this example be changed to be accepted by the compiler?

Sometimes, it's easier to see lifetime errors if you split values up onto multiple lines. Let's try that with the closure in question:
.and_then(move |_| {
let c = processor_clone;
let mut d = c.borrow_mut();
let e = d.process(request);
e
});
If you compile this... it works. If we try recombining the lines, we can get this to fail:
.and_then(move |_| {
let c = processor_clone;
c.borrow_mut().process(request)
});
And this to work:
.and_then(move |_| {
let c = processor_clone;
return c.borrow_mut().process(request);
});
The only difference is the explicit return and the semicolon. This is very similar to When returning the outcome of consuming a StdinLock, why was the borrow to stdin retained?, so let's try the suggestion of one of the answers to enable non-lexical lifetimes. This allows your original code to compile as well.
TL;DR: It's a weakness in Rust's current borrow checker implementation and will be magically fixed at some point. In the meantime, I suggest writing it as two lines:
.and_then(move |_| {
let mut c = processor_clone.borrow_mut();
c.process(request)
});

Related

Rust Async Borrow Lifetimes

I'm trying to make a helper that allows asynchronously chaining side effects, but I'm unable to get the generic bounds correct so that the compiler understands the output of the future outlives a reference used to build it.
Playground Link
The gist of it comes down to:
struct Chain<T> {
data: T
}
impl<T> Chain<T> {
pub async fn chain<E, Fut, F>(self, effect: F) -> Result<T, E>
where
Fut: Future<Output=Result<(), E>>,
F: FnOnce(&T) -> Fut
{
todo!()
}
}
gives a compiler error of
error: lifetime may not live long enough
--> src/main.rs:39:32
|
39 | let r = chain.chain(|this| this.good("bar")).await;
| ----- ^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
| | |
| | return type of closure `impl Future` contains a lifetime `'2`
| has type `&'1 MyData`
If we fix up chain so that it can infer that the reference is available for the same lifetime as the future:
impl<T> Chain<T> {
pub async fn chain<'a, E, Fut, F>(self, effect: F) -> Result<T, E>
where
T: 'a,
Fut: 'a + Future<Output=Result<(), E>>,
F: FnOnce(&'a T) -> Fut
{
effect(&self.data).await?;
Ok(self.data)
}
}
We get a new compiler error about moving self.data while it's still borrowed.
error[E0505]: cannot move out of `self.data` because it is borrowed
--> src/main.rs:30:12
|
23 | pub async fn chain<'a, E, Fut, F>(self, effect: F) -> Result<T, E>
| -- lifetime `'a` defined here
...
29 | effect(&self.data).await?;
| ------------------
| | |
| | borrow of `self.data` occurs here
| argument requires that `self.data` is borrowed for `'a`
30 | Ok(self.data)
| ^^^^^^^^^ move out of `self.data` occurs here
I guess there's a pathological closure along the lines of |this| futures::future::ready(Err(this)) that would cause an early return with the borrow still "alive".
Question
How can we get chain to work? My normal lifetime trick of block-scoping doesn't seem to help. Is there a set of where constraints that can be added to prove that the borrow and then eventual move are on disjoint lifetimes?
This particular situation is one where the current constraint syntax and lack of higher-kinded types does not let you express what you want.
You can use a higher-rank trait bound, the for<'a> syntax, to introduce an intermediate generic lifetime parameter 'a within a where clause to dictate that the constraint must be valid for any lifetime. This is necessary here and the reason your first fix didn't work was because 'a as a generic on chain meant that the lifetime was determined by the caller, however, lifetime of self is by construction less than any lifetime that could be picked by the caller. So the slightly more correct syntax (and identical to the de-sugared original code) would be:
pub async fn chain<E, Fut, F>(self, effect: F) -> Result<T, E>
where
Fut: Future<Output = Result<(), E>>,
F: for<'a> FnOnce(&'a T) -> Fut
{
...
But this doesn't help at all, since there is still no association between Fut and 'a. There's unfortunately no way to use the same for<'a> across multiple constraints. You could try using impl Trait to define it all at once, but that isn't supported:
pub async fn chain<E, F>(self, effect: F) -> Result<T, E>
where F: for<'a> FnOnce(&'a T) -> (impl Future<Output = Result<(), E>> + 'a)
{
...
error[E0562]: `impl Trait` not allowed outside of function and method return types
--> src/lib.rs:35:44
|
35 | where F: for<'a> FnOnce(&'a T) -> (impl Future<Output = Result<(), E>> + 'a)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There will hopefully be better support for higher-kinded types in the future. This particular case might have a solution on nightly by using the almost-complete generic associated types feature, but I haven't yet found it.
So the only real fix then is to use a named type as the return value, which really only leaves us with trait objects:
use std::pin::Pin;
use futures::future::FutureExt;
pub async fn chain<E, F>(self, effect: F) -> Result<T, E>
where F: for<'a> FnOnce(&'a T) -> Pin<Box<dyn Future<Output = Result<(), E>> + 'a>>
{
...
let r = chain.chain(|this| this.good("bar").boxed()).await;
As a side note, your bad case still does not compile and indeed cannot work, since you'd be returning a reference to a local value.
It looks like you are trying to implement future.then()
If you are aware of that and you are doing it as an exercise, you probably should design it in a way that the effect method returns values, and use these values to return from chain method. That way you enforce proper order of operations. As far as I understand your design, you do not benefit from awaiting on effect inside the chain method, since as your skip function is also async and will return future (the actual return type of chain method is Future<Output=Result<T, E>>, since async works that way: it wraps your explicit return type in future).
So there is no point in awaiting on the effect inside the chain, you still have to await it whenever you use it - and nothing will happen until you actually await for it outside of the chain - futures are lazy that way.
TL;DR I would arange your effect methods to return values and arrange chain to just return these values

How could the result of Arc::clone have a 'static lifetime?

Let's begin with a canonical example of Arc
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let msg = Arc::new(Mutex::new(String::new()));
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = Arc::clone(&msg);
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This compiles and runs as expected. Then I realized maybe the Mutex doesn't have to live on the heap and started wondering if I can get rid of Arc and just use a shared reference to a Mutex allocated on the stack. Here is my attempt
use std::sync::Mutex;
use std::thread;
fn main() {
let msg = Mutex::new(String::new());
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = &msg;
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This one doesn't compile, though
error[E0597]: `msg` does not live long enough
--> src/main.rs:8:25
|
8 | let local_msg = &msg;
| ^^^^ borrowed value does not live long enough
9 | handles.push(thread::spawn(move || {
| ______________________-
10 | | let mut locked = local_msg.lock().unwrap();
11 | | locked.push_str("hello, world\n");
12 | | }));
| |__________- argument requires that `msg` is borrowed for `'static`
...
20 | }
| - `msg` dropped here while still borrowed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0597`.
error: could not compile `hello`
To learn more, run the command again with --verbose.
The compiler complains that local_msg doesn't have a 'static lifetime. Well, it doesn't, so the error makes sense. However, this implies the variable let local_msg = Arc::clone(&msg); in the first snippet has 'static lifetime, otherwise I should get a similar error.
Questions:
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
As a bonus, what about other heap-backed smart pointers like Box and Rc? Do they all have a 'static lifetime because the borrow checker ensures that as long as these pointers are visible, then the addresses they point to are always valid?
The thing the compiler is looking for is a lifetime bound. A lifetime bound of 'a doesn't mean “this type is a reference with lifetime 'a”, but rather “all of the references this type contains have lifetimes of at least 'a”.
(When a lifetime bound is written explicitly, it looks like where T: 'a.)
Thus, any type which does not contain any references (or rather, has no lifetime parameters) automatically satisfies the 'static lifetime bound. If T: 'static, then Arc<T>: 'static (and the same for Box and Rc).
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
It does not point to the value using a reference, so it's fine. The type of your value is Arc<Mutex<String>>; there are no lifetime parameters here because there are no references. If it were, hypothetically, Arc<'a, Mutex<String>> (a lifetime parameter which Arc doesn't actually have), then that type would not satisfy the bound.
The job of Arc (or Rc or Box) is to own the value it points to. Ownership is not a reference and thus not subject to lifetimes.
However, if you had the type Arc<Mutex<&'a str>> then that would not satisfy the bound, because it contains a reference which is not 'static.

Lifetime error using traits and async function on protobuffers

I'm having some issues understanding lifetimes in Rust. It may also be the way I implement my design.
error[E0597]: `request` does not live long enough
--> src/service/session/server.rs:25:23
|
25 | let msg_ref = request.get_ref();
| ^^^^^^^ borrowed value does not live long enough
...
32 | let body: Box<dyn Body> = Box::new(signup);
| ---------------- cast requires that `request` is borrowed for `'static`
...
44 | }
| - `request` dropped here while still borrowed
The main source:
#[tonic::async_trait]
impl Session for SessionImplementation {
async fn signup(
&self,
request: Request<SignupRequest>,
) -> Result<Response<SessionResponse>, Status> {
let msg_ref = request.get_ref();
let signup = TxSignup::new(&msg_ref.name, &msg_ref.addr, &msg_ref.pwd);
let body: Box<dyn Body> = Box::new(signup);
let tx = Transaction::new(body);
let mut tx_signup: Box<dyn Tx> = Box::new(tx);
tx_signup.execute();
let response = SessionResponse {
deadline: 0,
cookie: "".to_string(),
status: 0,
};
Ok(Response::new(response))
}
/* more code */
}
Background
The idea is to have a Transaction, that implements Tx { execute(&self), result(&self) ... };. This Transaction has a parameter body of the type Box<dyn Box>, being the trait Body { /*some fn*/ }. Having this, I'm pretending to implement some kind of hierarchy.
The code above
On line 24 I'm getting some requests of the type SignupRequest (from proto file). This is the implementation of proto's server, using Tonic and Tokio.
After this, I have also an object TxSignup with some parameters of the type &str (set in from line 27 till 29). TxSignup implements the Body trait so I'm able to turn it into a Tx trait, apparently. The Transaction object wraps a Body implementation. I call the execute() function from the given trait Tx. All that explained has been done from line 32 till 35.
The problem
If I replace the &str type from TxSignup by type String it works. However, if I want them to be of the type &str, a lot of "incongruencies" with lifetimes emerge. I want it to be &str because none of these values will change. I think it is better to keep them on the stack instead of in the heap. Am I wrong?
If I want &str, I'm coerced to define TxSignup with <'a>, and here is where I get lost. I get why a lifetime is required, but not why all these problems appear.
As far I do understand, all elements inside the function should have the same lifetime, being killed at the end of its block (line 44). I will never send them outside.
I have tried giving to Body trait also an <'a>, and even to Tx trait (meaning the Transaction object must have one too to match the trait).
Is there any way to make it work? Am I misunderstanding the Trait use and how they work, or this patter design will never work?
Reproduction on GitHub
I have reproduced this same error in my rust-proto repository. Running cargo run should be enough.
I come from Go development and some C++, Java and Python, so I have a way of coding that may not be the most appropriate one using Rust. That's what I want to solve.

How to call to_socket_addrs on a collection of addresses?

How do I call to_socket_addrs() on an array or vector in Rust? The following code does not compile:
extern crate ws;
use std::net::ToSocketAddrs;
fn main() {
let addrs = ["127.0.0.1:8889", "127.0.0.1:0"]
.iter()
.flat_map(|a| a.to_socket_addrs());
ws::listen(addrs, |out| move |msg| out.send(msg));
}
error[E0277]: the trait bound `std::iter::FlatMap<std::slice::Iter<'_, &str>, std::result::Result<std::vec::IntoIter<std::net::SocketAddr>, std::io::Error>, [closure#src/main.rs:7:19: 7:42]>: std::net::ToSocketAddrs` is not satisfied
--> src/main.rs:8:5
|
8 | ws::listen(addrs, |out| move |msg| out.send(msg));
| ^^^^^^^^^^ the trait `std::net::ToSocketAddrs` is not implemented for `std::iter::FlatMap<std::slice::Iter<'_, &str>, std::result::Result<std::vec::IntoIter<std::net::SocketAddr>, std::io::Error>, [closure#src/main.rs:7:19: 7:42]>`
|
= note: required by `ws::listen`
It took me two hours to understand the error because "trait X is not implemented for std::iter::FlatMap" reads like the problem is caused by flat_map, instead of the demands of ws::listen.
to_socket_addrs returns a Result containing an iterator. Result itself implements IntoIterator, so flat_map flattens the result, but not the inner iterator, you need to do that in an extra step:
use std::net::ToSocketAddrs;
fn main() {
let addrs = ["127.0.0.1:8889", "127.0.0.1:0"]
.iter()
.flat_map(|a| a.to_socket_addrs())
.flatten()
.collect::<Vec<_>>();
println!("addrs = {:?}!", addrs);
}

Sharing mutable state between clients using async (tokio) rust-websocket

I am writing a websocket server in Rust using rust-websocket and its Tokio-based async system. I can serve clients just fine, however, I can not figure out how to share mutable state between the clients. Here is some (partial) code demonstrating this issue:
let mut core = Core::new().unwrap();
let handle = core.handle();
let server = Server::bind("localhost:62831", &handle).unwrap();
let mut state = State{
...
};
let f = server.incoming()
.map_err(|InvalidConnection {error, ..}| error)
.for_each(|upgrade, _)| {
let f = upgrade.accept()
.and_then(|s, _| {
let ctx = ClientContext{
// some other per-client values
state: &mut state,
}
...
return s.send(Message::binary(data).into())
.and_then(move |s| Ok(s, ctx)); // this could be the complete wrong way to insert context into the stream
}).and_then(|s, ctx| {
// client handling code here
});
handle.spawn(f
.map_err(...)
.map(...)
);
return Ok(())
});
core.run(f).unwrap();
This code errors with this:
error[E0373]: closure may outlive the current function, but it borrows `**state`, which is owned by the current function
--> src/main.rs:111:27
|
111 | .and_then(|(s, _)| {
| ^^^^^^^^ may outlive borrowed value `**state`
...
114 | state: &mut state,
| ----- `**state` is borrowed here
|
help: to force the closure to take ownership of `**state` (and any other referenced variables), use the `move` keyword, as shown:
| .and_then(move |(s, _)| {
When trying the compiler's suggestion, I get this:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/main.rs:111:27
|
111 | .and_then(move |(s, _)| {
| ^^^^^^^^^^^^^ cannot move out of captured outer variable in an `FnMut` closure
error: `state` does not live long enough
--> src/main.rs:114:37
|
114 | state: &mut state,
| ^^^^^ does not live long enough
...
122 | })
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
I also tried wrapping the state in a RefCell (creating the RefCell right after the state itself), however, the compiler gives a similar move error since it tries to move the RefCell into the closure that creates the client context.
You're pretty close with the RefCell. What you need now is an Rc to wrap that RefCell so you can clone the Rc and not capture the RefCell itself.
let shared_state = Rc::new(RefCell::new(State::new())));
incoming().for_each(move |s, _| {
let shared_state = shared_state.clone(); // Left uncaptured
shared_state.borrow_mut().do_mutable_state_stuff(); // Could panic
});
Note that since you're using Rc's and RefCell's now, you'll likely need to go ahead and convert your ClientContext struct to storing an Rc> instead of a &mut State. It may be possible to keep using &mut State's for some things, but your &mut State's will be tied to the lifetime of the RefMut, and if you keep it alive until the next closure runs, the borrows will panic (or fail if you use the try_ variants).
Also keep in mind if you decide you want to have multiple threads in your reactor, you will just need to change Rc to Arc, and RefCell to Mutex, which is a very natural conversion when it's needed.

Resources