Sharing mutable state between clients using async (tokio) rust-websocket - asynchronous

I am writing a websocket server in Rust using rust-websocket and its Tokio-based async system. I can serve clients just fine, however, I can not figure out how to share mutable state between the clients. Here is some (partial) code demonstrating this issue:
let mut core = Core::new().unwrap();
let handle = core.handle();
let server = Server::bind("localhost:62831", &handle).unwrap();
let mut state = State{
...
};
let f = server.incoming()
.map_err(|InvalidConnection {error, ..}| error)
.for_each(|upgrade, _)| {
let f = upgrade.accept()
.and_then(|s, _| {
let ctx = ClientContext{
// some other per-client values
state: &mut state,
}
...
return s.send(Message::binary(data).into())
.and_then(move |s| Ok(s, ctx)); // this could be the complete wrong way to insert context into the stream
}).and_then(|s, ctx| {
// client handling code here
});
handle.spawn(f
.map_err(...)
.map(...)
);
return Ok(())
});
core.run(f).unwrap();
This code errors with this:
error[E0373]: closure may outlive the current function, but it borrows `**state`, which is owned by the current function
--> src/main.rs:111:27
|
111 | .and_then(|(s, _)| {
| ^^^^^^^^ may outlive borrowed value `**state`
...
114 | state: &mut state,
| ----- `**state` is borrowed here
|
help: to force the closure to take ownership of `**state` (and any other referenced variables), use the `move` keyword, as shown:
| .and_then(move |(s, _)| {
When trying the compiler's suggestion, I get this:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/main.rs:111:27
|
111 | .and_then(move |(s, _)| {
| ^^^^^^^^^^^^^ cannot move out of captured outer variable in an `FnMut` closure
error: `state` does not live long enough
--> src/main.rs:114:37
|
114 | state: &mut state,
| ^^^^^ does not live long enough
...
122 | })
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
I also tried wrapping the state in a RefCell (creating the RefCell right after the state itself), however, the compiler gives a similar move error since it tries to move the RefCell into the closure that creates the client context.

You're pretty close with the RefCell. What you need now is an Rc to wrap that RefCell so you can clone the Rc and not capture the RefCell itself.
let shared_state = Rc::new(RefCell::new(State::new())));
incoming().for_each(move |s, _| {
let shared_state = shared_state.clone(); // Left uncaptured
shared_state.borrow_mut().do_mutable_state_stuff(); // Could panic
});
Note that since you're using Rc's and RefCell's now, you'll likely need to go ahead and convert your ClientContext struct to storing an Rc> instead of a &mut State. It may be possible to keep using &mut State's for some things, but your &mut State's will be tied to the lifetime of the RefMut, and if you keep it alive until the next closure runs, the borrows will panic (or fail if you use the try_ variants).
Also keep in mind if you decide you want to have multiple threads in your reactor, you will just need to change Rc to Arc, and RefCell to Mutex, which is a very natural conversion when it's needed.

Related

Rust: future factory method results in compiler error: value requires that `'1` must outlive `'2`

I want to repeat a certain user-provided function multiple times in an async context. Thus, I thought about a non-async closure that produces futures which can be consumed/awaited.
A minimal reproducible example is the following:
let counter = Arc::new(AtomicU64::new(0));
let counter_closure = counter.clone();
let future_producer = move || async {
counter_closure.fetch_add(1, Ordering::SeqCst);
};
but this results in this error
562 | let future_producer = move || async {
| _______________________________-------_^
| | | |
| | | return type of closure `impl Future<Output = ()>` contains a lifetime `'2`
| | lifetime `'1` represents this closure's body
563 | | counter_closure.fetch_add(1, Ordering::SeqCst);
564 | | };
| |_________^ returning this value requires that `'1` must outlive `'2`
How can I solve this problem?
The problem is that the closure owns counter_closure but the futures returned by the closure only reference this value owned by the closure. The signature of closures doesn't have a way to express this. It's akin to the "owning iterator" problem -- like how the Iterator trait doesn't permit expression of the case where the iterator dispenses references tied to its own lifetime, the family of closure traits that can be invoked on a reference to a closure (Fn and FnMut) don't have a way to express that the value returned from the closure borrows from the closure itself.
If the compiler permitted this to compile, it would be possible to create a use-after-free situation by dropping the closure while futures it has previously returned still exist. This would destroy counter_closure while it's still borrowed.
You can fix this by cloning the closure-owned counter_closure on each invocation of the closure before the async block, and moving this clone into the future with async move:
let counter = Arc::new(AtomicU64::new(0));
let counter_closure = counter.clone();
let future_producer = move || {
let counter_inner = counter_closure.clone();
async move {
counter_inner.fetch_add(1, Ordering::SeqCst);
}
};
This uncouples the lifetime of the future from the closure by giving the future its own Arc.

Rust lifetimes in async wrapper for sync code

I am trying to create a Stream using a camera with a blocking capture method. The blocking call is wrapped with blocking::unblock.
use futures::stream;
use rscam::{Camera, Config};
fn frame_stream() -> impl stream::Stream {
let mut camera = Camera::new("/dev/video0").unwrap();
camera.start(&Config {
interval: (1, 30),
resolution: (1280, 720),
format: b"H264",
..Default::default()
}).unwrap();
stream::unfold(camera, |c| async move {
let frame = blocking::unblock(|| c.capture().unwrap()).await;
Some((frame, c))
})
}
Compiling gives this error message:
error[E0373]: closure may outlive the current function, but it borrows `c`, which is owned by the current function
--> src/lib.rs:15:39
|
15 | let frame = blocking::unblock(|| c.capture().unwrap()).await;
| ^^ - `c` is borrowed here
| |
| may outlive borrowed value `c`
|
note: function requires argument type to outlive `'static`
--> src/lib.rs:15:21
|
15 | let frame = blocking::unblock(|| c.capture().unwrap()).await;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: to force the closure to take ownership of `c` (and any other referenced variables), use the `move` keyword
|
15 | let frame = blocking::unblock(move || c.capture().unwrap()).await;
| ++++
error[E0505]: cannot move out of `c` because it is borrowed
--> src/lib.rs:17:22
|
15 | let frame = blocking::unblock(|| c.capture().unwrap()).await;
| ------------------------------------------
| | | |
| | | borrow occurs due to use in closure
| | borrow of `c` occurs here
| argument requires that `c` is borrowed for `'static`
16 |
17 | Some((frame, c))
| ^ move out of `c` occurs here
How can I guarantee to the compiler that the reference to c taken in the closure will still be valid? I assume it will be, since execution of the closure is awaited before c is returned.
Solution
stream::unfold(camera, |c| async move {
Some(
blocking::unblock(|| {
(c.capture().unwrap(), c)
}).await
)
})
You could move the camera into the inner closure, then return it once the frame capture is complete:
stream::unfold(camera, |c| async move {
Some(blocking::unblock(|| move {
let frame = c.capture().unwrap()).await;
(frame,c)
})
})
.await does not guarantee liveness. This is the general problem of scoped async tasks. Futures can be canceled at any time. Consider:
let future = async {
let local = 123;
blocking::unblock(|| local).await;
};
// Poll `future`, but only once.
futures::block_on(async move { futures::poll!(future) });
We started a task using local data, then dropped the future. The task continues executing but the local data is gone. For this reason, there is currently no sound way to expose an async API allowing using local data similar to scoped threads. You have to use 'static data, for example by wrapping in Arc.
See also blocking issue #4.

How could the result of Arc::clone have a 'static lifetime?

Let's begin with a canonical example of Arc
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let msg = Arc::new(Mutex::new(String::new()));
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = Arc::clone(&msg);
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This compiles and runs as expected. Then I realized maybe the Mutex doesn't have to live on the heap and started wondering if I can get rid of Arc and just use a shared reference to a Mutex allocated on the stack. Here is my attempt
use std::sync::Mutex;
use std::thread;
fn main() {
let msg = Mutex::new(String::new());
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = &msg;
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This one doesn't compile, though
error[E0597]: `msg` does not live long enough
--> src/main.rs:8:25
|
8 | let local_msg = &msg;
| ^^^^ borrowed value does not live long enough
9 | handles.push(thread::spawn(move || {
| ______________________-
10 | | let mut locked = local_msg.lock().unwrap();
11 | | locked.push_str("hello, world\n");
12 | | }));
| |__________- argument requires that `msg` is borrowed for `'static`
...
20 | }
| - `msg` dropped here while still borrowed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0597`.
error: could not compile `hello`
To learn more, run the command again with --verbose.
The compiler complains that local_msg doesn't have a 'static lifetime. Well, it doesn't, so the error makes sense. However, this implies the variable let local_msg = Arc::clone(&msg); in the first snippet has 'static lifetime, otherwise I should get a similar error.
Questions:
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
As a bonus, what about other heap-backed smart pointers like Box and Rc? Do they all have a 'static lifetime because the borrow checker ensures that as long as these pointers are visible, then the addresses they point to are always valid?
The thing the compiler is looking for is a lifetime bound. A lifetime bound of 'a doesn't mean “this type is a reference with lifetime 'a”, but rather “all of the references this type contains have lifetimes of at least 'a”.
(When a lifetime bound is written explicitly, it looks like where T: 'a.)
Thus, any type which does not contain any references (or rather, has no lifetime parameters) automatically satisfies the 'static lifetime bound. If T: 'static, then Arc<T>: 'static (and the same for Box and Rc).
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
It does not point to the value using a reference, so it's fine. The type of your value is Arc<Mutex<String>>; there are no lifetime parameters here because there are no references. If it were, hypothetically, Arc<'a, Mutex<String>> (a lifetime parameter which Arc doesn't actually have), then that type would not satisfy the bound.
The job of Arc (or Rc or Box) is to own the value it points to. Ownership is not a reference and thus not subject to lifetimes.
However, if you had the type Arc<Mutex<&'a str>> then that would not satisfy the bound, because it contains a reference which is not 'static.

Lifetime error using traits and async function on protobuffers

I'm having some issues understanding lifetimes in Rust. It may also be the way I implement my design.
error[E0597]: `request` does not live long enough
--> src/service/session/server.rs:25:23
|
25 | let msg_ref = request.get_ref();
| ^^^^^^^ borrowed value does not live long enough
...
32 | let body: Box<dyn Body> = Box::new(signup);
| ---------------- cast requires that `request` is borrowed for `'static`
...
44 | }
| - `request` dropped here while still borrowed
The main source:
#[tonic::async_trait]
impl Session for SessionImplementation {
async fn signup(
&self,
request: Request<SignupRequest>,
) -> Result<Response<SessionResponse>, Status> {
let msg_ref = request.get_ref();
let signup = TxSignup::new(&msg_ref.name, &msg_ref.addr, &msg_ref.pwd);
let body: Box<dyn Body> = Box::new(signup);
let tx = Transaction::new(body);
let mut tx_signup: Box<dyn Tx> = Box::new(tx);
tx_signup.execute();
let response = SessionResponse {
deadline: 0,
cookie: "".to_string(),
status: 0,
};
Ok(Response::new(response))
}
/* more code */
}
Background
The idea is to have a Transaction, that implements Tx { execute(&self), result(&self) ... };. This Transaction has a parameter body of the type Box<dyn Box>, being the trait Body { /*some fn*/ }. Having this, I'm pretending to implement some kind of hierarchy.
The code above
On line 24 I'm getting some requests of the type SignupRequest (from proto file). This is the implementation of proto's server, using Tonic and Tokio.
After this, I have also an object TxSignup with some parameters of the type &str (set in from line 27 till 29). TxSignup implements the Body trait so I'm able to turn it into a Tx trait, apparently. The Transaction object wraps a Body implementation. I call the execute() function from the given trait Tx. All that explained has been done from line 32 till 35.
The problem
If I replace the &str type from TxSignup by type String it works. However, if I want them to be of the type &str, a lot of "incongruencies" with lifetimes emerge. I want it to be &str because none of these values will change. I think it is better to keep them on the stack instead of in the heap. Am I wrong?
If I want &str, I'm coerced to define TxSignup with <'a>, and here is where I get lost. I get why a lifetime is required, but not why all these problems appear.
As far I do understand, all elements inside the function should have the same lifetime, being killed at the end of its block (line 44). I will never send them outside.
I have tried giving to Body trait also an <'a>, and even to Tx trait (meaning the Transaction object must have one too to match the trait).
Is there any way to make it work? Am I misunderstanding the Trait use and how they work, or this patter design will never work?
Reproduction on GitHub
I have reproduced this same error in my rust-proto repository. Running cargo run should be enough.
I come from Go development and some C++, Java and Python, so I have a way of coding that may not be the most appropriate one using Rust. That's what I want to solve.

Running asynchronous mutable operations with Rust futures

I'm building a service in Rust using tokio-rs and was happy with this tech stack so far. I'm now trying to chain up async operations that includes writes and having a hard time with the borrow checker.
My simplified minimal code sample is this:
extern crate futures; // 0.1.21
use futures::Future;
use std::{cell::RefCell, rc::Rc};
trait RequestProcessor {
fn prepare(&self) -> Box<Future<Item = (), Error = ()>>;
fn process(&mut self, request: String) -> Box<Future<Item = (), Error = ()>>;
}
struct Service {
processor: Rc<RefCell<RequestProcessor>>,
}
impl Service {
fn serve(&mut self, request: String) -> Box<Future<Item = (), Error = ()>> {
let processor_clone = self.processor.clone();
let result_fut = self
.processor
.borrow()
.prepare()
.and_then(move |_| processor_clone.borrow_mut().process(request));
Box::new(result_fut)
}
}
fn main() {}
As a short summary, after an async preparation step I'm trying to run another async operation that writes a field of self. Without mutability this works easily with a plain Rc member, but mutability breaks it, producing the following error:
error[E0597]: `processor_clone` does not live long enough
--> src/main.rs:22:32
|
22 | .and_then(move |_| processor_clone.borrow_mut().process(request));
| ^^^^^^^^^^^^^^^ - `processor_clone` dropped here while still borrowed
| |
| borrowed value does not live long enough
|
= note: values in a scope are dropped in the opposite order they are created
I'd expect this should work, I don't see where the mutable reference is still borrowed. I think process() should release the processor's &mut self after the future is returned, so the compile error shouldn't occur.
Could you please explain the underlying reasons? How should this example be changed to be accepted by the compiler?
Sometimes, it's easier to see lifetime errors if you split values up onto multiple lines. Let's try that with the closure in question:
.and_then(move |_| {
let c = processor_clone;
let mut d = c.borrow_mut();
let e = d.process(request);
e
});
If you compile this... it works. If we try recombining the lines, we can get this to fail:
.and_then(move |_| {
let c = processor_clone;
c.borrow_mut().process(request)
});
And this to work:
.and_then(move |_| {
let c = processor_clone;
return c.borrow_mut().process(request);
});
The only difference is the explicit return and the semicolon. This is very similar to When returning the outcome of consuming a StdinLock, why was the borrow to stdin retained?, so let's try the suggestion of one of the answers to enable non-lexical lifetimes. This allows your original code to compile as well.
TL;DR: It's a weakness in Rust's current borrow checker implementation and will be magically fixed at some point. In the meantime, I suggest writing it as two lines:
.and_then(move |_| {
let mut c = processor_clone.borrow_mut();
c.process(request)
});

Resources