I'm using serde and serde_json 1.0 to decode data from a base64 string:
fn from_base64_str<T: Deserialize>(string: &str) -> T {
let slice = decode_config(string, URL_SAFE).unwrap();
serde_json::from_slice(&slice).unwrap()
}
When I compile, I got this:
error[E0106]: missing lifetime specifier
--> src/main.rs:6:23
|
6 | fn from_base64_str<T: Deserialize>(string: &str) -> T {
| ^^^^^^^^^^^ expected lifetime parameter
Checking the serde doc, Deserialize is defined as:
pub trait Deserialize<'de>: Sized {
So I added the lifetime:
fn from_base64_str<'de, T: Deserialize<'de>>(string: &str) -> T {
let slice = decode_config(string, URL_SAFE).unwrap();
serde_json::from_slice(&slice).unwrap()
}
The compiler then told me:
error: `slice` does not live long enough
--> src/main.rs:11:29
|
11 | serde_json::from_slice(&slice).unwrap()
| ^^^^^ does not live long enough
12 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the lifetime 'de as defined on the body at 9:64...
--> src/main.rs:9:65
|
9 | fn from_base64_str<'de, T: Deserialize<'de>>(string: &str) -> T {
| _________________________________________________________________^ starting here...
10 | | let slice = decode_config(string, URL_SAFE).unwrap();
11 | | serde_json::from_slice(&slice).unwrap()
12 | | }
| |_^ ...ending here
I'm using Rust 1.18.0-nightly (452bf0852 2017-04-19). I only know the very basics of lifetimes in Rust, so I'm very confused by the 'de in trait Deserialize.
How can I fix the lifetime error in such function?
I found an answer from Serde issue 891: I should use DeserializeOwned instead of Deserialize.
This section of the Serde website covers Deserialize bounds in detail.
There are two main ways to write Deserialize trait bounds, whether on an impl block or a function or anywhere else.
<'de, T> where T: Deserialize<'de>
This means "T can be deserialized from some lifetime." The caller gets to decide what lifetime that is. Typically this is used when the caller also provides the data that is being deserialized from, for example in a function like serde_json::from_str. In that case the input data must also have lifetime 'de, for example it could be &'de str.
<T> where T: DeserializeOwned
This means "T can be deserialized from any lifetime." The callee gets to decide what lifetime. Usually this is because the data that is being deserialized from is going to be thrown away before the function returns, so T must not be allowed to borrow from it. For example a function that accepts base64-encoded data as input, decodes it from base64, deserializes a value of type T, then throws away the result of base64 decoding. Another common use of this bound is functions that deserialize from an IO stream, such as serde_json::from_reader.
To say it more technically, the DeserializeOwned trait is equivalent to the higher-rank trait bound for<'de> Deserialize<'de>. The only difference is DeserializeOwned is more intuitive to read. It means T owns all the data that gets deserialized.
Related
I want to repeat a certain user-provided function multiple times in an async context. Thus, I thought about a non-async closure that produces futures which can be consumed/awaited.
A minimal reproducible example is the following:
let counter = Arc::new(AtomicU64::new(0));
let counter_closure = counter.clone();
let future_producer = move || async {
counter_closure.fetch_add(1, Ordering::SeqCst);
};
but this results in this error
562 | let future_producer = move || async {
| _______________________________-------_^
| | | |
| | | return type of closure `impl Future<Output = ()>` contains a lifetime `'2`
| | lifetime `'1` represents this closure's body
563 | | counter_closure.fetch_add(1, Ordering::SeqCst);
564 | | };
| |_________^ returning this value requires that `'1` must outlive `'2`
How can I solve this problem?
The problem is that the closure owns counter_closure but the futures returned by the closure only reference this value owned by the closure. The signature of closures doesn't have a way to express this. It's akin to the "owning iterator" problem -- like how the Iterator trait doesn't permit expression of the case where the iterator dispenses references tied to its own lifetime, the family of closure traits that can be invoked on a reference to a closure (Fn and FnMut) don't have a way to express that the value returned from the closure borrows from the closure itself.
If the compiler permitted this to compile, it would be possible to create a use-after-free situation by dropping the closure while futures it has previously returned still exist. This would destroy counter_closure while it's still borrowed.
You can fix this by cloning the closure-owned counter_closure on each invocation of the closure before the async block, and moving this clone into the future with async move:
let counter = Arc::new(AtomicU64::new(0));
let counter_closure = counter.clone();
let future_producer = move || {
let counter_inner = counter_closure.clone();
async move {
counter_inner.fetch_add(1, Ordering::SeqCst);
}
};
This uncouples the lifetime of the future from the closure by giving the future its own Arc.
Let's begin with a canonical example of Arc
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let msg = Arc::new(Mutex::new(String::new()));
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = Arc::clone(&msg);
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This compiles and runs as expected. Then I realized maybe the Mutex doesn't have to live on the heap and started wondering if I can get rid of Arc and just use a shared reference to a Mutex allocated on the stack. Here is my attempt
use std::sync::Mutex;
use std::thread;
fn main() {
let msg = Mutex::new(String::new());
let mut handles = Vec::new();
for _ in 1..10 {
let local_msg = &msg;
handles.push(thread::spawn(move || {
let mut locked = local_msg.lock().unwrap();
locked.push_str("hello, world\n");
}));
}
for handle in handles {
handle.join().unwrap();
}
println!("{}", msg.lock().unwrap());
}
This one doesn't compile, though
error[E0597]: `msg` does not live long enough
--> src/main.rs:8:25
|
8 | let local_msg = &msg;
| ^^^^ borrowed value does not live long enough
9 | handles.push(thread::spawn(move || {
| ______________________-
10 | | let mut locked = local_msg.lock().unwrap();
11 | | locked.push_str("hello, world\n");
12 | | }));
| |__________- argument requires that `msg` is borrowed for `'static`
...
20 | }
| - `msg` dropped here while still borrowed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0597`.
error: could not compile `hello`
To learn more, run the command again with --verbose.
The compiler complains that local_msg doesn't have a 'static lifetime. Well, it doesn't, so the error makes sense. However, this implies the variable let local_msg = Arc::clone(&msg); in the first snippet has 'static lifetime, otherwise I should get a similar error.
Questions:
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
As a bonus, what about other heap-backed smart pointers like Box and Rc? Do they all have a 'static lifetime because the borrow checker ensures that as long as these pointers are visible, then the addresses they point to are always valid?
The thing the compiler is looking for is a lifetime bound. A lifetime bound of 'a doesn't mean “this type is a reference with lifetime 'a”, but rather “all of the references this type contains have lifetimes of at least 'a”.
(When a lifetime bound is written explicitly, it looks like where T: 'a.)
Thus, any type which does not contain any references (or rather, has no lifetime parameters) automatically satisfies the 'static lifetime bound. If T: 'static, then Arc<T>: 'static (and the same for Box and Rc).
How could Arc::clone(&msg) get a 'static lifetime? The value it points to isn't known at compile-time, and could die before the whole program exits.
It does not point to the value using a reference, so it's fine. The type of your value is Arc<Mutex<String>>; there are no lifetime parameters here because there are no references. If it were, hypothetically, Arc<'a, Mutex<String>> (a lifetime parameter which Arc doesn't actually have), then that type would not satisfy the bound.
The job of Arc (or Rc or Box) is to own the value it points to. Ownership is not a reference and thus not subject to lifetimes.
However, if you had the type Arc<Mutex<&'a str>> then that would not satisfy the bound, because it contains a reference which is not 'static.
I'm having some issues understanding lifetimes in Rust. It may also be the way I implement my design.
error[E0597]: `request` does not live long enough
--> src/service/session/server.rs:25:23
|
25 | let msg_ref = request.get_ref();
| ^^^^^^^ borrowed value does not live long enough
...
32 | let body: Box<dyn Body> = Box::new(signup);
| ---------------- cast requires that `request` is borrowed for `'static`
...
44 | }
| - `request` dropped here while still borrowed
The main source:
#[tonic::async_trait]
impl Session for SessionImplementation {
async fn signup(
&self,
request: Request<SignupRequest>,
) -> Result<Response<SessionResponse>, Status> {
let msg_ref = request.get_ref();
let signup = TxSignup::new(&msg_ref.name, &msg_ref.addr, &msg_ref.pwd);
let body: Box<dyn Body> = Box::new(signup);
let tx = Transaction::new(body);
let mut tx_signup: Box<dyn Tx> = Box::new(tx);
tx_signup.execute();
let response = SessionResponse {
deadline: 0,
cookie: "".to_string(),
status: 0,
};
Ok(Response::new(response))
}
/* more code */
}
Background
The idea is to have a Transaction, that implements Tx { execute(&self), result(&self) ... };. This Transaction has a parameter body of the type Box<dyn Box>, being the trait Body { /*some fn*/ }. Having this, I'm pretending to implement some kind of hierarchy.
The code above
On line 24 I'm getting some requests of the type SignupRequest (from proto file). This is the implementation of proto's server, using Tonic and Tokio.
After this, I have also an object TxSignup with some parameters of the type &str (set in from line 27 till 29). TxSignup implements the Body trait so I'm able to turn it into a Tx trait, apparently. The Transaction object wraps a Body implementation. I call the execute() function from the given trait Tx. All that explained has been done from line 32 till 35.
The problem
If I replace the &str type from TxSignup by type String it works. However, if I want them to be of the type &str, a lot of "incongruencies" with lifetimes emerge. I want it to be &str because none of these values will change. I think it is better to keep them on the stack instead of in the heap. Am I wrong?
If I want &str, I'm coerced to define TxSignup with <'a>, and here is where I get lost. I get why a lifetime is required, but not why all these problems appear.
As far I do understand, all elements inside the function should have the same lifetime, being killed at the end of its block (line 44). I will never send them outside.
I have tried giving to Body trait also an <'a>, and even to Tx trait (meaning the Transaction object must have one too to match the trait).
Is there any way to make it work? Am I misunderstanding the Trait use and how they work, or this patter design will never work?
Reproduction on GitHub
I have reproduced this same error in my rust-proto repository. Running cargo run should be enough.
I come from Go development and some C++, Java and Python, so I have a way of coding that may not be the most appropriate one using Rust. That's what I want to solve.
I am writing a websocket server in Rust using rust-websocket and its Tokio-based async system. I can serve clients just fine, however, I can not figure out how to share mutable state between the clients. Here is some (partial) code demonstrating this issue:
let mut core = Core::new().unwrap();
let handle = core.handle();
let server = Server::bind("localhost:62831", &handle).unwrap();
let mut state = State{
...
};
let f = server.incoming()
.map_err(|InvalidConnection {error, ..}| error)
.for_each(|upgrade, _)| {
let f = upgrade.accept()
.and_then(|s, _| {
let ctx = ClientContext{
// some other per-client values
state: &mut state,
}
...
return s.send(Message::binary(data).into())
.and_then(move |s| Ok(s, ctx)); // this could be the complete wrong way to insert context into the stream
}).and_then(|s, ctx| {
// client handling code here
});
handle.spawn(f
.map_err(...)
.map(...)
);
return Ok(())
});
core.run(f).unwrap();
This code errors with this:
error[E0373]: closure may outlive the current function, but it borrows `**state`, which is owned by the current function
--> src/main.rs:111:27
|
111 | .and_then(|(s, _)| {
| ^^^^^^^^ may outlive borrowed value `**state`
...
114 | state: &mut state,
| ----- `**state` is borrowed here
|
help: to force the closure to take ownership of `**state` (and any other referenced variables), use the `move` keyword, as shown:
| .and_then(move |(s, _)| {
When trying the compiler's suggestion, I get this:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/main.rs:111:27
|
111 | .and_then(move |(s, _)| {
| ^^^^^^^^^^^^^ cannot move out of captured outer variable in an `FnMut` closure
error: `state` does not live long enough
--> src/main.rs:114:37
|
114 | state: &mut state,
| ^^^^^ does not live long enough
...
122 | })
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
I also tried wrapping the state in a RefCell (creating the RefCell right after the state itself), however, the compiler gives a similar move error since it tries to move the RefCell into the closure that creates the client context.
You're pretty close with the RefCell. What you need now is an Rc to wrap that RefCell so you can clone the Rc and not capture the RefCell itself.
let shared_state = Rc::new(RefCell::new(State::new())));
incoming().for_each(move |s, _| {
let shared_state = shared_state.clone(); // Left uncaptured
shared_state.borrow_mut().do_mutable_state_stuff(); // Could panic
});
Note that since you're using Rc's and RefCell's now, you'll likely need to go ahead and convert your ClientContext struct to storing an Rc> instead of a &mut State. It may be possible to keep using &mut State's for some things, but your &mut State's will be tied to the lifetime of the RefMut, and if you keep it alive until the next closure runs, the borrows will panic (or fail if you use the try_ variants).
Also keep in mind if you decide you want to have multiple threads in your reactor, you will just need to change Rc to Arc, and RefCell to Mutex, which is a very natural conversion when it's needed.
Is there any way to get the slices of this vector to last long enough so that I can use them in this kind of circular structure?
fn populate_chain(file_path: &str) -> HashMap<String, HashSet<&String>> {
println!("loading...");
let time = util::StopWatch::new();
let mut words = HashMap::new();
{
let f = |mut x: Vec<String>| {
let word = x.pop().unwrap();
words.insert(word, HashSet::new());
};
Csv::process_rows(f, file_path, "\t");
}
let col: Vec<(String, HashSet<&String>)> = words.clone().into_iter().collect();
let m: usize = col.len() - 1;
for i in 0..m {
let ref k: String = col[i].0;
for j in i..m {
let ref nk: String = col[j].0;
if check_link(k, nk) {
words.get_mut(k).unwrap().insert(nk);
words.get_mut(nk).unwrap().insert(k);
}
}
}
time.print_time();
words
}
I'm using the double for loops to chain words together which are related so that they can be quickly looked up later.
Here are the compiler errors...
error: `col` does not live long enough
--> src/main.rs:28:29
|
28 | let ref k: String = col[i].0;
| ^^^ does not live long enough
...
40 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the block at 13:72...
--> src/main.rs:13:73
|
13 | fn populate_chain(file_path: &str) -> HashMap<String, HashSet<& String>>{
| ^
error: `col` does not live long enough
--> src/main.rs:30:34
|
30 | let ref nk: String = col[j].0;
| ^^^ does not live long enough
...
40 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the block at 13:72...
--> src/main.rs:13:73
|
13 | fn populate_chain(file_path: &str) -> HashMap<String, HashSet<& String>>{
|
I can tell you from your function signature that you're going to have big problems trying to write this function, at least if you want it to contain non-empty HashSets in the result.
fn populate_chain(file_path: &str) -> HashMap<String, HashSet<&String>>
This function signature has references in it; they have elided lifetimes. If you make the inferred lifetimes explicit, it would look like this:
fn populate_chain<'a>(file_path: &'a str) -> HashMap<String, HashSet<&'a String>>
In other words: This function claims that, given some string slice with lifetime 'a, it will give back a collection that holds String objects with lifetime 'a.
But you have no means for allocating such String objects within your code. :(
So, you're stuck; no matter what you put into that function body, you're not going to be able to supply an implementation that returns a non-trivial result with respect to the HashSets.
However, all is not lost. You could, for example, revise your function so that it also takes as an additional parameter a reference to a TypedArena with an appropriate lifetime, and then allocate the strings there. Another (simpler) option would be to use HashSet<String> instead of HashSet<&String>...