I'm looking for a class much like Vec<RefCell<T>>, in that it is the ultimate owner & allocator of all of its data, yet different pieces of the array can be be mutably borrowed by multiple parties indefinitely.
I emphasize indefinitely because of course pieces of Vec<T> can also be mutably borrowed by multiple parties, but doing so involves making a split which can only be resolved after the parties are done borrowing.
Vec<RefCell<T>> seems to be a world of danger and many ugly if statements checking borrow_state, which seems to be unstable. If you do something wrong, then kablammo! Panic! This is not what a lending library is like. In a lending library, if you ask for a book that isn't there, they tell you "Oh, it's checked out." Nobody dies in an explosion.
So I would like to write code something like this:
let mut a = LendingLibrary::new();
a.push(Foo{x:10});
a.push(Foo{x:11});
let b1 = a.get(0); // <-- b1 is an Option<RefMut<Foo>>
let b2 = a.get(1); // <-- b2 is an Option<RefMut<Foo>>
// the 0th element has already been borrowed, so...
let b3 = a.get(0); // <-- b3 is Option::None
Does such a thing exist? Or is there another canonical way to get this kind of behavior? A kind of "friendly RefCell"?
If the answer happens to be yes, is there also a threadsafe variant?
RefCell is not designed for long-lived borrows. The typical use case is that in a function, you'll borrow the RefCell (either mutably or immutably), work with the value, then release the borrow before returning. I'm curious to know how you're hoping to recover from a borrowed RefCell in a single-threaded context.
The thread-safe equivalent to RefCell is RwLock. It has try_read and try_write functions that do not block or panic if an incompatible lock is still acquired (on any thread, including the current thread). Contrarily to RefCell, it makes sense to just retry later if locking a RwLock fails, since another thread might just happen to have locked it at the same time.
If you end up always using write or try_write, and never read or try_read, then you should probably use the simpler Mutex instead.
#![feature(borrow_state)]
use std::cell::{RefCell, RefMut, BorrowState};
struct LendingLibrary<T> {
items: Vec<RefCell<T>>
}
impl<T> LendingLibrary<T> {
fn new(items: Vec<T>) -> LendingLibrary<T> {
LendingLibrary {
items: items.into_iter().map(|e| RefCell::new(e)).collect()
}
}
fn get(&self, item: usize) -> Option<RefMut<T>> {
self.items.get(item)
.and_then(|cell| match cell.borrow_state() {
BorrowState::Unused => Some(cell.borrow_mut()),
_ => None
})
}
}
fn main() {
let lib = LendingLibrary::new(vec![1, 2, 3]);
let a = lib.get(0); // Some
let b = lib.get(1); // Some
let a2 = lib.get(0); // None
}
This currently requires a nightly release to work.
Related
I'm currently trying to write a function that is generally equivalent to numpy's tile. Currently each time I try to return a (altered or unaltered) clone of the input array, I get a warning about an overflow, and cargo prompts me to increase the recursion limit. however this function isn't recursive, so I'm assuming its happening somewhere in the implementation.
here is the stripped down function, (full version):
pub fn tile<A, D1, D2>(arr: &Array<A, D1>, reps: Vec<usize>) -> Array<A, D2>
where
A: Clone,
D1: Dimension,
D2: Dimension,
{
let num_of_reps = reps.len();
//just clone the array if reps is all ones
let mut res = arr.clone();
let bail_flag = true;
for &x in reps.iter() {
if x != 1 {
bail_flag = false;
}
}
if bail_flag {
let mut res_dim = res.shape().to_owned();
_new_shape(num_of_reps, res.ndim(), &mut res_dim);
res.to_shape(res_dim);
return res;
}
...
//otherwise do extra work
...
return res.reshape(shape_out);
}
this is the actual error I'm getting on returning res:
overflow evaluating the requirement `&ArrayBase<_, _>: Neg`
consider increasing the recursion limit by adding a `#![recursion_limit = "1024"]` attribute to your crate (`mfcc_2`)
required because of the requirements on the impl of `Neg` for `&ArrayBase<_, _>`
511 redundant requirements hidden
required because of the requirements on the impl of `Neg` for `&ArrayBase<OwnedRepr<A>, D1>`rustcE0275
I looked at the implementation of Neg in ndarray, it doesn't seem to be recursive, so I'm a little confused as to what is going on.
p.s. I'm aware there are other errors in this code, as those appeared after I switched from A to f64(the actual type I plan on using the function with), but those are mostly trivial to fix. Still if you have suggestions on any error you see I appreciate them nonetheless.
I try to go recursively through a rose tree. The following code also works as intended but I still have the problem that I need to clone the value due to issues with the borrow checker. Therefore, it would be nice if there is a way to change from cloning to something better.
Without the clone() rust complains (rightfully) that I borrow self mutable by looking at the child nodes and the second time in the closure.
The whole structure and code is more complicated and bigger than shown below but that are the core elements. Do I have to change the data structure or do I miss something obvious? If the data structure is the issue how would you change it?
Also, the NType enum seems kinda useless here but I have some additional kinds that I have to consider. Here Inner nodes always have children and Outer nodes never.
enum NType{
Inner,
Outer
}
#[derive(Eq, PartialEq, Clone, Debug)]
struct Node {
// isn't a i32 actually. In my real program it's another struct
count: i32,
n_type: NType,
children: Option<Vec<usize>>
}
#[derive(Eq, PartialEq, Clone, Debug)]
struct Tree {
nodes: Vec<Node>,
}
impl Tree{
pub fn calc(&mut self, features: &Vec<i32>) -> i32{
// root is the last node
self.calc_h(self.nodes.len() - 1, features);
self.nodes[self.nodes.len() - 1].count.clone()
}
fn calc_h(&mut self, current: usize, features: &Vec<i32>){
// do some other things to decide where to go into recursion and where not to
// also use the features
if self.nodes[current].n_type == Inner{
//cloneing is very expensiv and destroys the performance
self.nodes[current].children.as_ref().unwrap().clone().iter().for_each(|&n| self.calc_h(n, features));
self.do_smt(current)
}
self.do_smt(current)
}
}
Edit:
Lagerbaer suggested to use as_mut but that results into current being a &mut usize and that doesn't really solve the problem.
changed childs into children
The correct plural of child is children so this is what I will refer to in this answer. Presumably this is what childs means in your code.
Since node.children is already an Option, the best solution would be to .take() the vector out of the node at the start of the iteration and put it in at the end. This way we avoid holding a reference to tree.nodes during the iteration.
if self.nodes[current].n_type == Inner {
let children = self.nodes[current].children.take().unwrap();
for &child in children.iter() {
self.calc_h(child, features);
}
self.nodes[current].children = Some(children);
}
Note that the behavior is different from your original code in case of cycles, but this is not something you need to worry about if the rest of the tree is implemented correctly.
I'm having some issues understanding lifetimes in Rust. It may also be the way I implement my design.
error[E0597]: `request` does not live long enough
--> src/service/session/server.rs:25:23
|
25 | let msg_ref = request.get_ref();
| ^^^^^^^ borrowed value does not live long enough
...
32 | let body: Box<dyn Body> = Box::new(signup);
| ---------------- cast requires that `request` is borrowed for `'static`
...
44 | }
| - `request` dropped here while still borrowed
The main source:
#[tonic::async_trait]
impl Session for SessionImplementation {
async fn signup(
&self,
request: Request<SignupRequest>,
) -> Result<Response<SessionResponse>, Status> {
let msg_ref = request.get_ref();
let signup = TxSignup::new(&msg_ref.name, &msg_ref.addr, &msg_ref.pwd);
let body: Box<dyn Body> = Box::new(signup);
let tx = Transaction::new(body);
let mut tx_signup: Box<dyn Tx> = Box::new(tx);
tx_signup.execute();
let response = SessionResponse {
deadline: 0,
cookie: "".to_string(),
status: 0,
};
Ok(Response::new(response))
}
/* more code */
}
Background
The idea is to have a Transaction, that implements Tx { execute(&self), result(&self) ... };. This Transaction has a parameter body of the type Box<dyn Box>, being the trait Body { /*some fn*/ }. Having this, I'm pretending to implement some kind of hierarchy.
The code above
On line 24 I'm getting some requests of the type SignupRequest (from proto file). This is the implementation of proto's server, using Tonic and Tokio.
After this, I have also an object TxSignup with some parameters of the type &str (set in from line 27 till 29). TxSignup implements the Body trait so I'm able to turn it into a Tx trait, apparently. The Transaction object wraps a Body implementation. I call the execute() function from the given trait Tx. All that explained has been done from line 32 till 35.
The problem
If I replace the &str type from TxSignup by type String it works. However, if I want them to be of the type &str, a lot of "incongruencies" with lifetimes emerge. I want it to be &str because none of these values will change. I think it is better to keep them on the stack instead of in the heap. Am I wrong?
If I want &str, I'm coerced to define TxSignup with <'a>, and here is where I get lost. I get why a lifetime is required, but not why all these problems appear.
As far I do understand, all elements inside the function should have the same lifetime, being killed at the end of its block (line 44). I will never send them outside.
I have tried giving to Body trait also an <'a>, and even to Tx trait (meaning the Transaction object must have one too to match the trait).
Is there any way to make it work? Am I misunderstanding the Trait use and how they work, or this patter design will never work?
Reproduction on GitHub
I have reproduced this same error in my rust-proto repository. Running cargo run should be enough.
I come from Go development and some C++, Java and Python, so I have a way of coding that may not be the most appropriate one using Rust. That's what I want to solve.
In a language like C#, giving this code (I am not using the await keyword on purpose):
async Task Foo()
{
var task = LongRunningOperationAsync();
// Some other non-related operation
AnotherOperation();
result = task.Result;
}
In the first line, the long operation is run in another thread, and a Task is returned (that is a future). You can then do another operation that will run in parallel of the first one, and at the end, you can wait for the operation to be finished. I think that it is also the behavior of async/await in Python, JavaScript, etc.
On the other hand, in Rust, I read in the RFC that:
A fundamental difference between Rust's futures and those from other languages is that Rust's futures do not do anything unless polled. The whole system is built around this: for example, cancellation is dropping the future for precisely this reason. In contrast, in other languages, calling an async fn spins up a future that starts executing immediately.
In this situation, what is the purpose of async/await in Rust? Seeing other languages, this notation is a convenient way to run parallel operations, but I cannot see how it works in Rust if the calling of an async function does not run anything.
You are conflating a few concepts.
Concurrency is not parallelism, and async and await are tools for concurrency, which may sometimes mean they are also tools for parallelism.
Additionally, whether a future is immediately polled or not is orthogonal to the syntax chosen.
async / await
The keywords async and await exist to make creating and interacting with asynchronous code easier to read and look more like "normal" synchronous code. This is true in all of the languages that have such keywords, as far as I am aware.
Simpler code
This is code that creates a future that adds two numbers when polled
before
fn long_running_operation(a: u8, b: u8) -> impl Future<Output = u8> {
struct Value(u8, u8);
impl Future for Value {
type Output = u8;
fn poll(self: Pin<&mut Self>, _ctx: &mut Context) -> Poll<Self::Output> {
Poll::Ready(self.0 + self.1)
}
}
Value(a, b)
}
after
async fn long_running_operation(a: u8, b: u8) -> u8 {
a + b
}
Note that the "before" code is basically the implementation of today's poll_fn function
See also Peter Hall's answer about how keeping track of many variables can be made nicer.
References
One of the potentially surprising things about async/await is that it enables a specific pattern that wasn't possible before: using references in futures. Here's some code that fills up a buffer with a value in an asynchronous manner:
before
use std::io;
fn fill_up<'a>(buf: &'a mut [u8]) -> impl Future<Output = io::Result<usize>> + 'a {
futures::future::lazy(move |_| {
for b in buf.iter_mut() { *b = 42 }
Ok(buf.len())
})
}
fn foo() -> impl Future<Output = Vec<u8>> {
let mut data = vec![0; 8];
fill_up(&mut data).map(|_| data)
}
This fails to compile:
error[E0597]: `data` does not live long enough
--> src/main.rs:33:17
|
33 | fill_up_old(&mut data).map(|_| data)
| ^^^^^^^^^ borrowed value does not live long enough
34 | }
| - `data` dropped here while still borrowed
|
= note: borrowed value must be valid for the static lifetime...
error[E0505]: cannot move out of `data` because it is borrowed
--> src/main.rs:33:32
|
33 | fill_up_old(&mut data).map(|_| data)
| --------- ^^^ ---- move occurs due to use in closure
| | |
| | move out of `data` occurs here
| borrow of `data` occurs here
|
= note: borrowed value must be valid for the static lifetime...
after
use std::io;
async fn fill_up(buf: &mut [u8]) -> io::Result<usize> {
for b in buf.iter_mut() { *b = 42 }
Ok(buf.len())
}
async fn foo() -> Vec<u8> {
let mut data = vec![0; 8];
fill_up(&mut data).await.expect("IO failed");
data
}
This works!
Calling an async function does not run anything
The implementation and design of a Future and the entire system around futures, on the other hand, is unrelated to the keywords async and await. Indeed, Rust has a thriving asynchronous ecosystem (such as with Tokio) before the async / await keywords ever existed. The same was true for JavaScript.
Why aren't Futures polled immediately on creation?
For the most authoritative answer, check out this comment from withoutboats on the RFC pull request:
A fundamental difference between Rust's futures and those from other
languages is that Rust's futures do not do anything unless polled. The
whole system is built around this: for example, cancellation is
dropping the future for precisely this reason. In contrast, in other
languages, calling an async fn spins up a future that starts executing
immediately.
A point about this is that async & await in Rust are not inherently
concurrent constructions. If you have a program that only uses async &
await and no concurrency primitives, the code in your program will
execute in a defined, statically known, linear order. Obviously, most
programs will use some kind of concurrency to schedule multiple,
concurrent tasks on the event loop, but they don't have to. What this
means is that you can - trivially - locally guarantee the ordering of
certain events, even if there is nonblocking IO performed in between
them that you want to be asynchronous with some larger set of nonlocal
events (e.g. you can strictly control ordering of events inside of a
request handler, while being concurrent with many other request
handlers, even on two sides of an await point).
This property gives Rust's async/await syntax the kind of local
reasoning & low-level control that makes Rust what it is. Running up
to the first await point would not inherently violate that - you'd
still know when the code executed, it would just execute in two
different places depending on whether it came before or after an
await. However, I think the decision made by other languages to start
executing immediately largely stems from their systems which
immediately schedule a task concurrently when you call an async fn
(for example, that's the impression of the underlying problem I got
from the Dart 2.0 document).
Some of the Dart 2.0 background is covered by this discussion from munificent:
Hi, I'm on the Dart team. Dart's async/await was designed mainly by
Erik Meijer, who also worked on async/await for C#. In C#, async/await
is synchronous to the first await. For Dart, Erik and others felt that
C#'s model was too confusing and instead specified that an async
function always yields once before executing any code.
At the time, I and another on my team were tasked with being the
guinea pigs to try out the new in-progress syntax and semantics in our
package manager. Based on that experience, we felt async functions
should run synchronously to the first await. Our arguments were
mostly:
Always yielding once incurs a performance penalty for no good reason. In most cases, this doesn't matter, but in some it really
does. Even in cases where you can live with it, it's a drag to bleed a
little perf everywhere.
Always yielding means certain patterns cannot be implemented using async/await. In particular, it's really common to have code like
(pseudo-code here):
getThingFromNetwork():
if (downloadAlreadyInProgress):
return cachedFuture
cachedFuture = startDownload()
return cachedFuture
In other words, you have an async operation that you can call multiple times before it completes. Later calls use the same
previously-created pending future. You want to ensure you don't start
the operation multiple times. That means you need to synchronously
check the cache before starting the operation.
If async functions are async from the start, the above function can't use async/await.
We pleaded our case, but ultimately the language designers stuck with
async-from-the-top. This was several years ago.
That turned out to be the wrong call. The performance cost is real
enough that many users developed a mindset that "async functions are
slow" and started avoiding using it even in cases where the perf hit
was affordable. Worse, we see nasty concurrency bugs where people
think they can do some synchronous work at the top of a function and
are dismayed to discover they've created race conditions. Overall, it
seems users do not naturally assume an async function yields before
executing any code.
So, for Dart 2, we are now taking the very painful breaking change to
change async functions to be synchronous to the first await and
migrating all of our existing code through that transition. I'm glad
we're making the change, but I really wish we'd done the right thing
on day one.
I don't know if Rust's ownership and performance model place different
constraints on you where being async from the top really is better,
but from our experience, sync-to-the-first-await is clearly the better
trade-off for Dart.
cramert replies (note that some of this syntax is outdated now):
If you need code to execute immediately when a function is called
rather than later on when the future is polled, you can write your
function like this:
fn foo() -> impl Future<Item=Thing> {
println!("prints immediately");
async_block! {
println!("prints when the future is first polled");
await!(bar());
await!(baz())
}
}
Code examples
These examples use the async support in Rust 1.39 and the futures crate 0.3.1.
Literal transcription of the C# code
use futures; // 0.3.1
async fn long_running_operation(a: u8, b: u8) -> u8 {
println!("long_running_operation");
a + b
}
fn another_operation(c: u8, d: u8) -> u8 {
println!("another_operation");
c * d
}
async fn foo() -> u8 {
println!("foo");
let sum = long_running_operation(1, 2);
another_operation(3, 4);
sum.await
}
fn main() {
let task = foo();
futures::executor::block_on(async {
let v = task.await;
println!("Result: {}", v);
});
}
If you called foo, the sequence of events in Rust would be:
Something implementing Future<Output = u8> is returned.
That's it. No "actual" work is done yet. If you take the result of foo and drive it towards completion (by polling it, in this case via futures::executor::block_on), then the next steps are:
Something implementing Future<Output = u8> is returned from calling long_running_operation (it does not start work yet).
another_operation does work as it is synchronous.
the .await syntax causes the code in long_running_operation to start. The foo future will continue to return "not ready" until the computation is done.
The output would be:
foo
another_operation
long_running_operation
Result: 3
Note that there are no thread pools here: this is all done on a single thread.
async blocks
You can also use async blocks:
use futures::{future, FutureExt}; // 0.3.1
fn long_running_operation(a: u8, b: u8) -> u8 {
println!("long_running_operation");
a + b
}
fn another_operation(c: u8, d: u8) -> u8 {
println!("another_operation");
c * d
}
async fn foo() -> u8 {
println!("foo");
let sum = async { long_running_operation(1, 2) };
let oth = async { another_operation(3, 4) };
let both = future::join(sum, oth).map(|(sum, _)| sum);
both.await
}
Here we wrap synchronous code in an async block and then wait for both actions to complete before this function will be complete.
Note that wrapping synchronous code like this is not a good idea for anything that will actually take a long time; see What is the best approach to encapsulate blocking I/O in future-rs? for more info.
With a threadpool
// Requires the `thread-pool` feature to be enabled
use futures::{executor::ThreadPool, future, task::SpawnExt, FutureExt};
async fn foo(pool: &mut ThreadPool) -> u8 {
println!("foo");
let sum = pool
.spawn_with_handle(async { long_running_operation(1, 2) })
.unwrap();
let oth = pool
.spawn_with_handle(async { another_operation(3, 4) })
.unwrap();
let both = future::join(sum, oth).map(|(sum, _)| sum);
both.await
}
The purpose of async/await in Rust is to provide a toolkit for concurrency—same as in C# and other languages.
In C# and JavaScript, async methods start running immediately, and they're scheduled whether you await the result or not. In Python and Rust, when you call an async method, nothing happens (it isn't even scheduled) until you await it. But it's largely the same programming style either way.
The ability to spawn another task (that runs concurrent with and independent of the current task) is provided by libraries: see async_std::task::spawn and tokio::task::spawn.
As for why Rust async is not exactly like C#, well, consider the differences between the two languages:
Rust discourages global mutable state. In C# and JS, every async method call is implicitly added to a global mutable queue. It's a side effect to some implicit context. For better or worse, that's not Rust's style.
Rust is not a framework. It makes sense that C# provides a default event loop. It also provides a great garbage collector! Lots of things that come standard in other languages are optional libraries in Rust.
Consider this simple pseudo-JavaScript code that fetches some data, processes it, fetches some more data based on the previous step, summarises it, and then prints a result:
getData(url)
.then(response -> parseObjects(response.data))
.then(data -> findAll(data, 'foo'))
.then(foos -> getWikipediaPagesFor(foos))
.then(sumPages)
.then(sum -> console.log("sum is: ", sum));
In async/await form, that's:
async {
let response = await getData(url);
let objects = parseObjects(response.data);
let foos = findAll(objects, 'foo');
let pages = await getWikipediaPagesFor(foos);
let sum = sumPages(pages);
console.log("sum is: ", sum);
}
It introduces a lot of single-use variables and is arguably worse than the original version with promises. So why bother?
Consider this change, where the variables response and objects are needed later on in the computation:
async {
let response = await getData(url);
let objects = parseObjects(response.data);
let foos = findAll(objects, 'foo');
let pages = await getWikipediaPagesFor(foos);
let sum = sumPages(pages, objects.length);
console.log("sum is: ", sum, " and status was: ", response.status);
}
And try to rewrite it in the original form with promises:
getData(url)
.then(response -> Promise.resolve(parseObjects(response.data))
.then(objects -> Promise.resolve(findAll(objects, 'foo'))
.then(foos -> getWikipediaPagesFor(foos))
.then(pages -> sumPages(pages, objects.length)))
.then(sum -> console.log("sum is: ", sum, " and status was: ", response.status)));
Each time you need to refer back to a previous result, you need to nest the entire structure one level deeper. This can quickly become very difficult to read and maintain, but the async/await version does not suffer from this problem.
I try to run this code:
impl FibHeap {
fn insert(&mut self, key: int) -> () {
let new_node = Some(box create_node(key, None, None));
match self.min{
Some(ref mut t) => t.right = new_node,
None => (),
};
println!("{}",get_right(self.min));
}
}
fn get_right(e: Option<Box<Node>>) -> Option<Box<Node>> {
match e {
Some(t) => t.right,
None => None,
}
}
And get error
error: cannot move out of dereference of `&mut`-pointer
println!("{}",get_right(self.min));
^
I dont understand why I get this problem, and what I must use to avoid problem.
Your problem is that get_right() accepts Option<Box<Node>>, while it should really accept Option<&Node> and return Option<&Node> as well. The call site should be also changed appropriately.
Here is the explanation. Box<T> is a heap-allocated box. It obeys value semantics (that is, it behaves like plain T except that it has associated destructor so it is always moved, never copied). Hence passing just Box<T> into a function means giving up ownership of the value and moving it into the function. However, it is not what you really want and neither can do here. get_right() function only queries the existing structure, so it does not need ownership. And if ownership is not needed, then references are the answer. Moreover, it is just impossible to move the self.min into a function, because self.min is accessed through self, which is a borrowed pointer. However, you can't move out from a borrowed data, it is one of the basic safety guarantees provided by the compiler.
Change your get_right() definition to something like this:
fn get_right(e: Option<&Node>) -> Option<&Node> {
e.and_then(|n| n.right.as_ref().map(|r| &**r))
}
Then println!() call should be changed to this:
println!("{}", get_right(self.min.map(|r| &**r))
Here is what happens here. In order to obtain Option<&Node> from Option<Box<Node>> you need to apply the "conversion" to insides of the original Option. There is a method exactly for that, called map(). However, map() takes its target by value, which would mean moving Box<Node> into the closure. However, we only want to borrow Node, so first we need to go from Option<Box<Node>> to Option<&Box<Node>> in order for map() to work.
Option<T> has a method, as_ref(), which takes its target by reference and returns Option<&T>, a possible reference to the internals of the option. In our case it would be Option<&Box<Node>>. Now this value can be safely map()ped over since it contains a reference and a reference can be freely moved without affecting the original value.
So, next, map(|r| &**r) is a conversion from Option<&Box<Node>> to Option<&Node>. The closure argument is applied to the internals of the option if they are present, otherwise None is just passed through. &**r should be read inside out: &(*(*r)), that is, first we dereference &Box<Node>, obtaining Box<Node>, then we dereference the latter, obtaining just Node, and then we take a reference to it, finally getting &Node. Because these reference/dereference operations are juxtaposed, there is no movement/copying involved. So, we got an optional reference to a Node, Option<&Node>.
You can see that similar thing happens in get_right() function. However, there is also a new method, and_then() is called. It is equivalent to what you have written in get_right() initially: if its target is None, it returns None, otherwise it returns the result of Option-returning closure passed as its argument:
fn and_then<U>(self, f: |T| -> Option<U>) -> Option<U> {
match self {
Some(e) => f(e),
None => None
}
}
I strongly suggest reading the official guide which explains what ownership and borrowing are and how to use them, because these are the very foundation of Rust language and it is very important to grasp them in order to be productive with Rust.