Cannot move out of borrowed content when borrowing a generic type - pointers

I have a program that more or less looks like this
struct Test<T> {
vec: Vec<T>
}
impl<T> Test<T> {
fn get_first(&self) -> &T {
&self.vec[0]
}
fn do_something_with_x(&self, x: T) {
// Irrelevant
}
}
fn main() {
let t = Test { vec: vec![1i32, 2, 3] };
let x = t.get_first();
t.do_something_with_x(*x);
}
Basically, we call a method on the struct Test that borrows some value. Then we call another method on the same struct, passing the previously obtained value.
This example works perfectly fine. Now, when we make the content of main generic, it doesn't work anymore.
fn generic_main<T>(t: Test<T>) {
let x = t.get_first();
t.do_something_with_x(*x);
}
Then I get the following error:
error: cannot move out of borrowed content
src/main.rs:14 let raw_x = *x;
I'm not completely sure why this is happening. Can someone explain to me why Test<i32> isn't borrowed when calling get_first while Test<T> is?

The short answer is that i32 implements the Copy trait, but T does not. If you use fn generic_main<T: Copy>(t: Test<T>), then your immediate problem is fixed.
The longer answer is that Copy is a special trait which means values can be copied by simply copying bits. Types like i32 implement Copy. Types like String do not implement Copy because, for example, it requires a heap allocation. If you copied a String just by copying bits, you'd end up with two String values pointing to the same chunk of memory. That would not be good (it's unsafe!).
Therefore, giving your T a Copy bound is quite restrictive. A less restrictive bound would be T: Clone. The Clone trait is similar to Copy (in that it copies values), but it's usually done by more than just "copying bits." For example, the String type will implement Clone by creating a new heap allocation for the underlying memory.
This requires you to change how your generic_main is written:
fn generic_main<T: Clone>(t: Test<T>) {
let x = t.get_first();
t.do_something_with_x(x.clone());
}
Alternatively, if you don't want to have either the Clone or Copy bounds, then you could change your do_something_with_x method to take a reference to T rather than an owned T:
impl<T> Test<T> {
// other methods elided
fn do_something_with_x(&self, x: &T) {
// Irrelevant
}
}
And your generic_main stays mostly the same, except you don't dereference x:
fn generic_main<T>(t: Test<T>) {
let x = t.get_first();
t.do_something_with_x(x);
}
You can read more about Copy in the docs. There are some nice examples, including how to implement Copy for your own types.

Related

How to please the borrow checker when implementing a 'too large to fit in memory' datastructure?

I am implementing a simple B-tree in Rust (as hobby project).
The basic implementation works well.
As long as everything fits in memory, nodes can store direct references to their children and it is easy to traverse and use them.
However, to be able to store more data inside than fits in RAM, it should be possible to store tree nodes on disk and retrieve parts of the tree only when they are needed.
But now it becomes incredibly hard to return a reference to a part of the tree.
For instance, when implementing Node::lookup_value:
extern crate arrayvec;
use arrayvec::ArrayVec;
use std::collections::HashMap;
type NodeId = usize;
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum NodeContents<Value> {
Leaf(ArrayVec<Value, 15>),
Internal(ArrayVec<NodeId, 16>)
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Node<Key, Value> {
pub keys: ArrayVec<Key, 15>,
pub contents: NodeContents<Value>,
}
pub trait Pool<T> {
type PoolId;
/// Invariants:
/// - Ony call `lookup` with a key which you *know* exists. This should always be possible, as stuff is never 'simply' removed.
/// - Do not remove or alter the things backing the pool in the meantime (Adding more is obviously allowed!).
fn lookup(&self, key: &Self::PoolId) -> T;
}
#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Debug)]
pub struct EphemeralPoolId(usize);
// An example of a pool that keeps all nodes in memory
// Similar (more complicated) implementations can be made for pools storing nodes on disk, remotely, etc.
pub struct EphemeralPool<T> {
next_id: usize,
contents: HashMap<usize, T>,
}
impl<T: Clone> Pool<T> for EphemeralPool<T> {
type PoolId = EphemeralPoolId;
fn lookup(&self, key: &Self::PoolId) -> T {
let val = self.contents.get(&key.0).expect("Tried to look up a value in the pool which does not exist. This breaks the invariant that you should only use it with a key which you received earlier from writing to it.").clone();
val
}
}
impl<Key: Ord, Value> Node<Key, Value> {
pub fn lookup_value<'pool>(&self, key: &Key, pool: &'pool impl Pool<Node<Key, Value>, PoolId=NodeId>) -> Option<&Value> {
match &self.contents {
NodeContents::Leaf(values) => { // Simple base case. No problems here.
self.keys.binary_search(key).map(|index| &values[index]).ok()
},
NodeContents::Internal(children) => {
let index = self.keys.binary_search(key).map(|index| index + 1).unwrap_or_else(|index| index);
let child_id = &children[index];
// Here the borrow checker gets mad:
let child = pool.lookup(child_id); // NodePool::lookup(&self, NodeId) -> Node<K, V>
child.lookup_value(key, pool)
}
}
}
}
Rust Playground
The borrow checker gets mad.
I understand why, but not why to solve it.
It gets mad, because NodePool::lookup returns a node by value. (And it has to do so, because it reads it from disk. In the future there may be a cache in the middle, but elements from this cache might be removed at any time, so returning references to elements in this cache is also not possible.)
However, lookup_value returns a reference to a tiny part of this value. But the value stops being around once the function returns.
In the Borrow Checker's parlance:
child.lookup_value(key, pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returns a reference to data owned by the current function
How to solve this problem?
The simplest solution would be to change the function in general to always return by-value (Option<V>). However, these values are often large and I would like to refrain from needless copies. It is quite likely that lookup_value is called in quick succession with closely related keys. (And also, there is a more complex scenario which the borrow checker similarly complains about, where we look up a range of sequential values with an interator. The same situation applies there.)
Are there ways in which I can make sure that child lives long enough?
I have tried to pass in an extra &mut HashMap<NodeId, Arc<Node<K, V>>> argument to store all children needed in the current traversal to keep their references alive for long enough. (See this variant of the code on the Rust Playground here)
However, then you end up with a recursive function that takes (and modifies) a hashmap as well as an element in the hashmap. Rust (rightfully) blocks this from working: It's (a) not possible to convince Rust that you won't overwrite earlier values in the hashmap, and (b) it might need to be reallocated when it grows which would also invalidate the references.
So that does not work.
Are there other solutions? (Even ones possibly involving unsafe?)
Minimal Reproducible Example
Your example contained a lot of errors and undefined types. Please provide a proper minimal reproducible example next time to avoid frustration and increase the chance of getting a quality answer.
EDIT: I saw your modified question too late with the proper minimal example, I hope this one fits. If not, I'll take another look. Let me know.
Either way, I attempted to reverse-engineer what your minimal example could have been:
use std::marker::PhantomData;
type NodeId = usize;
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum NodeContents<Value> {
Leaf(Vec<Value>),
Internal(Vec<NodeId>),
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Node<Key, Value> {
pub keys: Vec<Key>,
pub contents: NodeContents<Value>,
}
struct NodePool<Key, Value> {
_p: PhantomData<(Key, Value)>,
}
impl<Key, Value> NodePool<Key, Value> {
fn lookup(&self, _child_id: NodeId) -> Node<Key, Value> {
todo!()
}
}
impl<Key, Value> Node<Key, Value>
where
Key: Ord,
{
pub fn lookup_value(&self, key: &Key, pool: &NodePool<Key, Value>) -> Option<&Value> {
match &self.contents {
NodeContents::Leaf(values) => {
// Simple base case. No problems here.
self.keys
.binary_search(key)
.map(|index| &values[index])
.ok()
}
NodeContents::Internal(children) => {
let index = self
.keys
.binary_search(key)
.map(|index| index + 1)
.unwrap_or_else(|index| index);
let child_id = &children[index];
// Here the borrow checker gets mad:
let child = pool.lookup(*child_id); // NodePool::lookup(&self, NodeId) -> Node<K, V>
child.lookup_value(key, pool)
}
}
}
}
error[E0515]: cannot return reference to local variable `child`
--> src/lib.rs:50:17
|
50 | child.lookup_value(key, pool)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returns a reference to data owned by the current function
Thoughts and Potential Solutions
The main question you have to ask yourself: Who owns the data once it's loaded from disk?
This is not a simple question with a simple answer. It opens many new questions:
If you query the same key twice, should it load from disk twice? Or should it be cached and some other previously cached value should be moved to disk?
If it should be loaded from disk twice, should the entire Node that lookup returned be kept alive until the borrow is returned, or should only the Value be moved out and kept alive?
If your answer is "it should be loaded twice" and "only the Value should be kept alive", you could use an enum that can carry either a borrowed or owned Value:
enum ValueHolder<'a, Value> {
Owned(Value),
Borrowed(&'a Value),
}
impl<'a, Value> std::ops::Deref for ValueHolder<'a, Value> {
type Target = Value;
fn deref(&self) -> &Self::Target {
match self {
ValueHolder::Owned(val) => val,
ValueHolder::Borrowed(val) => val,
}
}
}
In your code, it could look like this:
use std::marker::PhantomData;
type NodeId = usize;
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum NodeContents<Value> {
Leaf(Vec<Value>),
Internal(Vec<NodeId>),
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Node<Key, Value> {
pub keys: Vec<Key>,
pub contents: NodeContents<Value>,
}
pub struct NodePool<Key, Value> {
_p: PhantomData<(Key, Value)>,
}
impl<Key, Value> NodePool<Key, Value> {
fn lookup(&self, _child_id: NodeId) -> Node<Key, Value> {
todo!()
}
}
pub enum ValueHolder<'a, Value> {
Owned(Value),
Borrowed(&'a Value),
}
impl<'a, Value> std::ops::Deref for ValueHolder<'a, Value> {
type Target = Value;
fn deref(&self) -> &Self::Target {
match self {
ValueHolder::Owned(val) => val,
ValueHolder::Borrowed(val) => val,
}
}
}
impl<Key, Value> Node<Key, Value>
where
Key: Ord,
{
pub fn lookup_value(
&self,
key: &Key,
pool: &NodePool<Key, Value>,
) -> Option<ValueHolder<Value>> {
match &self.contents {
NodeContents::Leaf(values) => {
// Simple base case. No problems here.
self.keys
.binary_search(key)
.map(|index| ValueHolder::Borrowed(&values[index]))
.ok()
}
NodeContents::Internal(children) => {
let index = self
.keys
.binary_search(key)
.map(|index| index + 1)
.unwrap_or_else(|index| index);
let child_id = &children[index];
// Here the borrow checker gets mad:
let child = pool.lookup(*child_id); // NodePool::lookup(&self, NodeId) -> Node<K, V>
child
.into_lookup_value(key, pool)
.map(|val| ValueHolder::Owned(val))
}
}
}
pub fn into_lookup_value(self, key: &Key, pool: &NodePool<Key, Value>) -> Option<Value> {
match self.contents {
NodeContents::Leaf(mut values) => {
// Simple base case. No problems here.
self.keys
.binary_search(key)
.map(|index| values.swap_remove(index))
.ok()
}
NodeContents::Internal(children) => {
let index = self
.keys
.binary_search(key)
.map(|index| index + 1)
.unwrap_or_else(|index| index);
let child_id = &children[index];
// Here the borrow checker gets mad:
let child = pool.lookup(*child_id); // NodePool::lookup(&self, NodeId) -> Node<K, V>
child.into_lookup_value(key, pool)
}
}
}
}
In all other cases, however, it seems like further thought needs to be put into restructuring the project.
After a lot of searching and trying, and looking at what other databases are doing (great suggestion, #Finomnis!) the solution presented itself.
Use an Arena
The intuition to use a HashMap was a good one, but you have the problems with lifetimes, for two reasons:
A HashMap might be reallocated when growing, which would invalidate old references.
You cannot convince the compiler that new insertions into the Hashmap will not overwrite existing values.
However, there is an alternative datastructure which is able to provide these guarantees: Arena allocators.
With these, you essentially 'tie' the lifetimes of the values you insert into them, to the lifetime of the allocator as a whole.
Internally (at least conceptually), an arena keeps track of a list of memory regions in which data might be stored, the most recent one of which is not entirely full. Individual memory regions are never reallocated to ensure that old references remain valid; instead, when the current memory region is full, a new (larger) one is allocated and added to the list, becoming the new 'current'.
Everything is destroyed at once at the point the arena itself is deallocated.
There are two prevalent Rust libraries that provide arenas. They are very similar, with slightly different trade-offs:
typed-arena allows only a single type T per arena, but runs all Drop implementations when the arena is dropped.
bumpalo allows different objects to be stored in the same arena, but their Drop implementations do not run.
In the common situation of having only a single datatype with no special drop implementation, you might want to benchmark which one is faster in your use-case.
In the original question, we are dealing with a cache, and it makes sense to reduce copying of large datastructures by passing around Arcs between the cache and the arena. This means that you can only use typed-arena in this case, as Arc has a special Drop implementation.

How to convert a vector of vectors into a vector of slices without creating a new object? [duplicate]

I have the following:
enum SomeType {
VariantA(String),
VariantB(String, i32),
}
fn transform(x: SomeType) -> SomeType {
// very complicated transformation, reusing parts of x in order to produce result:
match x {
SomeType::VariantA(s) => SomeType::VariantB(s, 0),
SomeType::VariantB(s, i) => SomeType::VariantB(s, 2 * i),
}
}
fn main() {
let mut data = vec![
SomeType::VariantA("hello".to_string()),
SomeType::VariantA("bye".to_string()),
SomeType::VariantB("asdf".to_string(), 34),
];
}
I would now like to call transform on each element of data and store the resulting value back in data. I could do something like data.into_iter().map(transform).collect(), but this will allocate a new Vec. Is there a way to do this in-place, reusing the allocated memory of data? There once was Vec::map_in_place in Rust but it has been removed some time ago.
As a work-around, I've added a Dummy variant to SomeType and then do the following:
for x in &mut data {
let original = ::std::mem::replace(x, SomeType::Dummy);
*x = transform(original);
}
This does not feel right, and I have to deal with SomeType::Dummy everywhere else in the code, although it should never be visible outside of this loop. Is there a better way of doing this?
Your first problem is not map, it's transform.
transform takes ownership of its argument, while Vec has ownership of its arguments. Either one has to give, and poking a hole in the Vec would be a bad idea: what if transform panics?
The best fix, thus, is to change the signature of transform to:
fn transform(x: &mut SomeType) { ... }
then you can just do:
for x in &mut data { transform(x) }
Other solutions will be clunky, as they will need to deal with the fact that transform might panic.
No, it is not possible in general because the size of each element might change as the mapping is performed (fn transform(u8) -> u32).
Even when the sizes are the same, it's non-trivial.
In this case, you don't need to create a Dummy variant because creating an empty String is cheap; only 3 pointer-sized values and no heap allocation:
impl SomeType {
fn transform(&mut self) {
use SomeType::*;
let old = std::mem::replace(self, VariantA(String::new()));
// Note this line for the detailed explanation
*self = match old {
VariantA(s) => VariantB(s, 0),
VariantB(s, i) => VariantB(s, 2 * i),
};
}
}
for x in &mut data {
x.transform();
}
An alternate implementation that just replaces the String:
impl SomeType {
fn transform(&mut self) {
use SomeType::*;
*self = match self {
VariantA(s) => {
let s = std::mem::replace(s, String::new());
VariantB(s, 0)
}
VariantB(s, i) => {
let s = std::mem::replace(s, String::new());
VariantB(s, 2 * *i)
}
};
}
}
In general, yes, you have to create some dummy value to do this generically and with safe code. Many times, you can wrap your whole element in Option and call Option::take to achieve the same effect .
See also:
Change enum variant while moving the field to the new variant
Why is it so complicated?
See this proposed and now-closed RFC for lots of related discussion. My understanding of that RFC (and the complexities behind it) is that there's an time period where your value would have an undefined value, which is not safe. If a panic were to happen at that exact second, then when your value is dropped, you might trigger undefined behavior, a bad thing.
If your code were to panic at the commented line, then the value of self is a concrete, known value. If it were some unknown value, dropping that string would try to drop that unknown value, and we are back in C. This is the purpose of the Dummy value - to always have a known-good value stored.
You even hinted at this (emphasis mine):
I have to deal with SomeType::Dummy everywhere else in the code, although it should never be visible outside of this loop
That "should" is the problem. During a panic, that dummy value is visible.
See also:
How can I swap in a new value for a field in a mutable reference to a structure?
Temporarily move out of borrowed content
How do I move out of a struct field that is an Option?
The now-removed implementation of Vec::map_in_place spans almost 175 lines of code, most of having to deal with unsafe code and reasoning why it is actually safe! Some crates have re-implemented this concept and attempted to make it safe; you can see an example in Sebastian Redl's answer.
You can write a map_in_place in terms of the take_mut or replace_with crates:
fn map_in_place<T, F>(v: &mut [T], f: F)
where
F: Fn(T) -> T,
{
for e in v {
take_mut::take(e, f);
}
}
However, if this panics in the supplied function, the program aborts completely; you cannot recover from the panic.
Alternatively, you could supply a placeholder element that sits in the empty spot while the inner function executes:
use std::mem;
fn map_in_place_with_placeholder<T, F>(v: &mut [T], f: F, mut placeholder: T)
where
F: Fn(T) -> T,
{
for e in v {
let mut tmp = mem::replace(e, placeholder);
tmp = f(tmp);
placeholder = mem::replace(e, tmp);
}
}
If this panics, the placeholder you supplied will sit in the panicked slot.
Finally, you could produce the placeholder on-demand; basically replace take_mut::take with take_mut::take_or_recover in the first version.

How can I copy a vector to another location and reuse the existing allocated memory?

In C++, to copy the contents of a vector to another vector we use the assignment operator dest = src. However, in Rust src would be moved into dest and no longer usable.
I know the simplest answer is to do dest = src.clone() (for the sake of this question we'll assume T in Vec<T> is Clone). However - if I'm understanding correctly - this creates a brand new third vector with the copied contents of src and moves it into dest, throwing away dest's dynamically allocated array. If this is correct, it's a completely unnecessary dynamic allocation when we could have just copied the content directly into dest (assuming it had sufficient capacity).
Below is a function I've made that does exactly what I would like to do: empty out the dest vector and copy the elements of src to it.
// copy contents of src to dest without just cloning src
fn copy_content<T: Clone>(dest: &mut Vec<T>, src: &Vec<T>) {
dest.clear();
if dest.capacity() < src.len() {
dest.reserve(src.len());
}
for x in src {
dest.push(x.clone());
}
}
Is there a way to do this with builtin or standard library utilities? Is the dest = src.clone() optimized by the compiler to do this anyway?
I know that if T has dynamic resources then the extra allocation from src.clone() isn't a big deal, but if T is e.g. i32 or any other Copy type then it forces an allocation where none are necessary.
Did you ever look at the definition of Clone? It has the well known clone method but also a useful but often forgotten clone_from method:
pub trait Clone : Sized {
fn clone(&self) -> Self;
fn clone_from(&mut self, source: &Self) {
*self = source.clone()
}
}
To quote the doc:
Performs copy-assignment from source.
a.clone_from(&b) is equivalent to a = b.clone() in functionality, but can be overridden to reuse the resources of a to avoid unnecessary allocations.
Of course a type such as Vec does not use the provided-by-default clone_from and defines its own in a more efficient way, similar to what you would get in C++ from writing dest = src:
fn clone_from(&mut self, other: &Vec<T>) {
other.as_slice().clone_into(self);
}
with [T]::clone_into being defined as:
fn clone_into(&self, target: &mut Vec<T>) {
// drop anything in target that will not be overwritten
target.truncate(self.len());
let len = target.len();
// reuse the contained values' allocations/resources.
target.clone_from_slice(&self[..len]);
// target.len <= self.len due to the truncate above, so the
// slice here is always in-bounds.
target.extend_from_slice(&self[len..]);
}

Why does asserting on the result of Deref::deref fail with a type mismatch?

The following is the Deref example from The Rust Programming Language except I've added another assertion.
Why does the assert_eq with the deref also equal 'a'? Why do I need a * once I've manually called deref?
use std::ops::Deref;
struct DerefExample<T> {
value: T,
}
impl<T> Deref for DerefExample<T> {
type Target = T;
fn deref(&self) -> &T {
&self.value
}
}
fn main() {
let x = DerefExample { value: 'a' };
assert_eq!('a', *x.deref()); // this is true
// assert_eq!('a', x.deref()); // this is a compile error
assert_eq!('a', *x); // this is also true
println!("ok");
}
If I uncomment the line, I get this error:
error[E0308]: mismatched types
--> src/main.rs:18:5
|
18 | assert_eq!('a', x.deref());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected char, found &char
|
= note: expected type `char`
found type `&char`
= help: here are some functions which might fulfill your needs:
- .to_ascii_lowercase()
- .to_ascii_uppercase()
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
First, let's spell out the generic types for your specific example: 'a' is char, so we have:
impl Deref for DerefExample<char> {
type Target = char;
fn deref(&self) -> &char {
&self.value
}
}
Notably, the return type of deref is a reference to a char. Thus it shouldn't be surprising that, when you use just x.deref(), the result is a &char rather than a char. Remember, at that point deref is just another normal method — it's just implicitly invoked as part of some language-provided special syntax. *x, for example, will call deref and dereference the result, when applicable. x.char_method() and fn_taking_char(&x) will also call deref some number of times and then do something further with the result.
Why does deref return a reference to begin with, you ask? Isn't that circular? Well, no, it isn't circular: it reduces library-defined smart pointers to the built-in type &T which the compiler already knows how to dereference. By returning a reference instead of a value, you avoid a copy/move (which may not always be possible!) and allow &*x (or &x when it's coerced) to refer to the actual char that DerefExample holds rather than a temporary copy.
See also:
Why is the return type of Deref::deref itself a reference?

Pointer-stashing generics via `mem::transmute()`

I'm attempting to write Rust bindings for a C collection library (Judy Arrays [1]) which only provides itself room to store a pointer-width value. My company has a fair amount of existing code which uses this space to directly store non-pointer values such as pointer-width integers and small structs. I'd like my Rust bindings to allow type-safe access to such collections using generics, but am having trouble getting the pointer-stashing semantics working correctly.
The mem::transmute() function seems like one potential tool for implementing the desired behavior, but attempting to use it on an instance of a parameterized type yield a confusing-to-me compilation error.
Example code:
pub struct Example<T> {
v: usize,
t: PhantomData<T>,
}
impl<T> Example<T> {
pub fn new() -> Example<T> {
Example { v: 0, t: PhantomData }
}
pub fn insert(&mut self, val: T) {
unsafe {
self.v = mem::transmute(val);
}
}
}
Resulting error:
src/lib.rs:95:22: 95:36 error: cannot transmute to or from a type that contains type parameters in its interior [E0139]
src/lib.rs:95 self.v = mem::transmute(val);
^~~~~~~~~~~~~~
Does this mean a type consisting only of a parameter "contains type parameters in its interior" and thus transmute() just won't work here? Any suggestions of the right way to do this?
(Related question, attempting to achieve the same result, but not necessarily via mem::transmute().)
[1] I'm aware of the existing rust-judy project, but it doesn't support the pointer-stashing I want, and I'm writing these new bindings largely as a learning exercise anyway.
Instead of transmuting T to usize directly, you can transmute a &T to &usize:
pub fn insert(&mut self, val: T) {
unsafe {
let usize_ref: &usize = mem::transmute(&val);
self.v = *usize_ref;
}
}
Beware that this may read from an invalid memory location if the size of T is smaller than the size of usize or if the alignment requirements differ. This could cause a segfault. You can add an assertion to prevent this:
assert_eq!(mem::size_of::<T>(), mem::size_of::<usize>());
assert!(mem::align_of::<usize>() <= mem::align_of::<T>());

Resources