Trying to modify a Vec of futures inside a Rc<RefCell<...>> - vector

I try to wait and remove one-by-one future from a Vec of futures. It does not work. I understand why it does not work: Pin is not copyable. But how to correct this error?
extern crate futures;
use std::cell::{RefCell};
use std::rc::Rc;
use std::pin::Pin;
use std::future::Future;
use futures::channel::oneshot::Canceled;
use futures::executor::block_on;
use futures::future::select_all;
fn run_queries_body() {
let _futures: Vec<Pin<Box<dyn Future<Output=Result<(), Canceled>>>>> = Vec::new();
let futuresRc = Rc::new(RefCell::new(_futures)); // TODO: Cell instead
// This in actual could be called inside another future, so we need Rc<RefCell<...>>
let mut futures = futuresRc.borrow_mut();
let f3 = futures.iter().map(|x| *x);
let (_res, _idx, remaining_futures) = block_on(select_all(f3));
*futures = remaining_futures;
}
error[E0507]: cannot move out of `*x` which is behind a shared reference
--> src/lib.rs:16:37
|
16 | let f3 = futures.iter().map(|x| *x);
| ^^ move occurs because `*x` has type `std::pin::Pin<std::boxed::Box<dyn futures::Future<Output = std::result::Result<(), futures::channel::oneshot::Canceled>>>>`, which does not implement the `Copy` trait

The problem is not in the pin - a boxed future is safe to move along with its pin because box means the future is heap-allocated, so moving the box doesn't move the future. The pin serves to forbid moving the future out of its box, but you don't attempt that. Your code fails to compile because Vec::iter() iterates over references to elements, and you can't move an object out of a reference because it would leave the original value in an undefined state. This kind of move is only allowed for types that can be trivially copied such as numbers or bools, which is marked by the Copy trait. The compiler's message is confusing because it mentions Pin<...>, but it only does so because that's the literal type behind the reference, and the compiler reports that the type in question isn't Copy, without implying anything about Pin semantics.
A straightforward fix is to make futures a vector of options. That allows you to extract an element out of the vector just by possessing a mutable iterator to the element by calling Option::take on the &mut Option<T>. This is well-defined because it extracts the value, but also leaves None in the old place in the vector.
In your case you would iterate over the vector using iter_mut() (playground):
pub fn run_queries_body() {
let futures: Vec<Option<Pin<Box<dyn Future<Output = Result<(), Canceled>>>>>> = vec![];
let futures_rc = Rc::new(RefCell::new(futures));
let mut futures = futures_rc.borrow_mut();
let f3 = futures.iter_mut().map(|f| f.take().unwrap());
let (_res, _idx, remaining_futures) = block_on(select_all(f3));
*futures = remaining_futures.into_iter().map(Some).collect();
}
As pointed out by #Jmb, an even simpler approach is to use Vec::drain, which removes the elements from the vector and gives you an iterator over the removed elements (playground):
pub fn run_queries_body() {
let futures: Vec<Pin<Box<dyn Future<Output = Result<(), Canceled>>>>> = vec![];
let futures_rc = Rc::new(RefCell::new(futures));
let mut futures = futures_rc.borrow_mut();
let f3 = futures.drain(..);
let (_res, _idx, remaining_futures) = block_on(select_all(f3));
*futures = remaining_futures;
}

Related

How to perform a `flat_map` (or similar operation) on an iterator N times without runtime polymorphism?

I want to be able to repeat a process where a collection that we are iterating over is altered an n number of times. n is only known at runtime, and can be specified by the user, so we cannot hard-code it into the type.
An approach that uses intermediate data structures by collect-ing between iterations is possible, like so:
let n = 10;
let mut vec1 = vec![1, 2, 3];
{
for _index in 0..n {
let temp_vec = vec1.into_iter().flat_map(|x| vec![x, x * 2]).collect();
vec1 = temp_vec;
}
}
However, this seems wasteful, because we are creating intermediate datastructures, so I went on looking for a solution that chains iterators directly.
At first I thought one could just do something like:
let mut iter = vec![1, 2, 3].into_iter();
for index in 0..n {
iter = iter.flat_map(|x| vec![x, x * 2].into_iter());
}
However, this does not work because in Rust, all functions on iterators return their own kind of 'compound iterator' struct. (In for instance Haskell, functions on iterators return the appropriate kind of result iterator, which does not become a 'bigger and bigger compound type'.)
Rewriting this as a recursive function had similar problems because (a) I was returning 'some kind of Iterator' whose type was (near?)-impossible to write out by hand because of the recursion, and (b) this type was different in the base case from the recursive case.
I found this question about conditionally returning either one or the other iterator type, as well as using impl Iterator to indicate that we return some concrete type that implements the Iterator trait, but we do not care about its exact nature.
A similar example to the code in the linked answer has been implemented in the code below as maybe_flatmap. This works.
However, I don't want to run flat_map zero or one time, but rather N times on the incoming iterator. Therefore, I adapted the code to call itself recursively up to a depth of N.
Attempting to do that, then makes the Rust compiler complain with an error[E0720]: opaque type expands to a recursive type:
use either::Either; // 1.5.3
/// Later we want to work with any appropriate items,
/// but for simplicity's sake, just use plain integers for now.
type I = u64;
/// Works, but limited to single level.
fn maybe_flatmap<T: Iterator<Item = I>>(iter: T, flag: bool) -> impl Iterator<Item = I> {
match flag {
false => Either::Left(iter),
true => Either::Right(iter.flat_map(move |x| vec![x, x * 2].into_iter())),
}
}
/// Does not work: opaque type expands to a recursive type!
fn rec_flatmap<T: Iterator<Item = I>>(iter: T, depth: usize) -> impl Iterator<Item = I> {
match depth {
0 => Either::Left(iter),
_ => {
let iter2 = iter.flat_map(move |x| vec![x, x * 2]).into_iter();
Either::Right(rec_flatmap(iter2, depth - 1))
}
}
}
fn main() {
let xs = vec![1, 2, 3, 4];
let xs2 = xs.into_iter();
let xs3 = maybe_flatmap(xs2, true);
let xs4: Vec<_> = xs3.collect();
println!("{:?}", xs4);
let ys = vec![1, 2, 3, 4];
let ys2 = ys.into_iter();
let ys3 = rec_flatmap(ys2, 5);
let ys4: Vec<_> = ys3.collect();
println!("{:?}", ys4);
}
Rust playground
error[E0720]: opaque type expands to a recursive type
--> src/main.rs:16:65
|
16 | fn rec_flatmap<T: Iterator<Item = I>>(iter: T, depth: usize) -> impl Iterator<Item = I> {
| ^^^^^^^^^^^^^^^^^^^^^^^ expands to a recursive type
|
= note: expanded type is `either::Either<T, impl std::iter::Iterator>`
I am stuck.
Since regardless of how often you flat_map, the final answer is going to be an (iterator over) a vector of integers, it seems like there ought to be a way of writing this function using only a single concrete return type.
Is this possible? Is there a way out of this situation without resorting to runtime polymorphism?
I believe/hope that a solution without dynamic polymorphism (trait objects or the like) is possible because regardless of how often you call flat_map the end result should have (at least morally) have the same type. I hope there is a way to shoehorn the (non-matching) nested FlatMap struct in a matching single static type somehow.
Is there a way to resolve this without runtime polymorphism?
No.
To solve it using a trait object:
let mut iter: Box<dyn Iterator<Item = i32>> = Box::new(vec![1, 2, 3].into_iter());
for _ in 0..n {
iter = Box::new(iter.flat_map(|x| vec![x, x * 2].into_iter()));
}
regardless of how often you call flat_map the end result should have (at least morally) have the same type
I don't know which morality to apply to type systems, but the literal size in memory is (very likely to be) different for FlatMap<...> and FlatMap<FlatMap<...>>. They are different types.
See also:
Conditionally iterate over one of several possible iterators
Creating Diesel.rs queries with a dynamic number of .and()'s
How do I iterate over a Vec of functions returning Futures in Rust?
How can I extend the lifetime of a temporary variable inside of an iterator adaptor in Rust?
Why does Iterator::take_while take ownership of the iterator?

How to get Vec<&String> after using iter().filter().collect() instead of Vec<&&String>

I have a vector of Strings. I want to get two subsets from this vector:
strings with "zzz" and strings with "bye".
The type of each subset is Vec< && String>. Which is a vector of references to a reference to original String. How can I get just Vec<& String>?
Code I am currently working on:
use std::string::String;
use std::vec::Vec;
fn main() {
let a1 = String::from("hi");
let a2 = String::from("bye");
let a3 = String::from("zzz");
let v_before: Vec<&String> = vec![&a1, &a2, &a3];
// ref_a is a copy of v_before[0]
let ref_zzz_a: &String = v_before[2];
let ref_zzz_b: &String = v_before[2];
println!("before: {:?}", v_before);
// QUESTION: why can not this also be Vec<& String> ?
// v_zzz is just like making vec![ref_zzz_a, ref_zzz_b]
let v_zzz: Vec<&&String> = v_before
.iter()
.filter(|&&element| element.contains("zzz"))
.collect();
let v_bye: Vec<&&String> = v_before
.iter()
.filter(|&&element| element.contains("bye"))
.collect();
let hand_made: Vec<&String> = vec![ref_zzz_a, ref_zzz_b];
println!("v_zzz: {:?}", v_zzz);
println!("v_bye: {:?}", v_bye);
println!("hand_made: {:?}", hand_made);
}
(Playground)
One way would be to use vector.into_iter()..., but then I would not be able to run the second filter.
Also maybe this is just the way rust is? Due to automatic dereference maybe I should just not care that this is a Vec<&& String>? Would it lead to problems later?
You can use Iterator::cloned to clone each reference instead of referencing them again:
let v_zzz: Vec<&String> = v_before
.iter()
.filter(|&element| element.contains("zzz"))
.cloned()
.collect();
Cloning a reference is cheap, since it is implemented by dereferencing.
Aside: usually it's better to pass around &str instead of &String, as it's more flexible. When I see &String in Rust code, it stands out as something that is probably not intended, or can be improved.

How do I write a function that adds an element to a vector, allowing the element to be changed before insertion?

I'm trying to make a simple example involving a vector of structs to learn Rust. All examples of vectors in the Rust literature I've found only use vectors of integers.
I want to write a function aimed at filling a vector, allowing the possibility of the element to be inserted to be changed, I can't figure out what to do. I always got a compiler error[E0308]: mismatched types on
the push method, because elem is a reference to a Point. So
push() needs a Point structure because v is a vector of Point
but if I want to modify elem, I need to pass a (mutable?) reference
What is the right thing to do?
// structure used everywhere in Rust examples
#[derive(Debug)]
struct Point {
x: i16,
y: i16
}
fn add_element(v: &mut Vec<Point>, elem: &Point) {
// modify element
elem.x = 0;
// add element
v.push(elem);
}
// this example is meant to study a vector of structs
fn main() {
// declare 2 points. By default, live on the stack
let origin = Point {x:0, y:0};
println!("origin address\t: {:p}", &origin);
let mut p1 = Point {x:1, y:1};
println!("p1 address\t: {:p}", &p1);
// declare a new vector of structs. Allocation is made in the heap
// declare mutable because we'll add elements to vector
let mut v: Vec<Point> = Vec::new();
// add points
add_element(&mut v, &origin);
add_element(&mut v, &p1);
// change p1
p1.x = 2;
p1.y = 2;
}
Let's read the error messages together:
error[E0308]: mismatched types
--> src/main.rs:10:12
|
10 | v.push(elem);
| ^^^^ expected struct `Point`, found &Point
|
= note: expected type `Point`
= note: found type `&Point`
The code is attempting to store a reference to a Point in a Vec that is declared to hold entire Points. Since Rust is a statically- and strongly- typed language, the compiler tells you that you cannot do that. The fix is to accept a Point by value:
fn add_element(v: &mut Vec<Point>, elem: Point)
This leads to the next error:
error: cannot assign to immutable field `elem.x`
--> src/main.rs:9:5
|
9 | elem.x = 0;
| ^^^^^^^^^^
You cannot change members of elem because it is not marked as mutable. Mutability of a value is a property of the binding, so let's do that:
fn add_element(v: &mut Vec<Point>, mut elem: Point)
Then change the calling of that function to adapt:
fn main() {
let origin = Point { x: 0, y: 0 };
let p1 = Point { x: 1, y: 1 };
let mut v = Vec::new();
add_element(&mut v, origin);
add_element(&mut v, p1);
}
Note that neither origin nor p1 need to be mutable because this function doesn't modify either while it owns it. It transfers ownership to add_element, which chooses to make it mutable.
but if I want to modify elem, I need to pass a (mutable?) reference
As you can see, you can simply make the elem parameter mutable when transferring the entire value to the function. Since the function owns that value, it has full control over it, including choosing to make it mutable.

Build HashSet from a vector in Rust

I want to build a HashSet<u8> from a Vec<u8>. I'd like to do this
in one line of code,
copying the data only once,
using only 2n memory,
but the only thing I can get to compile is this piece of .. junk, which I think copies the data twice and uses 3n memory.
fn vec_to_set(vec: Vec<u8>) -> HashSet<u8> {
let mut victim = vec.clone();
let x: HashSet<u8> = victim.drain(..).collect();
return x;
}
I was hoping to write something simple, like this:
fn vec_to_set(vec: Vec<u8>) -> HashSet<u8> {
return HashSet::from_iter(vec.iter());
}
but that won't compile:
error[E0308]: mismatched types
--> <anon>:5:12
|
5 | return HashSet::from_iter(vec.iter());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found &u8
|
= note: expected type `std::collections::HashSet<u8>`
= note: found type `std::collections::HashSet<&u8, _>`
.. and I don't really understand the error message, probably because I need to RTFM.
Because the operation does not need to consume the vector¹, I think it should not consume it. That only leads to extra copying somewhere else in the program:
use std::collections::HashSet;
use std::iter::FromIterator;
fn hashset(data: &[u8]) -> HashSet<u8> {
HashSet::from_iter(data.iter().cloned())
}
Call it like hashset(&v) where v is a Vec<u8> or other thing that coerces to a slice.
There are of course more ways to write this, to be generic and all that, but this answer sticks to just introducing the thing I wanted to focus on.
¹This is based on that the element type u8 is Copy, i.e. it does not have ownership semantics.
The following should work nicely; it fulfills your requirements:
use std::collections::HashSet;
use std::iter::FromIterator;
fn vec_to_set(vec: Vec<u8>) -> HashSet<u8> {
HashSet::from_iter(vec)
}
from_iter() works on types implementing IntoIterator, so a Vec argument is sufficient.
Additional remarks:
you don't need to explicitly return function results; you only need to omit the semi-colon in the last expression in its body
I'm not sure which version of Rust you are using, but on current stable (1.12) to_iter() doesn't exist
Converting Vec to HashSet
Moving data ownership
let vec: Vec<u8> = vec![1, 2, 3, 4];
let hash_set: HashSet<u8> = vec.into_iter().collect();
Cloning data
let vec: Vec<u8> = vec![1, 2, 3, 4];
let hash_set: HashSet<u8> = vec.iter().cloned().collect();

Concatenate a vector of vectors of strings

I'm trying to write a function that receives a vector of vectors of strings and returns all vectors concatenated together, i.e. it returns a vector of strings.
The best I could do so far has been the following:
fn concat_vecs(vecs: Vec<Vec<String>>) -> Vec<String> {
let vals : Vec<&String> = vecs.iter().flat_map(|x| x.into_iter()).collect();
vals.into_iter().map(|v: &String| v.to_owned()).collect()
}
However, I'm not happy with this result, because it seems I should be able to get Vec<String> from the first collect call, but somehow I am not able to figure out how to do it.
I am even more interested to figure out why exactly the return type of collect is Vec<&String>. I tried to deduce this from the API documentation and the source code, but despite my best efforts, I couldn't even understand the signatures of functions.
So let me try and trace the types of each expression:
- vecs.iter(): Iter<T=Vec<String>, Item=Vec<String>>
- vecs.iter().flat_map(): FlatMap<I=Iter<Vec<String>>, U=???, F=FnMut(Vec<String>) -> U, Item=U>
- vecs.iter().flat_map().collect(): (B=??? : FromIterator<U>)
- vals was declared as Vec<&String>, therefore
vals == vecs.iter().flat_map().collect(): (B=Vec<&String> : FromIterator<U>). Therefore U=&String.
I'm assuming above that the type inferencer is able to figure out that U=&String based on the type of vals. But if I give the expression the explicit types in the code, this compiles without error:
fn concat_vecs(vecs: Vec<Vec<String>>) -> Vec<String> {
let a: Iter<Vec<String>> = vecs.iter();
let b: FlatMap<Iter<Vec<String>>, Iter<String>, _> = a.flat_map(|x| x.into_iter());
let c = b.collect();
print_type_of(&c);
let vals : Vec<&String> = c;
vals.into_iter().map(|v: &String| v.to_owned()).collect()
}
Clearly, U=Iter<String>... Please help me clear up this mess.
EDIT: thanks to bluss' hint, I was able to achieve one collect as follows:
fn concat_vecs(vecs: Vec<Vec<String>>) -> Vec<String> {
vecs.into_iter().flat_map(|x| x.into_iter()).collect()
}
My understanding is that by using into_iter I transfer ownership of vecs to IntoIter and further down the call chain, which allows me to avoid copying the data inside the lambda call and therefore - magically - the type system gives me Vec<String> where it used to always give me Vec<&String> before. While it is certainly very cool to see how the high-level concept is reflected in the workings of the library, I wish I had any idea how this is achieved.
EDIT 2: After a laborious process of guesswork, looking at API docs and using this method to decipher the types, I got them fully annotated (disregarding the lifetimes):
fn concat_vecs(vecs: Vec<Vec<String>>) -> Vec<String> {
let a: Iter<Vec<String>> = vecs.iter();
let f : &Fn(&Vec<String>) -> Iter<String> = &|x: &Vec<String>| x.into_iter();
let b: FlatMap<Iter<Vec<String>>, Iter<String>, &Fn(&Vec<String>) -> Iter<String>> = a.flat_map(f);
let vals : Vec<&String> = b.collect();
vals.into_iter().map(|v: &String| v.to_owned()).collect()
}
I'd think about: why do you use iter() on the outer vec but into_iter() on the inner vecs? Using into_iter() is actually crucial, so that we don't have to copy first the inner vectors, then the strings inside, we just receive ownership of them.
We can actually write this just like a summation: concatenate the vectors two by two. Since we always reuse the allocation & contents of the same accumulation vector, this operation is linear time.
To minimize time spent growing and reallocating the vector, calculate the space needed up front.
fn concat_vecs(vecs: Vec<Vec<String>>) -> Vec<String> {
let size = vecs.iter().fold(0, |a, b| a + b.len());
vecs.into_iter().fold(Vec::with_capacity(size), |mut acc, v| {
acc.extend(v); acc
})
}
If you do want to clone all the contents, there's already a method for that, and you'd just use vecs.concat() /* -> Vec<String> */
The approach with .flat_map is fine, but if you don't want to clone the strings again you have to use .into_iter() on all levels: (x is Vec<String>).
vecs.into_iter().flat_map(|x| x.into_iter()).collect()
If instead you want to clone each string you can use this: (Changed .into_iter() to .iter() since x here is a &Vec<String> and both methods actually result in the same thing!)
vecs.iter().flat_map(|x| x.iter().map(Clone::clone)).collect()

Resources