How can I implement a Future which is woken up from another thread in Futures 0.2? - asynchronous

I'm trying to find a simple way to implement the Future trait from the future crate, version 0.2.1:
extern crate futures;
use futures::executor::ThreadPool;
use futures::prelude::*;
use futures::task::Context;
use std::{thread, time::Duration};
struct SendThree {
firstTime: bool,
}
impl Future for SendThree {
type Item = u32;
type Error = Never;
fn poll(&mut self, ctx: &mut Context) -> Result<Async<Self::Item>, Never> {
if self.firstTime {
self.firstTime = false;
thread::spawn(move || {
thread::sleep(Duration::from_millis(10));
ctx.waker().wake();
});
Ok(Async::Pending)
} else {
Ok(Async::Ready(3))
}
}
}
fn main() {
let mut fut = SendThree { firstTime: true };
let mut executor: ThreadPool = ThreadPool::new().unwrap();
let result = executor.run(fut).unwrap();
println!("{}", result);
}
playground
My problem is that the Context variable is not Send so I can't call wake from another thread:
error[E0277]: the trait bound `futures::executor::Executor: std::marker::Send` is not satisfied
--> src/main.rs:19:13
|
19 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `futures::executor::Executor` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `futures::executor::Executor`
= note: required because of the requirements on the impl of `std::marker::Send` for `&mut futures::executor::Executor`
= note: required because it appears within the type `std::option::Option<&mut futures::executor::Executor>`
= note: required because it appears within the type `futures::task::Context<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `&mut futures::task::Context<'_>`
= note: required because it appears within the type `[closure#src/main.rs:19:27: 22:14 ctx:&mut futures::task::Context<'_>]`
= note: required by `std::thread::spawn`
If I change the code to this it works, but I can't do a thread::sleep without blocking:
if self.firstTime {
self.firstTime = false;
ctx.waker().wake();
Ok(Async::Pending)
}
Is there an idiomatic way to implement this?

While Context isn't able to be sent across threads, a Waker is:
fn poll(&mut self, ctx: &mut Context) -> Result<Async<Self::Item>, Never> {
if self.first_time {
self.first_time = false;
let waker = ctx.waker().clone();
thread::spawn(move || {
thread::sleep(Duration::from_millis(10));
waker.wake();
});
Ok(Async::Pending)
} else {
Ok(Async::Ready(3))
}
}
Note that this is a really inefficient way to implement timeouts. There are Futures-native methods that should be used instead.

Related

How can I await on HashMap future values in rust?

I'm trying to kick off some async tasks in rust, then await them later in the code. Here's a simplified version of my code:
async fn my_async_fn() -> i64 {
return 0;
}
async fn main() {
let mut futures = HashMap::new();
futures.insert("a", my_async_fn());
// this is where I would do other work not blocked by these futures
let res_a = futures.get("a").expect("unreachable").await;
println!("my result: {}", res_a);
}
But when I try to run this, I get this self-contradictory message:
error[E0277]: `&impl futures::Future` is not a future
--> my/code:a:b
|
111 | let res_a = futures.get("a").expect("unreachable").await;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `&impl futures::Future` is not a future
|
= help: the trait `futures::Future` is not implemented for `&impl futures::Future`
= note: required by `futures::Future::poll`
How can I await futures that I've put into a HashMap? Or is there another way altogether?
Using await requires the Future is pinned and mutable.
use std::collections::HashMap;
async fn my_async_fn() -> i64 {
return 0;
}
#[tokio::main]
async fn main() {
let mut futures = HashMap::new();
futures.insert("a", Box::pin(my_async_fn()));
// this is where I would do other work not blocked by these futures
let res_a = futures.get_mut("a").expect("unreachable").await;
println!("my result: {}", res_a);
}

How to use predicates in an async function?

I use thirtyfour in my Rust script, and it uses tokio as the async runtime.
When I use find in a Vec::iter, it doesn't work as I expect:
#[tokio::main]
async fn main() -> WebDriverResult<()> {
let dropdown = driver.find_element(By::Tag("select")).await?;
dropdown
.find_elements(By::Tag("option"))
.await?
.iter()
.find(|&&x| x.text() == book_time.date) // Error, x.text() return a futures::Future type while book_time.date return a &str.
.click()
.await?;
}
After I tried Ibraheem Ahmed's solution, I met more errors:
let dropdown = driver.find_element(By::Tag("select")).await?;
let elements = dropdown.find_elements(By::Tag("option")).await?;
let stream = stream::iter(elements);
let elements = stream.filter(|x| async move { x.text().await.unwrap() == target_date });
error: lifetime may not live long enough
--> src\main.rs:125:38
|
125 | let elements = stream.filter(|x| async move { x.text().await.unwrap() == target_date });
| -- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
| ||
| |return type of closure is impl futures::Future
| has type `&'1 thirtyfour::WebElement<'_>`
There's a good thread on the Rust User's Forum that covers a similar question.
I tried the code snippets below by modifying the thirtyfour tokio_async example. I didn't have your full context, so I created an example that found a link based on its text on the wikipedia.org home page.
filter_map.
let target_value = "Terms of Use";
let found_elements: Vec<_> = stream
.filter_map(|x| async move {
if let Ok(text) = x.text().await {
if text == target_value {
println!("found");
return Some(x);
}
}
None
})
.collect()
.await;
while loop (which is probably not what you are after, but could be a simple solution if your logic fits easily inside)...
while let Some(element) = stream.next().await {
let text = element.text().await?;
println!("link text result: {}", text);
}
You can use a Stream, which is the asynchronous version of Iterator:
use futures::stream::{self, StreamExt};
fn main() {
// ...
let stream = stream::iter(elements).await?;
let elements = stream
.filter(|x| async move { x.text().await.as_deref() == Ok(book_time.date) })
.next()
.click()
.await?;
}

How to correctly call async functions in a WebSocket handler in Actix-web

I have made some progress with this, using into_actor().spawn(), but I am struggling to access the ctx variable inside the async block.
I'll start with showing a compiling snippet of the web socket handler, then a failing snippet of the handler, then for reference the full code example.
Working snippet:
Focus on the match case Ok(ws::Message::Text(text))
/// Handler for `ws::Message`
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWebSocket {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
// process websocket messages
println!("WS: {:?}", msg);
match msg {
Ok(ws::Message::Ping(msg)) => {
self.hb = Instant::now();
ctx.pong(&msg);
}
Ok(ws::Message::Pong(_)) => {
self.hb = Instant::now();
}
Ok(ws::Message::Text(text)) => {
let future = async move {
let reader = processrunner::run_process(text).await;
let mut reader = reader.ok().unwrap();
while let Some(line) = reader.next_line().await.unwrap() {
// ctx.text(line);
println!("line = {}", line);
}
};
future.into_actor(self).spawn(ctx);
}
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
Ok(ws::Message::Close(reason)) => {
ctx.close(reason);
ctx.stop();
}
_ => ctx.stop(),
}
}
}
Not working snippet with ctx line uncommented.
/// Handler for `ws::Message`
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWebSocket {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
// process websocket messages
println!("WS: {:?}", msg);
match msg {
Ok(ws::Message::Ping(msg)) => {
self.hb = Instant::now();
ctx.pong(&msg);
}
Ok(ws::Message::Pong(_)) => {
self.hb = Instant::now();
}
Ok(ws::Message::Text(text)) => {
let future = async move {
let reader = processrunner::run_process(text).await;
let mut reader = reader.ok().unwrap();
while let Some(line) = reader.next_line().await.unwrap() {
ctx.text(line);
println!("line = {}", line);
}
};
future.into_actor(self).spawn(ctx);
}
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
Ok(ws::Message::Close(reason)) => {
ctx.close(reason);
ctx.stop();
}
_ => ctx.stop(),
}
}
}
Full code snippet split over two files.
main.rs
//! Simple echo websocket server.
//! Open `http://localhost:8080/ws/index.html` in browser
//! or [python console client](https://github.com/actix/examples/blob/master/websocket/websocket-client.py)
//! could be used for testing.
mod processrunner;
use std::time::{Duration, Instant};
use actix::prelude::*;
use actix_files as fs;
use actix_web::{middleware, web, App, Error, HttpRequest, HttpResponse, HttpServer};
use actix_web_actors::ws;
/// How often heartbeat pings are sent
const HEARTBEAT_INTERVAL: Duration = Duration::from_secs(5);
/// How long before lack of client response causes a timeout
const CLIENT_TIMEOUT: Duration = Duration::from_secs(10);
/// do websocket handshake and start `MyWebSocket` actor
async fn ws_index(r: HttpRequest, stream: web::Payload) -> Result<HttpResponse, Error> {
println!("{:?}", r);
let res = ws::start(MyWebSocket::new(), &r, stream);
println!("{:?}", res);
res
}
/// websocket connection is long running connection, it easier
/// to handle with an actor
struct MyWebSocket {
/// Client must send ping at least once per 10 seconds (CLIENT_TIMEOUT),
/// otherwise we drop connection.
hb: Instant,
}
impl Actor for MyWebSocket {
type Context = ws::WebsocketContext<Self>;
/// Method is called on actor start. We start the heartbeat process here.
fn started(&mut self, ctx: &mut Self::Context) {
self.hb(ctx);
}
}
/// Handler for `ws::Message`
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWebSocket {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
// process websocket messages
println!("WS: {:?}", msg);
match msg {
Ok(ws::Message::Ping(msg)) => {
self.hb = Instant::now();
ctx.pong(&msg);
}
Ok(ws::Message::Pong(_)) => {
self.hb = Instant::now();
}
Ok(ws::Message::Text(text)) => {
let future = async move {
let reader = processrunner::run_process(text).await;
let mut reader = reader.ok().unwrap();
while let Some(line) = reader.next_line().await.unwrap() {
// ctx.text(line);
println!("line = {}", line);
}
};
future.into_actor(self).spawn(ctx);
}
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
Ok(ws::Message::Close(reason)) => {
ctx.close(reason);
ctx.stop();
}
_ => ctx.stop(),
}
}
}
impl MyWebSocket {
fn new() -> Self {
Self { hb: Instant::now() }
}
/// helper method that sends ping to client every second.
///
/// also this method checks heartbeats from client
fn hb(&self, ctx: &mut <Self as Actor>::Context) {
ctx.run_interval(HEARTBEAT_INTERVAL, |act, ctx| {
// check client heartbeats
if Instant::now().duration_since(act.hb) > CLIENT_TIMEOUT {
// heartbeat timed out
println!("Websocket Client heartbeat failed, disconnecting!");
// stop actor
ctx.stop();
// don't try to send a ping
return;
}
ctx.ping(b"");
});
}
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
std::env::set_var("RUST_LOG", "actix_server=info,actix_web=info");
env_logger::init();
HttpServer::new(|| {
App::new()
// enable logger
.wrap(middleware::Logger::default())
// websocket route
.service(web::resource("/ws/").route(web::get().to(ws_index)))
// static files
.service(fs::Files::new("/", "static/").index_file("index.html"))
})
// start http server on 127.0.0.1:8080
.bind("127.0.0.1:8080")?
.run()
.await
}
processrunner.rs
extern crate tokio;
use tokio::io::*;
use tokio::process::Command;
use std::process::Stdio;
//#[tokio::main]
pub async fn run_process(
text: String,
) -> std::result::Result<
tokio::io::Lines<BufReader<tokio::process::ChildStdout>>,
Box<dyn std::error::Error>,
> {
let mut cmd = Command::new(text);
cmd.stdout(Stdio::piped());
let mut child = cmd.spawn().expect("failed to spawn command");
let stdout = child
.stdout
.take()
.expect("child did not have a handle to stdout");
let lines = BufReader::new(stdout).lines();
// Ensure the child process is spawned in the runtime so it can
// make progress on its own while we await for any output.
tokio::spawn(async {
let status = child.await.expect("child process encountered an error");
println!("child status was: {}", status);
});
Ok(lines)
}
Error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:57:41
|
57 | let future = async move {
| _________________________________________^
58 | | let reader = processrunner::run_process(text).await;
59 | | let mut reader = reader.ok().unwrap();
60 | | while let Some(line) = reader.next_line().await.unwrap() {
... |
63 | | }
64 | | };
| |_________________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #2 defined on the method body at 45:5...
--> src/main.rs:45:5
|
45 | / fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
46 | | // process websocket messages
47 | | println!("WS: {:?}", msg);
48 | | match msg {
... |
74 | | }
75 | | }
| |_____^
note: ...so that the types are compatible
--> src/main.rs:57:41
|
57 | let future = async move {
| _________________________________________^
58 | | let reader = processrunner::run_process(text).await;
59 | | let mut reader = reader.ok().unwrap();
60 | | while let Some(line) = reader.next_line().await.unwrap() {
... |
63 | | }
64 | | };
| |_________________^
= note: expected `&mut actix_web_actors::ws::WebsocketContext<MyWebSocket>`
found `&mut actix_web_actors::ws::WebsocketContext<MyWebSocket>`
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `actix::fut::FutureWrap<impl std::future::Future, MyWebSocket>` will meet its required lifetime bounds
--> src/main.rs:66:41
|
66 | future.into_actor(self).spawn(ctx);
| ^^^^^
error: aborting due to previous error
For more information about this error, try `rustc --explain E0495`.
cargo
[package]
name = "removed"
version = "0.1.0"
authors = ["removed"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
tokio = { version = "0.2", features = ["full"] }
actix = "0.10"
actix-codec = "0.3"
actix-web = "3"
actix-web-actors = "3"
actix-files = "0.3"
awc = "2"
env_logger = "0.7"
futures = "0.3.1"
bytes = "0.5.3"
Here are the basics. You may need to do a little work here and there but this works.
use actix::prelude::*;
use tokio::process::Command;
use actix_web::{ web, App, Error, HttpRequest, HttpResponse, HttpServer};
use actix_web_actors::ws;
use tokio::io::{ AsyncBufReadExt};
use actix::AsyncContext;
use tokio::stream::{ StreamExt};
use tokio::io::{BufReader};
use std::process::Stdio;
#[derive(Message)]
#[rtype(result = "Result<(), ()>")]
struct CommandRunner(String);
/// Define HTTP actor
struct MyWs;
impl Actor for MyWs {
type Context = ws::WebsocketContext<Self>;
}
#[derive(Debug)]
struct Line(String);
impl StreamHandler<Result<Line, ws::ProtocolError>> for MyWs {
fn handle(
&mut self,
msg: Result<Line, ws::ProtocolError>,
ctx: &mut Self::Context,
) {
match msg {
Ok(line) => ctx.text(line.0),
_ => () //Handle errors
}
}
}
/// Handler for ws::Message message
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWs {
fn handle(
&mut self,
msg: Result<ws::Message, ws::ProtocolError>,
ctx: &mut Self::Context,
) {
match msg {
Ok(ws::Message::Ping(msg)) => ctx.pong(&msg),
Ok(ws::Message::Text(text)) => {
ctx.notify(CommandRunner(text.to_string()));
},
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
_ => (),
}
}
}
impl Handler<CommandRunner> for MyWs {
type Result = Result<(), ()>;
fn handle(&mut self, msg: CommandRunner, ctx: &mut Self::Context) -> Self::Result {
let mut cmd = Command::new(msg.0);
// Specify that we want the command's standard output piped back to us.
// By default, standard input/output/error will be inherited from the
// current process (for example, this means that standard input will
// come from the keyboard and standard output/error will go directly to
// the terminal if this process is invoked from the command line).
cmd.stdout(Stdio::piped());
let mut child = cmd.spawn()
.expect("failed to spawn command");
let stdout = child.stdout.take()
.expect("child did not have a handle to stdout");
let reader = BufReader::new(stdout).lines();
// Ensure the child process is spawned in the runtime so it can
// make progress on its own while we await for any output.
let fut = async move {
let status = child.await
.expect("child process encountered an error");
println!("child status was: {}", status);
};
let fut = actix::fut::wrap_future::<_, Self>(fut);
ctx.spawn(fut);
ctx.add_stream(reader.map(|l| Ok(Line(l.expect("Not a line")))));
Ok(())
}
}
async fn index(req: HttpRequest, stream: web::Payload) -> Result<HttpResponse, Error> {
let resp = ws::start(MyWs {}, &req, stream);
println!("{:?}", resp);
resp
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
HttpServer::new(|| App::new().route("/ws/", web::get().to(index)))
.bind("127.0.0.1:8080")?
.run()
.await
}
Running ls Looks like this.
So I just understood what was going wrong at the same time that I discovered the accepted answer.
The accepted answer proposes a clean solution but I thought I would pose an alternative view point, the code snippet I propose below makes fewer changes to my original attempt (as shown in the question) in the hope that it will demonstrate my fundamental miss understanding.
The fundamental issue with my code is that I was ignoring the rule that "every actor has its own context". As you see from the compile error in the question, luckily Actix uses the rust compiler to enforce this rule.
Now that I understand that, it looks like the wrong thing I was trying to do is to spawn another actor and have that actor somehow move/copy in the original actor's context, just so it could respond with the process output lines. There is no need to do this of course, because the Actor model is all about letting Actors communicate by messages.
Instead, when spawning a new actor, I should have passed it the address of the original actor, allowing the newly spawned actor to send updates back. The original actor handles these messages (struct Line below) using a handler.
As I said, the accepted answer also does this but using a mapper which looks like a more elegant solution than my loop.
mod processrunner;
use std::time::{Duration, Instant};
use actix::prelude::*;
use actix_files as fs;
use actix_web::{middleware, web, App, Error, HttpRequest, HttpResponse, HttpServer};
use actix_web_actors::ws;
/// How often heartbeat pings are sent
const HEARTBEAT_INTERVAL: Duration = Duration::from_secs(5);
/// How long before lack of client response causes a timeout
const CLIENT_TIMEOUT: Duration = Duration::from_secs(10);
/// do websocket handshake and start `MyWebSocket` actor
async fn ws_index(r: HttpRequest, stream: web::Payload) -> Result<HttpResponse, Error> {
println!("{:?}", r);
let res = ws::start(MyWebSocket::new(), &r, stream);
println!("{:?}", res);
res
}
/// websocket connection is long running connection, it easier
/// to handle with an actor
struct MyWebSocket {
/// Client must send ping at least once per 10 seconds (CLIENT_TIMEOUT),
/// otherwise we drop connection.
hb: Instant,
}
impl Actor for MyWebSocket {
type Context = ws::WebsocketContext<Self>;
/// Method is called on actor start. We start the heartbeat process here.
fn started(&mut self, ctx: &mut Self::Context) {
self.hb(ctx);
}
}
#[derive(Message)]
#[rtype(result = "()")]
pub struct Line {
line: String,
}
impl Handler<Line> for MyWebSocket {
type Result = ();
fn handle(&mut self, msg: Line, ctx: &mut Self::Context) {
ctx.text(msg.line);
}
}
/// Handler for `ws::Message`
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWebSocket {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
// process websocket messages
println!("WS: {:?}", msg);
match msg {
Ok(ws::Message::Ping(msg)) => {
self.hb = Instant::now();
ctx.pong(&msg);
}
Ok(ws::Message::Pong(_)) => {
self.hb = Instant::now();
}
Ok(ws::Message::Text(text)) => {
let recipient = ctx.address().recipient();
let future = async move {
let reader = processrunner::run_process(text).await;
let mut reader = reader.ok().unwrap();
while let Some(line) = reader.next_line().await.unwrap() {
println!("line = {}", line);
recipient.do_send(Line { line });
}
};
future.into_actor(self).spawn(ctx);
}
Ok(ws::Message::Binary(bin)) => ctx.binary(bin),
Ok(ws::Message::Close(reason)) => {
ctx.close(reason);
ctx.stop();
}
_ => ctx.stop(),
}
}
}
impl MyWebSocket {
fn new() -> Self {
Self { hb: Instant::now() }
}
/// helper method that sends ping to client every second.
///
/// also this method checks heartbeats from client
fn hb(&self, ctx: &mut <Self as Actor>::Context) {
ctx.run_interval(HEARTBEAT_INTERVAL, |act, ctx| {
// check client heartbeats
if Instant::now().duration_since(act.hb) > CLIENT_TIMEOUT {
// heartbeat timed out
println!("Websocket Client heartbeat failed, disconnecting!");
// stop actor
ctx.stop();
// don't try to send a ping
return;
}
ctx.ping(b"");
});
}
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
std::env::set_var("RUST_LOG", "actix_server=info,actix_web=info");
env_logger::init();
HttpServer::new(|| {
App::new()
// enable logger
.wrap(middleware::Logger::default())
// websocket route
.service(web::resource("/ws/").route(web::get().to(ws_index)))
// static files
.service(fs::Files::new("/", "static/").index_file("index.html"))
})
// start http server on 127.0.0.1:8080
.bind("127.0.0.1:8080")?
.run()
.await
}

How can I mutate the HTML inside a hyper::Response? [duplicate]

I want to write a server using the current master branch of Hyper that saves a message that is delivered by a POST request and sends this message to every incoming GET request.
I have this, mostly copied from the Hyper examples directory:
extern crate futures;
extern crate hyper;
extern crate pretty_env_logger;
use futures::future::FutureResult;
use hyper::{Get, Post, StatusCode};
use hyper::header::{ContentLength};
use hyper::server::{Http, Service, Request, Response};
use futures::Stream;
struct Echo {
data: Vec<u8>,
}
impl Echo {
fn new() -> Self {
Echo {
data: "text".into(),
}
}
}
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = FutureResult<Response, hyper::Error>;
fn call(&self, req: Self::Request) -> Self::Future {
let resp = match (req.method(), req.path()) {
(&Get, "/") | (&Get, "/echo") => {
Response::new()
.with_header(ContentLength(self.data.len() as u64))
.with_body(self.data.clone())
},
(&Post, "/") => {
//self.data.clear(); // argh. &self is not mutable :(
// even if it was mutable... how to put the entire body into it?
//req.body().fold(...) ?
let mut res = Response::new();
if let Some(len) = req.headers().get::<ContentLength>() {
res.headers_mut().set(ContentLength(0));
}
res.with_body(req.body())
},
_ => {
Response::new()
.with_status(StatusCode::NotFound)
}
};
futures::future::ok(resp)
}
}
fn main() {
pretty_env_logger::init().unwrap();
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo::new())).unwrap();
println!("Listening on http://{} with 1 thread.", server.local_addr().unwrap());
server.run().unwrap();
}
How do I turn the req.body() (which seems to be a Stream of Chunks) into a Vec<u8>? I assume I must somehow return a Future that consumes the Stream and turns it into a single Vec<u8>, maybe with fold(). But I have no clue how to do that.
Hyper 0.13 provides a body::to_bytes function for this purpose.
use hyper::body;
use hyper::{Body, Response};
pub async fn read_response_body(res: Response<Body>) -> Result<String, hyper::Error> {
let bytes = body::to_bytes(res.into_body()).await?;
Ok(String::from_utf8(bytes.to_vec()).expect("response was not valid utf-8"))
}
I'm going to simplify the problem to just return the total number of bytes, instead of echoing the entire stream.
Futures 0.3
Hyper 0.13 + TryStreamExt::try_fold
See euclio's answer about hyper::body::to_bytes if you just want all the data as one giant blob.
Accessing the stream allows for more fine-grained control:
use futures::TryStreamExt; // 0.3.7
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.13.9
use std::convert::Infallible;
use tokio; // 0.2.22
#[tokio::main]
async fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(service::make_service_fn(|_conn| async {
Ok::<_, Infallible>(service::service_fn(echo))
}));
println!("Listening on http://{}.", server.local_addr());
if let Err(e) = server.await {
eprintln!("Error: {}", e);
}
}
async fn echo(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body
.try_fold(Vec::new(), |mut data, chunk| async move {
data.extend_from_slice(&chunk);
Ok(data)
})
.await;
entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
})
}
_ => {
let body = Body::from("Can only POST to /");
Ok(Response::new(body))
}
}
}
Unfortunately, the current implementation of Bytes is no longer compatible with TryStreamExt::try_concat, so we have to switch back to a fold.
Futures 0.1
hyper 0.12 + Stream::concat2
Since futures 0.1.14, you can use Stream::concat2 to stick together all the data into one:
fn concat2(self) -> Concat2<Self>
where
Self: Sized,
Self::Item: Extend<<Self::Item as IntoIterator>::Item> + IntoIterator + Default,
use futures::{
future::{self, Either},
Future, Stream,
}; // 0.1.25
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.12.20
use tokio; // 0.1.14
fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(|| service::service_fn(echo));
println!("Listening on http://{}.", server.local_addr());
let server = server.map_err(|e| eprintln!("Error: {}", e));
tokio::run(server);
}
fn echo(req: Request<Body>) -> impl Future<Item = Response<Body>, Error = hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body.concat2();
let resp = entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
});
Either::A(resp)
}
_ => {
let body = Body::from("Can only POST to /");
let resp = future::ok(Response::new(body));
Either::B(resp)
}
}
}
You could also convert the Bytes into a Vec<u8> via entire_body.to_vec() and then convert that to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
hyper 0.11 + Stream::fold
Similar to Iterator::fold, Stream::fold takes an accumulator (called init) and a function that operates on the accumulator and an item from the stream. The result of the function must be another future with the same error type as the original. The total result is itself a future.
fn fold<F, T, Fut>(self, init: T, f: F) -> Fold<Self, F, Fut, T>
where
F: FnMut(T, Self::Item) -> Fut,
Fut: IntoFuture<Item = T>,
Self::Error: From<Fut::Error>,
Self: Sized,
We can use a Vec as the accumulator. Body's Stream implementation returns a Chunk. This implements Deref<[u8]>, so we can use that to append each chunk's data to the Vec.
extern crate futures; // 0.1.23
extern crate hyper; // 0.11.27
use futures::{Future, Stream};
use hyper::{
server::{Http, Request, Response, Service}, Post,
};
fn main() {
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo)).unwrap();
println!(
"Listening on http://{} with 1 thread.",
server.local_addr().unwrap()
);
server.run().unwrap();
}
struct Echo;
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = Box<futures::Future<Item = Response, Error = Self::Error>>;
fn call(&self, req: Self::Request) -> Self::Future {
match (req.method(), req.path()) {
(&Post, "/") => {
let f = req.body()
.fold(Vec::new(), |mut acc, chunk| {
acc.extend_from_slice(&*chunk);
futures::future::ok::<_, Self::Error>(acc)
})
.map(|body| Response::new().with_body(format!("Read {} bytes", body.len())));
Box::new(f)
}
_ => panic!("Nope"),
}
}
}
You could also convert the Vec<u8> body to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
Output
When called from the command line, we can see the result:
$ curl -X POST --data hello http://127.0.0.1:12346/
Read 5 bytes
Warning
All of these solutions allow a malicious end user to POST an infinitely sized file, which would cause the machine to run out of memory. Depending on the intended use, you may wish to establish some kind of cap on the number of bytes read, potentially writing to the filesystem at some breakpoint.
See also:
How do I apply a limit to the number of bytes read by futures::Stream::concat2?
Most of the answers on this topic are outdated or overly complicated. The solution is pretty simple:
/*
WARNING for beginners!!! This use statement
is important so we can later use .data() method!!!
*/
use hyper::body::HttpBody;
let my_vector: Vec<u8> = request.into_body().data().await.unwrap().unwrap().to_vec();
let my_string = String::from_utf8(my_vector).unwrap();
You can also use body::to_bytes as #euclio answered. Both approaches are straight-forward! Don't forget to handle unwrap properly.

How to implement a Future or Stream that polls an async fn?

I have a struct Test I want to implement std::future::Future that would poll function:
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
struct Test;
impl Test {
async fn function(&mut self) {}
}
impl Future for Test {
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
match self.function() {
Poll::Pending => Poll::Pending,
Poll::Ready(_) => Poll::Ready(()),
}
}
}
That didn't work:
error[E0308]: mismatched types
--> src/lib.rs:17:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
17 | Poll::Pending => Poll::Pending,
| ^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
error[E0308]: mismatched types
--> src/lib.rs:18:13
|
10 | async fn function(&mut self) {}
| - the `Output` of this `async fn`'s expected opaque type
...
18 | Poll::Ready(_) => Poll::Ready(()),
| ^^^^^^^^^^^^^^ expected opaque type, found enum `Poll`
|
= note: expected opaque type `impl Future`
found enum `Poll<_>`
I understand that function must be called once, the returned Future must be stored somewhere in the struct, and then the saved future must be polled. I tried this:
struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
impl Test {
async fn function(&mut self) {}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(Box::pin(s.function()));
s
}
}
That also didn't work:
error[E0277]: the size for values of type `(dyn Future<Output = ()> + 'static)` cannot be known at compilation time
--> src/lib.rs:7:13
|
7 | struct Test(Option<Box<Pin<dyn Future<Output = ()>>>>);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `Sized` is not implemented for `(dyn Future<Output = ()> + 'static)`
After I call function() I have taken a &mut reference of Test, because of that I can't change the Test variable, and therefore can't store the returned Future inside the Test.
I did get an unsafe solution (inspired by this)
struct Test<'a>(Option<BoxFuture<'a, ()>>);
impl Test<'_> {
async fn function(&mut self) {
println!("I'm alive!");
}
fn new() -> Self {
let mut s = Self(None);
s.0 = Some(unsafe { &mut *(&mut s as *mut Self) }.function().boxed());
s
}
}
impl Future for Test<'_> {
type Output = ();
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
self.0.as_mut().unwrap().poll_unpin(cx)
}
}
I hope that there is another way.
Though there are times when you may want to do things similar to what you're trying to accomplish here, they are a rarity. So most people reading this, maybe even OP, may wish to restructure such that struct state and data used for a single async execution are different objects.
To answer your question, yes it is somewhat possible. Unless you want to absolutely resort to unsafe code you will need to use Mutex and Arc. All fields you wish to manipulate inside the async fn will have to be wrapped inside a Mutex and the function itself will accept an Arc<Self>.
I must stress, however, that this is not a beautiful solution and you probably don't want to do this. Depending on your specific case your solution may vary, but my guess of what OP is trying to accomplish while using Streams would be better solved by something similar to this gist that I wrote.
use std::{
future::Future,
pin::Pin,
sync::{Arc, Mutex},
};
struct Test {
state: Mutex<Option<Pin<Box<dyn Future<Output = ()>>>>>,
// if available use your async library's Mutex to `.await` locks on `buffer` instead
buffer: Mutex<Vec<u8>>,
}
impl Test {
async fn function(self: Arc<Self>) {
for i in 0..16u8 {
let data: Vec<u8> = vec![i]; // = fs::read(&format("file-{}.txt", i)).await.unwrap();
let mut buflock = self.buffer.lock().unwrap();
buflock.extend_from_slice(&data);
}
}
pub fn new() -> Arc<Self> {
let s = Arc::new(Self {
state: Default::default(),
buffer: Default::default(),
});
{
// start by trying to aquire a lock to the Mutex of the Box
let mut lock = s.state.lock().unwrap();
// create boxed future
let b = Box::pin(s.clone().function());
// insert value into the mutex
*lock = Some(b);
} // block causes the lock to be released
s
}
}
impl Future for Test {
type Output = ();
fn poll(
self: std::pin::Pin<&mut Self>,
ctx: &mut std::task::Context<'_>,
) -> std::task::Poll<<Self as std::future::Future>::Output> {
let mut lock = self.state.lock().unwrap();
let fut: &mut Pin<Box<dyn Future<Output = ()>>> = lock.as_mut().unwrap();
Future::poll(fut.as_mut(), ctx)
}
}
I'm not sure what you want to achieve and why, but I suspect that you're trying to implement Future for Test based on some ancient tutorial or misunderstanding and just overcomplicating things.
You don't have to implement Future manually. An async function
async fn function(...) {...}
is really just syntax sugar translated behind the scenes into something like
fn function(...) -> Future<()> {...}
All you have to do is to use the result of the function the same way as any future, e.g. use await on it or call block a reactor until it's finished. E.g. based on your first version, you can simply call:
let mut test = Test{};
test.function().await;
UPDATE1
Based on your descriptions I still think you're trying to overcomplicate this minimal working snippet without the need to manually implement Future for anything:
async fn asyncio() { println!("Doing async IO"); }
struct Test {
count: u32,
}
impl Test {
async fn function(&mut self) {
asyncio().await;
self.count += 1;
}
}
#[tokio::main]
async fn main() {
let mut test = Test{count: 0};
test.function().await;
println!("Count: {}", test.count);
}

Resources