I want to write a server using the current master branch of Hyper that saves a message that is delivered by a POST request and sends this message to every incoming GET request.
I have this, mostly copied from the Hyper examples directory:
extern crate futures;
extern crate hyper;
extern crate pretty_env_logger;
use futures::future::FutureResult;
use hyper::{Get, Post, StatusCode};
use hyper::header::{ContentLength};
use hyper::server::{Http, Service, Request, Response};
use futures::Stream;
struct Echo {
data: Vec<u8>,
}
impl Echo {
fn new() -> Self {
Echo {
data: "text".into(),
}
}
}
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = FutureResult<Response, hyper::Error>;
fn call(&self, req: Self::Request) -> Self::Future {
let resp = match (req.method(), req.path()) {
(&Get, "/") | (&Get, "/echo") => {
Response::new()
.with_header(ContentLength(self.data.len() as u64))
.with_body(self.data.clone())
},
(&Post, "/") => {
//self.data.clear(); // argh. &self is not mutable :(
// even if it was mutable... how to put the entire body into it?
//req.body().fold(...) ?
let mut res = Response::new();
if let Some(len) = req.headers().get::<ContentLength>() {
res.headers_mut().set(ContentLength(0));
}
res.with_body(req.body())
},
_ => {
Response::new()
.with_status(StatusCode::NotFound)
}
};
futures::future::ok(resp)
}
}
fn main() {
pretty_env_logger::init().unwrap();
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo::new())).unwrap();
println!("Listening on http://{} with 1 thread.", server.local_addr().unwrap());
server.run().unwrap();
}
How do I turn the req.body() (which seems to be a Stream of Chunks) into a Vec<u8>? I assume I must somehow return a Future that consumes the Stream and turns it into a single Vec<u8>, maybe with fold(). But I have no clue how to do that.
Hyper 0.13 provides a body::to_bytes function for this purpose.
use hyper::body;
use hyper::{Body, Response};
pub async fn read_response_body(res: Response<Body>) -> Result<String, hyper::Error> {
let bytes = body::to_bytes(res.into_body()).await?;
Ok(String::from_utf8(bytes.to_vec()).expect("response was not valid utf-8"))
}
I'm going to simplify the problem to just return the total number of bytes, instead of echoing the entire stream.
Futures 0.3
Hyper 0.13 + TryStreamExt::try_fold
See euclio's answer about hyper::body::to_bytes if you just want all the data as one giant blob.
Accessing the stream allows for more fine-grained control:
use futures::TryStreamExt; // 0.3.7
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.13.9
use std::convert::Infallible;
use tokio; // 0.2.22
#[tokio::main]
async fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(service::make_service_fn(|_conn| async {
Ok::<_, Infallible>(service::service_fn(echo))
}));
println!("Listening on http://{}.", server.local_addr());
if let Err(e) = server.await {
eprintln!("Error: {}", e);
}
}
async fn echo(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body
.try_fold(Vec::new(), |mut data, chunk| async move {
data.extend_from_slice(&chunk);
Ok(data)
})
.await;
entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
})
}
_ => {
let body = Body::from("Can only POST to /");
Ok(Response::new(body))
}
}
}
Unfortunately, the current implementation of Bytes is no longer compatible with TryStreamExt::try_concat, so we have to switch back to a fold.
Futures 0.1
hyper 0.12 + Stream::concat2
Since futures 0.1.14, you can use Stream::concat2 to stick together all the data into one:
fn concat2(self) -> Concat2<Self>
where
Self: Sized,
Self::Item: Extend<<Self::Item as IntoIterator>::Item> + IntoIterator + Default,
use futures::{
future::{self, Either},
Future, Stream,
}; // 0.1.25
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.12.20
use tokio; // 0.1.14
fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(|| service::service_fn(echo));
println!("Listening on http://{}.", server.local_addr());
let server = server.map_err(|e| eprintln!("Error: {}", e));
tokio::run(server);
}
fn echo(req: Request<Body>) -> impl Future<Item = Response<Body>, Error = hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body.concat2();
let resp = entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
});
Either::A(resp)
}
_ => {
let body = Body::from("Can only POST to /");
let resp = future::ok(Response::new(body));
Either::B(resp)
}
}
}
You could also convert the Bytes into a Vec<u8> via entire_body.to_vec() and then convert that to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
hyper 0.11 + Stream::fold
Similar to Iterator::fold, Stream::fold takes an accumulator (called init) and a function that operates on the accumulator and an item from the stream. The result of the function must be another future with the same error type as the original. The total result is itself a future.
fn fold<F, T, Fut>(self, init: T, f: F) -> Fold<Self, F, Fut, T>
where
F: FnMut(T, Self::Item) -> Fut,
Fut: IntoFuture<Item = T>,
Self::Error: From<Fut::Error>,
Self: Sized,
We can use a Vec as the accumulator. Body's Stream implementation returns a Chunk. This implements Deref<[u8]>, so we can use that to append each chunk's data to the Vec.
extern crate futures; // 0.1.23
extern crate hyper; // 0.11.27
use futures::{Future, Stream};
use hyper::{
server::{Http, Request, Response, Service}, Post,
};
fn main() {
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo)).unwrap();
println!(
"Listening on http://{} with 1 thread.",
server.local_addr().unwrap()
);
server.run().unwrap();
}
struct Echo;
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = Box<futures::Future<Item = Response, Error = Self::Error>>;
fn call(&self, req: Self::Request) -> Self::Future {
match (req.method(), req.path()) {
(&Post, "/") => {
let f = req.body()
.fold(Vec::new(), |mut acc, chunk| {
acc.extend_from_slice(&*chunk);
futures::future::ok::<_, Self::Error>(acc)
})
.map(|body| Response::new().with_body(format!("Read {} bytes", body.len())));
Box::new(f)
}
_ => panic!("Nope"),
}
}
}
You could also convert the Vec<u8> body to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
Output
When called from the command line, we can see the result:
$ curl -X POST --data hello http://127.0.0.1:12346/
Read 5 bytes
Warning
All of these solutions allow a malicious end user to POST an infinitely sized file, which would cause the machine to run out of memory. Depending on the intended use, you may wish to establish some kind of cap on the number of bytes read, potentially writing to the filesystem at some breakpoint.
See also:
How do I apply a limit to the number of bytes read by futures::Stream::concat2?
Most of the answers on this topic are outdated or overly complicated. The solution is pretty simple:
/*
WARNING for beginners!!! This use statement
is important so we can later use .data() method!!!
*/
use hyper::body::HttpBody;
let my_vector: Vec<u8> = request.into_body().data().await.unwrap().unwrap().to_vec();
let my_string = String::from_utf8(my_vector).unwrap();
You can also use body::to_bytes as #euclio answered. Both approaches are straight-forward! Don't forget to handle unwrap properly.
Related
I am getting data from the open weather map API. Currently the data is being retrieved synchronously which is slow. However, the function has to be synchronous as it is part of a library, but it can call an async function. How might I still make concurrent requests to increase performance? A solution that does not use reqwests works, but reqwests is preferred.
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let mut data: Vec<HashMap<String, String>> = Vec::new();
for url in urls {
let request = reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected");
data.push(request);
}
return data;
}
If your program isn't already async, probably the easiest way might be to use rayon.
use reqwest;
use std::collections::HashMap;
use rayon::prelude::*;
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let data : Vec<_>= urls
.par_iter()
.map(|&url| reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected"))
.collect();
return data;
}
The easiest is probably to use tokios new_current_thread runtime and blocking on the data retreival.
use std::collections::HashMap;
use tokio::runtime;
pub fn collect_data() -> Vec<HashMap<String, String>> {
let rt = runtime::Builder::new_current_thread()
.build()
.expect("couldn't start runtime");
let urls = vec!["https://example.com/a", "https://example.com/b"];
rt.block_on(async move {
let mut data = vec![];
for url in urls {
data.push(async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
});
}
futures::future::join_all(data).await
})
}
You need an asynchronous runtime in order to call asynchronous functions. The easiest way to get one is to use the #[tokio::main] attribute (which despite the name can be applied to any function):
#[tokio::main]
fn get_combined_data(
open_weather_map_api_url: String,
open_weather_map_api_key: String,
coordinates: Vec<String>,
metric: bool,
) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(
open_weather_map_api_url,
open_weather_map_api_key,
coordinates
.get(0)
.expect("Improper coordinates")
.to_string()
+ ","
+ coordinates.get(1).expect("Improper coordinates"),
metric,
);
futures::future::join_all (urls.map (|u| {
async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
}
})).await
}
This question already has answers here:
How to use async/await in Rust when you can't make main function async
(4 answers)
How do I synchronously return a value calculated in an asynchronous Future?
(3 answers)
Closed 3 months ago.
I have a program that does various simple things based on user selection.
fn main() {
let mut user_input = String::new(); // Initialize variable to store user input
println!("Select an option:\r\n[1] Get sysinfo\r\n[2] Read/Write File\r\n[3] Download file\r\n[4] Exit"); // Print options
io::stdin().read_line(&mut user_input).expect("You entered something weird you donkey!"); // Get user input and store in variable
let int_input = user_input.trim().parse::<i32>().unwrap(); // Convert user input to int (i32 means signed integer 32 bits)
match int_input { // If Else statement
1 => getsysinfo(), // If int_input == 1, call getsysinfo()
2 => readwritefile(),
3 => downloadfile(),
4 => process::exit(1), // end program
_ => println!("You didn't choose one of the given options, you donkey!") // input validation
}
}
My function downloadfile() looks like this, referenced from the Rust cookbook on downloading files.
error_chain! {
foreign_links {
Io(std::io::Error);
HttpRequest(reqwest::Error);
}
}
async fn downloadfile() -> Result<()> {
let tmp_dir = Builder::new().prefix("example").tempdir()?;
let target = "localhost:8000/downloaded.txt";
let response = reqwest::get(target).await?;
let mut dest = {
let fname = response
.url()
.path_segments()
.and_then(|segments| segments.last())
.and_then(|name| if name.is_empty() { None } else { Some(name) })
.unwrap_or("tmp.bin");
println!("File to download: {}", fname);
let fname = tmp_dir.path().join(fname);
println!("Will be located under: {:?}", fname);
File::create(fname)?
};
let content = response.text().await?;
copy(&mut content.as_bytes(), &mut dest)?;
Ok(())
}
I get the following error:
`match` arms have incompatible types
expected unit type `()`
found opaque type `impl Future<Output = std::result::Result<(), Error>>`
I presume its because the async function returns a Future type, so how can I make this code work?
You need to use the block_on function.
Add futures as a dependency in your cargo.toml for the following example to work.
use futures::executor::block_on;
async fn hello() -> String {
return String::from("Hello world!");
}
fn main() {
let output = block_on(hello());
println!("{output}");
}
I am trying to build an program that performs async http requests and returns their Response + body.
This is what the function that returns the Response looks like:
let responses = stream::iter(urls)
.map(|line| {
let client = &client;
async move {
client.get(&line).send().await.map(|resp| {
(line, resp)
})}
})
.buffer_unordered(concurrency_amount);
However, after returning resp, I can't use resp.text(), since resp.text() is of type Future<Output=Result<String>>.
How can I make the function also return resp.text() in the tuple?
Assuming that you are using reqwest, it should be possible to collect the bytes of the response body with Response::chunk(), text() consumes self but chunk() only takes a mutable reference.
Something like the following collects the response body and decodes it to a String in a lossy manner.
use futures_util::StreamExt;
#[tokio::main]
async fn main() {
let cli = reqwest::Client::new();
let urls = vec![
"https://stackoverflow.com".to_string(),
"https://google.com".into(),
"https://tokio.rs".into(),
];
let responses = futures_util::stream::iter(urls.into_iter())
.then(|url| { // note that I replaced `map` with `then` here.
let cli = cli.clone();
async move {
let mut resp = cli.get(url.clone()).send().await.unwrap();
let mut body = Vec::new();
while let Some(chunk) = resp.chunk().await.unwrap() {
body.extend_from_slice(&*chunk);
}
(url, resp, String::from_utf8_lossy(&body).to_string())
}
})
.collect::<Vec<_>>()
.await;
for (url, response, text) in responses {
println!("url: {} status: {} text: {}", url, response.status(), text);
}
}
As the in-line comment notes: I changed the map() call to then() so the stream yields the tuples rather than futures with the tuples as output.
Does this work?
let responses = stream::iter(urls)
.map(|line| {
let client = &client;
(async move || {
let resp = client.get(&line).send().await?;
let text = resp.text().await?;
(line, resp, text)
})()
})
.buffer_unordered(concurrency_amount);
I am trying to rewrite the proxy example of Asynchronous Programming in Rust book by migrating to :
futures-preview = { version = "0.3.0-alpha.19", features = ["async-await"]}`
hyper = "0.13.0-alpha.4"`
from:
futures-preview = { version = "=0.3.0-alpha.17", features = ["compat"] }`
hyper = "0.12.9"
The current example converts the returned Future from a futures 0.3 into a futures 0.1, because hyper = "0.12.9" is not compatible with futures 0.3's async/await.
My code:
use {
futures::future::{FutureExt, TryFutureExt},
hyper::{
rt::run,
service::{make_service_fn, service_fn},
Body, Client, Error, Request, Response, Server, Uri,
},
std::net::SocketAddr,
std::str::FromStr,
};
fn forward_uri<B>(forward_url: &'static str, req: &Request<B>) -> Uri {
let forward_uri = match req.uri().query() {
Some(query) => format!("{}{}?{}", forward_url, req.uri().path(), query),
None => format!("{}{}", forward_url, req.uri().path()),
};
Uri::from_str(forward_uri.as_str()).unwrap()
}
async fn call(
forward_url: &'static str,
mut _req: Request<Body>,
) -> Result<Response<Body>, hyper::Error> {
*_req.uri_mut() = forward_uri(forward_url, &_req);
let url_str = forward_uri(forward_url, &_req);
let res = Client::new().get(url_str).await;
res
}
async fn run_server(forward_url: &'static str, addr: SocketAddr) {
let forwarded_url = forward_url;
let serve_future = service_fn(move |req| call(forwarded_url, req).boxed());
let server = Server::bind(&addr).serve(serve_future);
if let Err(err) = server.await {
eprintln!("server error: {}", err);
}
}
fn main() {
// Set the address to run our socket on.
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
let url = "http://127.0.0.1:9061";
let futures_03_future = run_server(url, addr);
run(futures_03_future);
}
First, I receive this error for server in run_server function:
the trait tower_service::Service<&'a
hyper::server::tcp::addr_stream::AddrStream> is not implemented for
hyper::service::service::ServiceFn<[closure#src/main.rs:35:35: 35:78
forwarded_url:_], hyper::body::body::Body>
Also, I cannot use hyper::rt::run because it might have been implemented differently in hyper = 0.13.0-alpha.4.
I will be grateful if you tell me your ideas on how to fix it.
By this issue, to create a new service for each connection you need to create MakeService in hyper = "0.13.0-alpha.4". You can create MakeService with a closure by using make_service_fn.
Also, I cannot use hyper::rt::run because it might have been implemented differently in hyper = 0.13.0-alpha.4.
Correct, under the hood hyper::rt::run was calling tokio::run, it has been removed from the api but currently i don't know the reason. You can run your future with calling tokio::run by yourself or use #[tokio::main] annotation. To do this you need to add tokio to your cargo:
#this is the version of tokio inside hyper "0.13.0-alpha.4"
tokio = "=0.2.0-alpha.6"
then change your run_server like this:
async fn run_server(forward_url: &'static str, addr: SocketAddr) {
let server = Server::bind(&addr).serve(make_service_fn(move |_| {
async move { Ok::<_, Error>(service_fn(move |req| call(forward_url, req))) }
}));
if let Err(err) = server.await {
eprintln!("server error: {}", err);
}
}
and main :
#[tokio::main]
async fn main() -> () {
// Set the address to run our socket on.
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
let url = "http://www.google.com:80"; // i have tested with google
run_server(url, addr).await
}
I have a concept project where the client sends a server a number (PrimeClientRequest), the server computes if the value is prime or not, and returns a response (PrimeClientResponse). I want the client to be a simple CLI which prompts the user for a number, sends the request to the server, and displays the response. Ideally I want to do this using TcpClient from Tokio and Streams from Futures-Rs.
I've written a Tokio server using services and I want to reuse the same codec and proto for the client.
Part of the client is a function called read_prompt which returns a Stream. Essentially it is an infinite loop at which each iteration reads in some input from stdin.
Here's the relevant code:
main.rs
use futures::{Future, Stream};
use std::env;
use std::net::SocketAddr;
use tokio_core::reactor::Core;
use tokio_prime::protocol::PrimeClientProto;
use tokio_prime::request::PrimeRequest;
use tokio_proto::TcpClient;
use tokio_service::Service;
mod cli;
fn main() {
let mut core = Core::new().unwrap();
let handle = core.handle();
let addr_string = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let remote_addr = addr_string.parse::<SocketAddr>().unwrap();
println!("Connecting on {}", remote_addr);
let tcp_client = TcpClient::new(PrimeClientProto).connect(&remote_addr, &handle);
core.run(tcp_client.and_then(|client| {
client
.call(PrimeRequest { number: Ok(0) })
.and_then(|response| {
println!("RESP = {:?}", response);
Ok(())
})
})).unwrap();
}
cli.rs
use futures::{Future, Sink, Stream};
use futures::sync::mpsc;
use std::{io, thread};
use std::io::{Stdin, Stdout};
use std::io::prelude::*;
pub fn read_prompt() -> impl Stream<Item = u64, Error = ()> {
let (tx, rx) = mpsc::channel(1);
thread::spawn(move || loop {
let thread_tx = tx.clone();
let input = prompt(io::stdout(), io::stdin()).unwrap();
let parsed_input = input
.parse::<u64>()
.map_err(|_| io::Error::new(io::ErrorKind::Other, "invalid u64"));
thread_tx.send(parsed_input.unwrap()).wait().unwrap();
});
rx
}
fn prompt(stdout: Stdout, stdin: Stdin) -> io::Result<String> {
let mut stdout_handle = stdout.lock();
stdout_handle.write(b"> ")?;
stdout_handle.flush()?;
let mut buf = String::new();
let mut stdin_handle = stdin.lock();
stdin_handle.read_line(&mut buf)?;
Ok(buf.trim().to_string())
}
With the code above, the client sends a single request to the server before the client terminates. I want to be able to use the stream generated from read_prompt to provide input to the TcpClient and make a request per item in the stream. How would I go about doing this?
The full code can be found at joshleeb/tokio-prime.
The solution I have come up with (so far) has been to use the LoopFn in the Future-Rs crate. It's not ideal as a new connection still has to be made but it is at least a step in the right direction.
main.rs
use futures::{future, Future};
use std::{env, io};
use std::net::SocketAddr;
use tokio_core::reactor::{Core, Handle};
use tokio_prime::protocol::PrimeClientProto;
use tokio_prime::request::PrimeRequest;
use tokio_proto::TcpClient;
use tokio_service::Service;
mod cli;
fn handler<'a>(
handle: &'a Handle, addr: &'a SocketAddr
) -> impl Future<Item = (), Error = ()> + 'a {
cli::prompt(io::stdin(), io::stdout())
.and_then(move |number| {
TcpClient::new(PrimeClientProto)
.connect(addr, handle)
.and_then(move |client| Ok((client, number)))
})
.and_then(|(client, number)| {
client
.call(PrimeRequest { number: Ok(number) })
.and_then(|response| {
println!("{:?}", response);
Ok(())
})
})
.or_else(|err| {
println!("! {}", err);
Ok(())
})
}
fn main() {
let mut core = Core::new().unwrap();
let handle = core.handle();
let addr_string = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let remote_addr = addr_string.parse::<SocketAddr>().unwrap();
println!("Connecting on {}", remote_addr);
let client = future::loop_fn((), |_| {
handler(&handle, &remote_addr)
.map(|_| -> future::Loop<(), ()> { future::Loop::Continue(()) })
});
core.run(client).ok();
}
cli.rs
use futures::prelude::*;
use std::io;
use std::io::{Stdin, Stdout};
use std::io::prelude::*;
#[async]
pub fn prompt(stdin: Stdin, stdout: Stdout) -> io::Result<u64> {
let mut stdout_handle = stdout.lock();
stdout_handle.write(b"> ")?;
stdout_handle.flush()?;
let mut buf = String::new();
let mut stdin_handle = stdin.lock();
stdin_handle.read_line(&mut buf)?;
parse_input(buf.trim().to_string())
}
fn parse_input(s: String) -> io::Result<u64> {
s.parse::<u64>()
.map_err(|_| io::Error::new(io::ErrorKind::Other, "invalid u64"))
}