I am trying to build an program that performs async http requests and returns their Response + body.
This is what the function that returns the Response looks like:
let responses = stream::iter(urls)
.map(|line| {
let client = &client;
async move {
client.get(&line).send().await.map(|resp| {
(line, resp)
})}
})
.buffer_unordered(concurrency_amount);
However, after returning resp, I can't use resp.text(), since resp.text() is of type Future<Output=Result<String>>.
How can I make the function also return resp.text() in the tuple?
Assuming that you are using reqwest, it should be possible to collect the bytes of the response body with Response::chunk(), text() consumes self but chunk() only takes a mutable reference.
Something like the following collects the response body and decodes it to a String in a lossy manner.
use futures_util::StreamExt;
#[tokio::main]
async fn main() {
let cli = reqwest::Client::new();
let urls = vec![
"https://stackoverflow.com".to_string(),
"https://google.com".into(),
"https://tokio.rs".into(),
];
let responses = futures_util::stream::iter(urls.into_iter())
.then(|url| { // note that I replaced `map` with `then` here.
let cli = cli.clone();
async move {
let mut resp = cli.get(url.clone()).send().await.unwrap();
let mut body = Vec::new();
while let Some(chunk) = resp.chunk().await.unwrap() {
body.extend_from_slice(&*chunk);
}
(url, resp, String::from_utf8_lossy(&body).to_string())
}
})
.collect::<Vec<_>>()
.await;
for (url, response, text) in responses {
println!("url: {} status: {} text: {}", url, response.status(), text);
}
}
As the in-line comment notes: I changed the map() call to then() so the stream yields the tuples rather than futures with the tuples as output.
Does this work?
let responses = stream::iter(urls)
.map(|line| {
let client = &client;
(async move || {
let resp = client.get(&line).send().await?;
let text = resp.text().await?;
(line, resp, text)
})()
})
.buffer_unordered(concurrency_amount);
Related
I am getting data from the open weather map API. Currently the data is being retrieved synchronously which is slow. However, the function has to be synchronous as it is part of a library, but it can call an async function. How might I still make concurrent requests to increase performance? A solution that does not use reqwests works, but reqwests is preferred.
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let mut data: Vec<HashMap<String, String>> = Vec::new();
for url in urls {
let request = reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected");
data.push(request);
}
return data;
}
If your program isn't already async, probably the easiest way might be to use rayon.
use reqwest;
use std::collections::HashMap;
use rayon::prelude::*;
fn get_combined_data(open_weather_map_api_url: String, open_weather_map_api_key: String,
coordinates: Vec<String>, metric: bool) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(open_weather_map_api_url, open_weather_map_api_key,
coordinates.get(0).expect("Improper coordinates").to_string() + "," +
coordinates.get(1).expect("Improper coordinates"), metric);
let data : Vec<_>= urls
.par_iter()
.map(|&url| reqwest::blocking::get(url).expect("Url Get failed").json().expect("json expected"))
.collect();
return data;
}
The easiest is probably to use tokios new_current_thread runtime and blocking on the data retreival.
use std::collections::HashMap;
use tokio::runtime;
pub fn collect_data() -> Vec<HashMap<String, String>> {
let rt = runtime::Builder::new_current_thread()
.build()
.expect("couldn't start runtime");
let urls = vec!["https://example.com/a", "https://example.com/b"];
rt.block_on(async move {
let mut data = vec![];
for url in urls {
data.push(async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
});
}
futures::future::join_all(data).await
})
}
You need an asynchronous runtime in order to call asynchronous functions. The easiest way to get one is to use the #[tokio::main] attribute (which despite the name can be applied to any function):
#[tokio::main]
fn get_combined_data(
open_weather_map_api_url: String,
open_weather_map_api_key: String,
coordinates: Vec<String>,
metric: bool,
) -> Vec<HashMap<String, String>> {
let urls: Vec<String> = get_urls(
open_weather_map_api_url,
open_weather_map_api_key,
coordinates
.get(0)
.expect("Improper coordinates")
.to_string()
+ ","
+ coordinates.get(1).expect("Improper coordinates"),
metric,
);
futures::future::join_all (urls.map (|u| {
async move {
reqwest::get(url)
.await
.expect("Url Get Failed")
.json()
.await
.expect("json expected")
}
})).await
}
I use thirtyfour in my Rust script, and it uses tokio as the async runtime.
When I use find in a Vec::iter, it doesn't work as I expect:
#[tokio::main]
async fn main() -> WebDriverResult<()> {
let dropdown = driver.find_element(By::Tag("select")).await?;
dropdown
.find_elements(By::Tag("option"))
.await?
.iter()
.find(|&&x| x.text() == book_time.date) // Error, x.text() return a futures::Future type while book_time.date return a &str.
.click()
.await?;
}
After I tried Ibraheem Ahmed's solution, I met more errors:
let dropdown = driver.find_element(By::Tag("select")).await?;
let elements = dropdown.find_elements(By::Tag("option")).await?;
let stream = stream::iter(elements);
let elements = stream.filter(|x| async move { x.text().await.unwrap() == target_date });
error: lifetime may not live long enough
--> src\main.rs:125:38
|
125 | let elements = stream.filter(|x| async move { x.text().await.unwrap() == target_date });
| -- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
| ||
| |return type of closure is impl futures::Future
| has type `&'1 thirtyfour::WebElement<'_>`
There's a good thread on the Rust User's Forum that covers a similar question.
I tried the code snippets below by modifying the thirtyfour tokio_async example. I didn't have your full context, so I created an example that found a link based on its text on the wikipedia.org home page.
filter_map.
let target_value = "Terms of Use";
let found_elements: Vec<_> = stream
.filter_map(|x| async move {
if let Ok(text) = x.text().await {
if text == target_value {
println!("found");
return Some(x);
}
}
None
})
.collect()
.await;
while loop (which is probably not what you are after, but could be a simple solution if your logic fits easily inside)...
while let Some(element) = stream.next().await {
let text = element.text().await?;
println!("link text result: {}", text);
}
You can use a Stream, which is the asynchronous version of Iterator:
use futures::stream::{self, StreamExt};
fn main() {
// ...
let stream = stream::iter(elements).await?;
let elements = stream
.filter(|x| async move { x.text().await.as_deref() == Ok(book_time.date) })
.next()
.click()
.await?;
}
I want to write a server using the current master branch of Hyper that saves a message that is delivered by a POST request and sends this message to every incoming GET request.
I have this, mostly copied from the Hyper examples directory:
extern crate futures;
extern crate hyper;
extern crate pretty_env_logger;
use futures::future::FutureResult;
use hyper::{Get, Post, StatusCode};
use hyper::header::{ContentLength};
use hyper::server::{Http, Service, Request, Response};
use futures::Stream;
struct Echo {
data: Vec<u8>,
}
impl Echo {
fn new() -> Self {
Echo {
data: "text".into(),
}
}
}
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = FutureResult<Response, hyper::Error>;
fn call(&self, req: Self::Request) -> Self::Future {
let resp = match (req.method(), req.path()) {
(&Get, "/") | (&Get, "/echo") => {
Response::new()
.with_header(ContentLength(self.data.len() as u64))
.with_body(self.data.clone())
},
(&Post, "/") => {
//self.data.clear(); // argh. &self is not mutable :(
// even if it was mutable... how to put the entire body into it?
//req.body().fold(...) ?
let mut res = Response::new();
if let Some(len) = req.headers().get::<ContentLength>() {
res.headers_mut().set(ContentLength(0));
}
res.with_body(req.body())
},
_ => {
Response::new()
.with_status(StatusCode::NotFound)
}
};
futures::future::ok(resp)
}
}
fn main() {
pretty_env_logger::init().unwrap();
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo::new())).unwrap();
println!("Listening on http://{} with 1 thread.", server.local_addr().unwrap());
server.run().unwrap();
}
How do I turn the req.body() (which seems to be a Stream of Chunks) into a Vec<u8>? I assume I must somehow return a Future that consumes the Stream and turns it into a single Vec<u8>, maybe with fold(). But I have no clue how to do that.
Hyper 0.13 provides a body::to_bytes function for this purpose.
use hyper::body;
use hyper::{Body, Response};
pub async fn read_response_body(res: Response<Body>) -> Result<String, hyper::Error> {
let bytes = body::to_bytes(res.into_body()).await?;
Ok(String::from_utf8(bytes.to_vec()).expect("response was not valid utf-8"))
}
I'm going to simplify the problem to just return the total number of bytes, instead of echoing the entire stream.
Futures 0.3
Hyper 0.13 + TryStreamExt::try_fold
See euclio's answer about hyper::body::to_bytes if you just want all the data as one giant blob.
Accessing the stream allows for more fine-grained control:
use futures::TryStreamExt; // 0.3.7
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.13.9
use std::convert::Infallible;
use tokio; // 0.2.22
#[tokio::main]
async fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(service::make_service_fn(|_conn| async {
Ok::<_, Infallible>(service::service_fn(echo))
}));
println!("Listening on http://{}.", server.local_addr());
if let Err(e) = server.await {
eprintln!("Error: {}", e);
}
}
async fn echo(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body
.try_fold(Vec::new(), |mut data, chunk| async move {
data.extend_from_slice(&chunk);
Ok(data)
})
.await;
entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
})
}
_ => {
let body = Body::from("Can only POST to /");
Ok(Response::new(body))
}
}
}
Unfortunately, the current implementation of Bytes is no longer compatible with TryStreamExt::try_concat, so we have to switch back to a fold.
Futures 0.1
hyper 0.12 + Stream::concat2
Since futures 0.1.14, you can use Stream::concat2 to stick together all the data into one:
fn concat2(self) -> Concat2<Self>
where
Self: Sized,
Self::Item: Extend<<Self::Item as IntoIterator>::Item> + IntoIterator + Default,
use futures::{
future::{self, Either},
Future, Stream,
}; // 0.1.25
use hyper::{server::Server, service, Body, Method, Request, Response}; // 0.12.20
use tokio; // 0.1.14
fn main() {
let addr = "127.0.0.1:12346".parse().expect("Unable to parse address");
let server = Server::bind(&addr).serve(|| service::service_fn(echo));
println!("Listening on http://{}.", server.local_addr());
let server = server.map_err(|e| eprintln!("Error: {}", e));
tokio::run(server);
}
fn echo(req: Request<Body>) -> impl Future<Item = Response<Body>, Error = hyper::Error> {
let (parts, body) = req.into_parts();
match (parts.method, parts.uri.path()) {
(Method::POST, "/") => {
let entire_body = body.concat2();
let resp = entire_body.map(|body| {
let body = Body::from(format!("Read {} bytes", body.len()));
Response::new(body)
});
Either::A(resp)
}
_ => {
let body = Body::from("Can only POST to /");
let resp = future::ok(Response::new(body));
Either::B(resp)
}
}
}
You could also convert the Bytes into a Vec<u8> via entire_body.to_vec() and then convert that to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
hyper 0.11 + Stream::fold
Similar to Iterator::fold, Stream::fold takes an accumulator (called init) and a function that operates on the accumulator and an item from the stream. The result of the function must be another future with the same error type as the original. The total result is itself a future.
fn fold<F, T, Fut>(self, init: T, f: F) -> Fold<Self, F, Fut, T>
where
F: FnMut(T, Self::Item) -> Fut,
Fut: IntoFuture<Item = T>,
Self::Error: From<Fut::Error>,
Self: Sized,
We can use a Vec as the accumulator. Body's Stream implementation returns a Chunk. This implements Deref<[u8]>, so we can use that to append each chunk's data to the Vec.
extern crate futures; // 0.1.23
extern crate hyper; // 0.11.27
use futures::{Future, Stream};
use hyper::{
server::{Http, Request, Response, Service}, Post,
};
fn main() {
let addr = "127.0.0.1:12346".parse().unwrap();
let server = Http::new().bind(&addr, || Ok(Echo)).unwrap();
println!(
"Listening on http://{} with 1 thread.",
server.local_addr().unwrap()
);
server.run().unwrap();
}
struct Echo;
impl Service for Echo {
type Request = Request;
type Response = Response;
type Error = hyper::Error;
type Future = Box<futures::Future<Item = Response, Error = Self::Error>>;
fn call(&self, req: Self::Request) -> Self::Future {
match (req.method(), req.path()) {
(&Post, "/") => {
let f = req.body()
.fold(Vec::new(), |mut acc, chunk| {
acc.extend_from_slice(&*chunk);
futures::future::ok::<_, Self::Error>(acc)
})
.map(|body| Response::new().with_body(format!("Read {} bytes", body.len())));
Box::new(f)
}
_ => panic!("Nope"),
}
}
}
You could also convert the Vec<u8> body to a String.
See also:
How do I convert a Vector of bytes (u8) to a string
Output
When called from the command line, we can see the result:
$ curl -X POST --data hello http://127.0.0.1:12346/
Read 5 bytes
Warning
All of these solutions allow a malicious end user to POST an infinitely sized file, which would cause the machine to run out of memory. Depending on the intended use, you may wish to establish some kind of cap on the number of bytes read, potentially writing to the filesystem at some breakpoint.
See also:
How do I apply a limit to the number of bytes read by futures::Stream::concat2?
Most of the answers on this topic are outdated or overly complicated. The solution is pretty simple:
/*
WARNING for beginners!!! This use statement
is important so we can later use .data() method!!!
*/
use hyper::body::HttpBody;
let my_vector: Vec<u8> = request.into_body().data().await.unwrap().unwrap().to_vec();
let my_string = String::from_utf8(my_vector).unwrap();
You can also use body::to_bytes as #euclio answered. Both approaches are straight-forward! Don't forget to handle unwrap properly.
I am trying to rewrite the proxy example of Asynchronous Programming in Rust book by migrating to :
futures-preview = { version = "0.3.0-alpha.19", features = ["async-await"]}`
hyper = "0.13.0-alpha.4"`
from:
futures-preview = { version = "=0.3.0-alpha.17", features = ["compat"] }`
hyper = "0.12.9"
The current example converts the returned Future from a futures 0.3 into a futures 0.1, because hyper = "0.12.9" is not compatible with futures 0.3's async/await.
My code:
use {
futures::future::{FutureExt, TryFutureExt},
hyper::{
rt::run,
service::{make_service_fn, service_fn},
Body, Client, Error, Request, Response, Server, Uri,
},
std::net::SocketAddr,
std::str::FromStr,
};
fn forward_uri<B>(forward_url: &'static str, req: &Request<B>) -> Uri {
let forward_uri = match req.uri().query() {
Some(query) => format!("{}{}?{}", forward_url, req.uri().path(), query),
None => format!("{}{}", forward_url, req.uri().path()),
};
Uri::from_str(forward_uri.as_str()).unwrap()
}
async fn call(
forward_url: &'static str,
mut _req: Request<Body>,
) -> Result<Response<Body>, hyper::Error> {
*_req.uri_mut() = forward_uri(forward_url, &_req);
let url_str = forward_uri(forward_url, &_req);
let res = Client::new().get(url_str).await;
res
}
async fn run_server(forward_url: &'static str, addr: SocketAddr) {
let forwarded_url = forward_url;
let serve_future = service_fn(move |req| call(forwarded_url, req).boxed());
let server = Server::bind(&addr).serve(serve_future);
if let Err(err) = server.await {
eprintln!("server error: {}", err);
}
}
fn main() {
// Set the address to run our socket on.
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
let url = "http://127.0.0.1:9061";
let futures_03_future = run_server(url, addr);
run(futures_03_future);
}
First, I receive this error for server in run_server function:
the trait tower_service::Service<&'a
hyper::server::tcp::addr_stream::AddrStream> is not implemented for
hyper::service::service::ServiceFn<[closure#src/main.rs:35:35: 35:78
forwarded_url:_], hyper::body::body::Body>
Also, I cannot use hyper::rt::run because it might have been implemented differently in hyper = 0.13.0-alpha.4.
I will be grateful if you tell me your ideas on how to fix it.
By this issue, to create a new service for each connection you need to create MakeService in hyper = "0.13.0-alpha.4". You can create MakeService with a closure by using make_service_fn.
Also, I cannot use hyper::rt::run because it might have been implemented differently in hyper = 0.13.0-alpha.4.
Correct, under the hood hyper::rt::run was calling tokio::run, it has been removed from the api but currently i don't know the reason. You can run your future with calling tokio::run by yourself or use #[tokio::main] annotation. To do this you need to add tokio to your cargo:
#this is the version of tokio inside hyper "0.13.0-alpha.4"
tokio = "=0.2.0-alpha.6"
then change your run_server like this:
async fn run_server(forward_url: &'static str, addr: SocketAddr) {
let server = Server::bind(&addr).serve(make_service_fn(move |_| {
async move { Ok::<_, Error>(service_fn(move |req| call(forward_url, req))) }
}));
if let Err(err) = server.await {
eprintln!("server error: {}", err);
}
}
and main :
#[tokio::main]
async fn main() -> () {
// Set the address to run our socket on.
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
let url = "http://www.google.com:80"; // i have tested with google
run_server(url, addr).await
}
I have this code which- as far as I understand- is async:
func testConnection() -> Bool {
let url = URL(string: uri)!
let task = URLSession.shared.dataTask(with: url) {
(data, response, error) in
guard let data = data, let _:URLResponse = response , error == nil else {
print("error")
return
}
let dataString = String(data: data, encoding: String.Encoding.utf8)
}
task.resume()
}
How would a synchronous version look that allows to return the testConnection result to the sender?
You could use a semaphore to wait for the async call to finish:
func testConnection() -> Bool {
let url = URL(string: uri)!
var dataStringOrNil: String?
let semaphore = DispatchSemaphore(value: 0)
let task = URLSession.shared.dataTask(with: url) { (data, response, error) in
defer {
semaphore.signal()
}
guard let data = data, error == nil else {
print("error")
return
}
dataStringOrNil = String(data: data, encoding: .utf8)
}
task.resume()
semaphore.wait()
guard let dataString = dataStringOrNil else {
return false
}
// some substring checking
return dataString.contains("href")
}
I wouldn't recommend it, but there can be reasons for sync calls. We use them sometimes in command-line tools.