Rust newbie here since I recently started working on it. I'm trying get a rest api working and the following code works completely fine for me.
MyRest.rs
pub struct RestBro;
impl RestBro {
pub async fn run_bro() {
let routes_post = warp::post()
.and(warp::path!("v1" / "homie").and_then(my_function));
warp::serve(routes)
.run(([127, 0, 0, 1], 3003))
.await;
}
}
main.rs
#[tokio::main]
async fn main() {
let rb = RestBro;
rb.run_bro().await;
}
Now the thing is that I don't want my main to be an async and I just can't figure out how to run that run_bro() function indefinitely like its happening above. I've tried block_on and that just blocks and waits for run_bro to interrupt which is expected and when I tried spawn it just runs through and exits. The documentation on Tokio confused me and that's why I'm looking for some help here.
block_on
fn main() {
let async_block = async {
let rb = RestBro;
rb.run_bro().await;
};
let tr = tokio::runtime::Runtime::new().unwrap();
tr.block_on(async_block);
println!("Everything working good!");
}
spawn
fn main() {
let tr = tokio::runtime::Runtime::new().unwrap();
tr.spawn(async {
let rb = RestBro;
rb.run_bro().await;
});
println!("Everything working good!");
}
To be clear, my question is how can I call asynchronous run_bro() function and await from a synchronous main?
Thanks in advance!!
To be clear, my question is how can I call asynchronous run_bro()
function and await from a synchronous main?
I do not understand why you'd want to wrap server in spawn. In your code, it's not working because your main program closes and the spawn closes with it.
fn main() {
let tr = tokio::runtime::Runtime::new().unwrap();
tr.spawn(async {
let rb = RestBro;
rb.run_bro().await;
});
println!("Everything working good!");
}
If you change it to this it'll work for five seconds.
use std::{thread, time};
fn main() {
let tr = tokio::runtime::Runtime::new().unwrap();
tr.spawn(async {
let rb = RestBro;
rb.run_bro().await;
});
println!("Everything working good!");
thread::sleep(time::Duration::from_secs(5));
}
Or indefinitely:
fn main() {
let tr = tokio::runtime::Runtime::new().unwrap();
tr.spawn(async {
let rb = RestBro;
rb.run_bro().await;
});
println!("Everything working good!");
tr.join().unwrap();
}
Related
I started learning rust 2 weeks ago, and has been making this application that watches a log file, and sends a bulk of the information to an elasticsearch DB.
The problem is that after certain amount of time, it freezes (using 100% CPU) and I don't understand why.
I've cut down on a lot of code to try to figure out the issue, but it still keeps freezing on this line according to clion debugger
let _response = reqwest::Client::new()
.post("http://127.0.0.1/test.php")
.header("Content-Type", "application/json")
.body("{\"test\": true}")
.timeout(Duration::from_secs(30))
.send() // <-- Exactly here
.await;
It freezes and doesn't return any error message.
This is the code in context:
use std::{env};
use std::io::{stdout, Write};
use std::path::Path;
use std::time::Duration;
use logwatcher::{LogWatcher, LogWatcherAction};
use serde_json::{json, Value};
use serde_json::Value::Null;
use tokio;
#[tokio::main]
async fn main() {
let mut log_watcher = LogWatcher::register("/var/log/test.log").unwrap();
let mut counter = 0;
let BULK_SIZE = 500;
log_watcher.watch(&mut move |line: String| { // This triggers each time a new line is appended to /var/log/test.log
counter += 1;
if counter >= BULK_SIZE {
futures::executor::block_on(async { // This has to be async because log_watcher is not async
let _response = reqwest::Client::new()
.post("http://127.0.0.1/test.php") // <-- This is just for testing, it fails towards the DB too
.header("Content-Type", "application/json")
.body("{\"test\": true}")
.timeout(Duration::from_secs(30))
.send() // <-- Freezes here
.await;
if _response.is_ok(){
println!("Ok");
}
});
counter = 0;
}
LogWatcherAction::None
});
}
The log file gets about 625 new lines every minute. The crash happends after about ~5500 - ~25000 lines has gone through, or it seems a bit random in general.
I'm suspecting the issue is either something to do with LogWatcher, reqwest, the block_on or the mix of async.
Does anyone have any clue why it randomly freezes?
The problem was indeed because of a mix of async with tokio and block_on, NOT directly reqwest.
The problem was solved when changing main to be non-async, and using tokio as the block_on for async calls instead of futures::executor::block_on.
fn main() {
let mut log_watcher = LogWatcher::register("/var/log/test.log").unwrap();
let mut counter = 0;
let BULK_SIZE = 500;
log_watcher.watch(&mut move |line: String| {
counter += 1;
if counter >= BULK_SIZE {
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap()
.block_on(async {
let _response = reqwest::Client::new()
.post("http://127.0.0.1/test.php")
.header("Content-Type", "application/json")
.body("{\"test\": true}")
.timeout(Duration::from_secs(30))
.send()
.await;
if _response.is_ok(){
println!("Ok");
}
});
counter = 0;
}
LogWatcherAction::None
});
}
Here's the example from the Rust book.
async fn learn_and_sing() {
// Wait until the song has been learned before singing it.
// We use `.await` here rather than `block_on` to prevent blocking the
// thread, which makes it possible to `dance` at the same time.
let song = learn_song().await;
sing_song(song).await;
}
async fn async_main() {
let f1 = learn_and_sing();
let f2 = dance();
// `join!` is like `.await` but can wait for multiple futures concurrently.
// If we're temporarily blocked in the `learn_and_sing` future, the `dance`
// future will take over the current thread. If `dance` becomes blocked,
// `learn_and_sing` can take back over. If both futures are blocked, then
// `async_main` is blocked and will yield to the executor.
futures::join!(f1, f2);
}
fn main() {
block_on(async_main());
}
And it's says
In this example, learning the song must happen before singing the song, but both learning and singing can happen at the same time as dancing.
But I can't get this point. I wrote a short code in Rust
async fn learn_song() -> &'static str {
println!("learn_song");
"some song"
}
#[allow(unused_variables)]
async fn sing_song(song: &str) {
println!("sing_song");
}
async fn dance() {
println!("dance");
}
async fn learn_and_sing() {
let song = learn_song().await;
std::thread::sleep(std::time::Duration::from_secs(1));
sing_song(song).await;
}
async fn async_main() {
let f1 = learn_and_sing();
let f2 = dance();
let f3 = learn_and_sing();
futures::join!(f1, f2, f3);
}
fn main() {
futures::executor::block_on(async_main());
}
And it seems like all the async functions in the async_main executed synchronously.
The output is
learn_song
sing_song
dance
learn_song
sing_song
If they run asynchronously, I would expect to get something like this in my output
learn_song
dance
learn_song
sing_song
sing_song
If I add an extra call of learn_and_sing it would steel be printed like in a synchronous function.
The question Why so? Is it possible to make a real async using only async/.await and no threads?
Like tkausl's comment states, std::thread::sleep makes the whole thread sleep, which prevents any code on the thread from executing during the sleeping duration. You could use async_std::task::sleep in this situation, as it is an asynchronous version of the sleep function.
async fn learn_song() -> &'static str {
println!("learn_song");
"some song"
}
#[allow(unused_variables)]
async fn sing_song(song: &str) {
println!("sing_song");
}
async fn dance() {
println!("dance");
}
async fn learn_and_sing() {
let song = learn_song().await;
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
sing_song(song).await;
}
#[async_std::main]
async fn main() {
let f1 = learn_and_sing();
let f2 = dance();
let f3 = learn_and_sing();
futures::join!(f1, f2, f3);
}
I am trying to use hyper to grab the content of an HTML page and would like to synchronously return the output of a future. I realized I could have picked a better example since synchronous HTTP requests already exist, but I am more interested in understanding whether we could return a value from an async calculation.
extern crate futures;
extern crate hyper;
extern crate hyper_tls;
extern crate tokio;
use futures::{future, Future, Stream};
use hyper::Client;
use hyper::Uri;
use hyper_tls::HttpsConnector;
use std::str;
fn scrap() -> Result<String, String> {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
futures::future::ok(s_body)
})
}).map_err(|err| format!("Error scraping web page: {:?}", &err))
});
scraped_content.wait()
}
fn read() {
let scraped_content = future::lazy(|| {
let https = HttpsConnector::new(4).unwrap();
let client = Client::builder().build::<_, hyper::Body>(https);
client
.get("https://hyper.rs".parse::<Uri>().unwrap())
.and_then(|res| {
res.into_body().concat2().and_then(|body| {
let s_body: String = str::from_utf8(&body).unwrap().to_string();
println!("Reading body: {}", s_body);
Ok(())
})
}).map_err(|err| {
println!("Error reading webpage: {:?}", &err);
})
});
tokio::run(scraped_content);
}
fn main() {
read();
let content = scrap();
println!("Content = {:?}", &content);
}
The example compiles and the call to read() succeeds, but the call to scrap() panics with the following error message:
Content = Err("Error scraping web page: Error { kind: Execute, cause: None }")
I understand that I failed to launch the task properly before calling .wait() on the future but I couldn't find how to properly do it, assuming it's even possible.
Standard library futures
Let's use this as our minimal, reproducible example:
async fn example() -> i32 {
42
}
Call executor::block_on:
use futures::executor; // 0.3.1
fn main() {
let v = executor::block_on(example());
println!("{}", v);
}
Tokio
Use the tokio::main attribute on any function (not just main!) to convert it from an asynchronous function to a synchronous one:
use tokio; // 0.3.5
#[tokio::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
tokio::main is a macro that transforms this
#[tokio::main]
async fn main() {}
Into this:
fn main() {
tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap()
.block_on(async { {} })
}
This uses Runtime::block_on under the hood, so you can also write this as:
use tokio::runtime::Runtime; // 0.3.5
fn main() {
let v = Runtime::new().unwrap().block_on(example());
println!("{}", v);
}
For tests, you can use tokio::test.
async-std
Use the async_std::main attribute on the main function to convert it from an asynchronous function to a synchronous one:
use async_std; // 1.6.5, features = ["attributes"]
#[async_std::main]
async fn main() {
let v = example().await;
println!("{}", v);
}
For tests, you can use async_std::test.
Futures 0.1
Let's use this as our minimal, reproducible example:
use futures::{future, Future}; // 0.1.27
fn example() -> impl Future<Item = i32, Error = ()> {
future::ok(42)
}
For simple cases, you only need to call wait:
fn main() {
let s = example().wait();
println!("{:?}", s);
}
However, this comes with a pretty severe warning:
This method is not appropriate to call on event loops or similar I/O situations because it will prevent the event loop from making progress (this blocks the thread). This method should only be called when it's guaranteed that the blocking work associated with this future will be completed by another thread.
Tokio
If you are using Tokio 0.1, you should use Tokio's Runtime::block_on:
use tokio; // 0.1.21
fn main() {
let mut runtime = tokio::runtime::Runtime::new().expect("Unable to create a runtime");
let s = runtime.block_on(example());
println!("{:?}", s);
}
If you peek in the implementation of block_on, it actually sends the future's result down a channel and then calls wait on that channel! This is fine because Tokio guarantees to run the future to completion.
See also:
How can I efficiently extract the first element of a futures::Stream in a blocking manner?
As this is the top result that come up in search engines by the query "How to call async from sync in Rust", I decided to share my solution here. I think it might be useful.
As #Shepmaster mentioned, back in version 0.1 futures crate had beautiful method .wait() that could be used to call an async function from a sync one. This must-have method, however, was removed from later versions of the crate.
Luckily, it's not that hard to re-implement it:
trait Block {
fn wait(self) -> <Self as futures::Future>::Output
where Self: Sized, Self: futures::Future
{
futures::executor::block_on(self)
}
}
impl<F,T> Block for F
where F: futures::Future<Output = T>
{}
After that, you can just do following:
async fn example() -> i32 {
42
}
fn main() {
let s = example().wait();
println!("{:?}", s);
}
Beware that this comes with all the caveats of original .wait() explained in the #Shepmaster's answer.
This works for me using tokio:
tokio::runtime::Runtime::new()?.block_on(fooAsyncFunction())?;
I need a way to run the same function many times with different inputs.
And since the function depends on a slow web API, I need to run it concurrently and collect the results in one variable.
I use the following:
use tokio_stream::StreamExt;
async fn run(input: &str) -> Vec<String> {
vec![String::from(input), String::from(input)]
}
async fn main() {
let mut input = tokio_stream::iter(vec!["1","2","3","4","5","6","7","8"]);
let mut handles = vec![];
while let Some(domain) = input.next().await {
handles.push(run(domain));
}
let mut results = vec![];
let mut handles = tokio_stream::iter(handles);
while let Some(handle) = handles.next().await {
results.extend(handle.await);
}
}
I know there is a way with the futures crate, but I don't know if I can use it with tokio. Also tokio_stream::StreamExt contains fold and map methods but I can't find a way to use them without calling await.
What is the best way to do this?
IIUC what you want, you can use tokio::spawn to launch your tasks in the background and futures::join_all to wait until they have all completed. E.g. something like this (untested):
async fn run(input: &str) -> Vec<String> {
vec![String::from(input), String::from(input)]
}
async fn main() {
let input = vec!["1","2","3","4","5","6","7","8"];
let handles = input.iter().map (|domain| {
tokio::spawn (async move { run (domain).await })
});
let results = futures::join_all (handles).await;
}
I have a small program that executes the aws s3 cli commands but with different arguments. I'm using the Command crate and the the command makes a network call and returns some response. At first I have this synchronous & single-threaded implementation:
fn make_call<'a>(_name: &'a str, _bucket_poll: &mut BucketPoll<'a>) -> Option<BucketDetails<'a>> {
let invoke_result = invoke_network_call(_name);
let mut bucket = BucketDetails::new(_name);
match invoke_result {
Ok(invoke_str) => {
bucket.output = invoke_str;
_bucket_poll.insert_bucket(bucket.clone());
_bucket_poll.successful_count += 1;
Some(bucket)
}
Err(_) => {
_bucket_poll.insert_bucket(bucket);
None
}
}
}
// I invoke this function in sequential order, something like
make_call('name_1');
make_call('name_2');
make_call('name_3');
Because I don't really care at which order this function is executed, I decided to learn Tokio to help with performance. I changed the make_call function to be async:
async fn make_call_race() -> ExecutionResult {
let bucket_poll = BucketPoll::new();
let bucket_poll_guard = Arc::new(Mutex::new(bucket_poll));
loop {
let bucket_details = tokio::select! {
Some(bucket_details) = make_call_async("name_1", &bucket_poll_guard) => bucket_details,
Some(bucket_details) = make_call_async("name_2", &bucket_poll_guard) => bucket_details,
Some(bucket_details) = make_call_async("name_3", &bucket_poll_guard) => bucket_details,
Some(bucket_details) = make_call_async("name_4", &bucket_poll_guard) => bucket_details,
else => { break }
};
success_printer(bucket_details);
}
// more printing, no more network calls
ExecutionResult::Success
}
make_call_async is essentially the same as make_call:
async fn make_call_async<'a>(
_name: &'a str,
_bucket_poll_guard: &'a Arc<Mutex<BucketPoll<'a>>>,
) -> Option<BucketDetails<'a>> {
{
if let Ok(bucket_poll_guard) = _bucket_poll_guard.lock() {
if bucket_poll_guard.has_polled(_name) {
return None;
}
}
}
let invoke_result = invoke_network_call(_name);
let mut bucket = BucketDetails::new(_name);
match invoke_result {
Ok(invoke_str) => {
bucket.output = invoke_str;
{
if let Ok(mut bucket_poll_guard) = _bucket_poll_guard.lock() {
bucket_poll_guard.insert_bucket(bucket.clone());
bucket_poll_guard.successful_count += 1;
}
}
Some(bucket)
}
Err(_) => {
{
if let Ok(mut bucket_poll_guard) = _bucket_poll_guard.lock() {
bucket_poll_guard.insert_bucket(bucket);
}
}
None
}
}
}
When I run the async version, I do see that my network calls are made a random order but I do not notice any speedups. I increased the number of network calls to ~50ish invocations but the runtime is nearly the same if not slightly worse. As I am new to async programming and Rust in general, I would like to understand why my async implementation does not seem to offer any improvement.
Extra:
Here is the invoke_network_call method:
fn invoke_network_call(_name: &str) -> core::result::Result<String, AwsCliError> {
let output = Command::new("aws")
.arg("s3")
.arg("ls")
.arg(_name)
.output()
.expect("Could not list s3 objects");
if !output.status.success() {
err_printer(format!("Failed to list s3 objects for bucket {}.", _name));
return Err(AwsCliError);
}
let output_str = get_stdout_string_from_output(&output);
Ok(output_str)
}
EDIT: yorodm's comment makes sense. What I did was use Tokio's Command instead of std::process's Command and made the invoke_network_call async. This reduced my runtime by half. Thank you!
You could rewrite invoke_network_call using an async version of Command.
async fn invoke_network_call(_name: &str) -> core::result::Result<String, AwsCliError> {
let output = tokio::process::Command::new("aws")
.arg("s3")
.arg("ls")
.arg(_name)
.output()
.await
.expect("Could not list s3 objects");
if !output.status.success() {
err_printer(format!("Failed to list s3 objects for bucket {}.", _name));
return Err(AwsCliError);
}
let output_str = get_stdout_string_from_output(&output);
Ok(output_str)
}
Thus removing the blocking std::process::Command call. However I would say that if you're going to access AWS services you should go with rusoto