How to use Hyper body channel? - http

I am trying to make a POST HTTP request using the Hyper library, and then write data to it until I close it (from multiple functions/threads). I've found in documentation that I can use Body::channel() to do this, but I am using it improperly as I can only write to the channel once. I haven't found any examples, can someone please point me into the right direction?
let (mut sender, body) = Body::channel();
let request = Request::builder()
.method(Method::POST)
.uri("http://localhost:3000/")
.header("content-type", "text")
.body(body)
.unwrap();
let client = Client::new();
let response = client.request(request);
//Does get sent
println!("Body: {:?}", sender.send_data(hyper::body::Bytes::from("test\n")).await);
//Stuck on this one
println!("Body: {:?}", sender.send_data(hyper::body::Bytes::from("test2\n")).await);
//Debug print
println!("{:?}", response.await);

You'd need to wrap the send_data() lines in tokio::spawn(async move { ... });.
The issue is that Body/Sender only has a buffer size of 1. So the second send_data() call causes it to await (internally) until the Sender is ready. This is what subsequently causes it to "get stuck".
This is resolved by using tokio::spawn() as it finally allows awaiting the ResponseFuture, which causes the request to be performed.
let (sender, body) = Body::channel();
let request = Request::builder()
.method(Method::POST)
.uri("http://localhost:3000/")
.header("content-type", "text")
.body(body)
.unwrap();
let client = Client::new();
let response = client.request(request);
tokio::spawn(async move {
let mut sender = sender;
println!("Body: {:?}", sender.send_data(hyper::body::Bytes::from("test\n")).await);
println!("Body: {:?}", sender.send_data(hyper::body::Bytes::from("test2\n")).await);
});
println!("{:?}", response.await);
The buffer size isn't mentioned in the docs (that I know of). However, you can figure that out by checking the source related to Body::channel(), where you can see it constructs a MPSC using futures_channel::mpsc::channel(0). The docs mention that the buffer size would be buffer + num-senders, which in our case would be 0 + 1.

Related

Understanding how to use rust TOKIO library

I want to learn the rust Tokio library. To facilitate this I want to write an ASYNC TCP logger in rust.
Basically a TCP client that connects to a TCP server (172.16.10.10 port 7777) and just logs messages received asynchronously file to a log file. I want the main function to read user input from the console - in my case was for pressing ‘q’ key - simulate the program doing some other task.
I expect to receive multiple TCP responses whilst waiting for user to press ‘q’ key.
I am trying to workout how to read and log multiple TCP responses independently of waiting for the user input
let mut buf_reader = BufReader::new(&stream);
let mut data = vec![];
buf_reader.read_to_end(&mut data).await.unwrap();
log_writer.write_all(&data).await.unwrap();`
Here is the code I have
use tokio::net::TcpStream;
use tokio::prelude::*;
use std::io::{stdin, stdout, Write, BufWriter, BufReader};
use std::fs::File;
#[tokio::main]
async fn main() {
let ip = "172.16.10.10:7777";
let mut stream = TcpStream::connect(ip).await.unwrap();
let message = [0x16, 0x02];
stream.write(&message).await.unwrap();
// Open a file for logging
let file = File::create("log.txt").unwrap();
let mut log_writer = BufWriter::new(file);
println!("Press 'q' to exit and receive response:");
stdout().flush().unwrap();
let mut input = String::new();
stdin().read_line(&mut input).unwrap();
if input.trim() == "q" {
// SIMULATE doing time consuming task
println!(“Quitting”);
}
}
I tried the following but this loops over the waiting for user input. This is not behaviour I want. I want to be reading and logging the TCP messages independent of the awaiting user inout.
loop {
stdin().read_line(&mut input).unwrap();
if input.trim() == "q" {
break;
}
let mut data = vec![];
buf_reader.read_to_end(&mut data).await.unwrap();
log_writer.write_all(&data).await.unwrap();
}
When I needed async multithreading, I defined multiple async fn to do what I want, then called them in async fn main as:
let handle1 = tokio::spawn(do_it("test_data.txt"));
let handle2 = tokio::spawn(do_something_else("test_data.txt"));
handle1.await.unwrap();
handle2.await.unwrap();
Since I'm zealous about keeping fn main as minimal as possible, this may not exactly work for you, but may give you a direction.

expected struct `Vec`, found enum `Result` in tokio, cacache and match

I find it difficult to understand what's wrong with the below code. I'm getting expected struct Vec, found enum Result error at Ok(from_cache), but I have adopted the code from https://github.com/platy/update-tracker/blob/843092708906063704442f352231bfbac5b06196/server/src/web/mod.rs#L216-L226
During web scraping, I'm trying to cache the content of the URL in the cache and trying to reuse it.
use std::error::Error;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let url: &str = "https://example.com/";
let html = match cacache::read("./cache", url).await? {
Ok(from_cache) => String::from_utf8(from_cache),
Err(_) => {
let t_html = reqwest::get(url).await?.text().await?;
cacache::write("./cache", url, &t_html).await?;
t_html
},
};
println!("html = {:?}", html);
Ok(())
}
Here's the playground (but, it shows other errors due to missing dependencies). Can anyone please explain this or share any relevant guide to gather more information about this topic?
Recall that the ? operator unwraps a Result (or Option) by propagating the Err (or None) case out of the current function. Therefore, this expression:
cacache::read("./cache", url).await?
Has type Vec<u8> as the ? operator has unwrapped the Result. If you want to handle errors yourself, then omit the ? operator:
cacache::read("./cache", url).await

How to write type definition with tokio_serde::SymmetricallyFramed::new()?

I am rying to serialize cbor using serde_tokio.
I can make a simple program work, but I need to actually store the tokio_serde::SymmetricallyFramed::new() in a structure to use it more than once.
(It consumes the socket, which is cool).
I can't seem to write a type that will store the value.
use futures::prelude::*;
use tokio::net::TcpStream;
use tokio_serde::formats::*;
use tokio_util::codec::{FramedWrite, LengthDelimitedCodec};
#[tokio::main]
pub async fn main() {
// Bind a server socket
let socket = TcpStream::connect("127.0.0.1:17653").await.unwrap();
// Delimit frames using a length header
let length_delimited = FramedWrite::new(socket, LengthDelimitedCodec::new());
// Serialize frames with JSON
let mut serialized = tokio_serde::SymmetricallyFramed::new(length_delimited, SymmetricalCbor::default());
// Send the value
serialized
.send(vec![1i32,2,3])
.await
.unwrap()
}
produces the right output. (Adopted from the json example in tokio-serde crate, here: https://github.com/carllerche/tokio-serde/blob/master/examples/client.rs
I want to put "serialized" into a structure (and hide how it is created in a fn), but I can't seem to write the right type.
use futures::prelude::*;
use serde_cbor;
use tokio::net::TcpStream;
use tokio_serde::formats::*;
use tokio_util::codec::{FramedWrite, LengthDelimitedCodec};
type CborWriter = tokio_serde::Framed<tokio_util::codec::FramedWrite<tokio::net::TcpStream, tokio_util::codec::LengthDelimitedCodec>, serde_cbor::Value, serde_cbor::Value, tokio_serde::formats::Cbor<serde_cbor::Value, serde_cbor::Value>>;
// something like this has been suggested, but so far no luck.
// fn setup_writer(socket: tokio::net::TcpStream) -> impl Sink<??>+Debug {
fn setup_writer(socket: tokio::net::TcpStream) -> CborWriter {
// Delimit frames using a length header
let length_delimited = FramedWrite::new(socket, LengthDelimitedCodec::new());
// Serialize frames with CBOR
let serialized = tokio_serde::SymmetricallyFramed::new(length_delimited, SymmetricalCbor::default());
return serialized;
}
#[tokio::main]
pub async fn main() {
// Bind a server socket
let socket = TcpStream::connect("127.0.0.1:17653").await.unwrap();
// Serialize frames with CBOR
let mut serialized = setup_writer(socket);
// Send the value
serialized
.send(serde_cbor::Value::Array(vec![serde_cbor::Value::Integer(1i128),
serde_cbor::Value::Integer(2i128),
serde_cbor::Value::Integer(3i128)]))
.await
.unwrap()
}
But, I don't want to put cbor::Value in. I should just be able to put the Serializable objects in. So I am obviously going in the wrong direction here. The JSON example in the tokio-serde crate is happy to put in/out serde_json::Value, but I should have to do that, I think.
A suggestion on Discord was made to change the first example as:
let mut serialized: () = tokio_serde::SymmetricallyFramed::new(length_delimited, SymmetricalCbor::default());
and let the compiler tell me what the type is:
= note: expected unit type `()`
found struct `tokio_serde::Framed<tokio_util::codec::FramedWrite<tokio::net::TcpStream, tokio_util::codec::LengthDelimitedCodec>, _, _, tokio_serde::formats::Cbor<_, _>>`
Well, I can't put _ into the type alias, or write it directly.
I think it should say something like "impl Serialize", but that's not yet a thing.
Obviously, the compiler gets the first example right, so there must be something that will go in there... but what?

How to interrupt await'ing in a loop

Consider this function receiving datagrams from UDP socket:
async fn recv_multiple(socket: &async_std::net::UdpSocket) -> Vec<String> {
let mut buf = [0; 1024];
let mut v = vec![];
while let Ok((amt, _src)) = socket.recv_from(&mut buf).await {
v.push(String::from_utf8_lossy(&buf[..amt]).to_string());
}
v
}
And then it is executed for example like this:
let fut = recv_multiple(&socket);
async_std::task::block_on(fut);
How to add timeout functionality so that after exactly 1 minute a Vec<String> (empty or not) is returned from recv_multiple? I need to collect incoming datagrams for a minute and then return what was captured during that time. Please note that I don't need to timeout a single recv_from operation because datagrams can appear very often and the function will never timeout.
I've found a couple of partial solutions but they don't fit well:
async_std::future::timeout times out a Future but partially filled Vec is discarded and only Err is returned
async_std::future::Future::timeout same story, partial output is discarded
As I see it I should pass some kind of a "timeout" Future inside this function and in while let I have to await on both socket and "timeout" Future. Then if socket fires we proceed but if "timeout" fires then we return from the function. But I'm not sure how to accomplish this.
Instead of timeouting on recv_multiple you can timeout on socket.recv_from inside recv_multiple. It'll require to recalculate remaining time after each iteration. And when remaining time is 0 then you can return your Vec<_>.

F# - The type was expected to have type Async<'a> but has string -> Asnyc<'a> instead

After shamelessly pilfering a code snippet from Tomas Petricek's Blog:
http://tomasp.net/blog/csharp-fsharp-async-intro.aspx
Specifically, this one (and making a few alterations to it):
let downloadPage(url:string) (postData:string) = async {
let request = HttpWebRequest.Create(url)
// Asynchronously get response and dispose it when we're done
use! response = request.AsyncGetResponse()
use stream = response.GetResponseStream()
let temp = new MemoryStream()
let buffer = Array.zeroCreate 4096
// Loop that downloads page into a buffer (could use 'while'
// but recursion is more typical for functional language)
let rec download() = async {
let! count = stream.AsyncRead(buffer, 0, buffer.Length)
do! temp.AsyncWrite(buffer, 0, count)
if count > 0 then return! download() }
// Start the download asynchronously and handle results
do! download()
temp.Seek(0L, SeekOrigin.Begin) |> ignore
let html = (new StreamReader(temp)).ReadToEnd()
return html };;
I tried to do the following with it, and got the error on the last line:
The type was expected to have type Async<'a> but has string -> Asnyc<'a> instead
I googled the error but couldn't find anything that revealed my particular issue.
let postData = "userid=" + userId + "&password=" + password + "&source=" + sourceId + "&version=" + version
let url = postUrlBase + "100/LogIn?" + postData
Async.RunSynchronously (downloadPage(url, postData));;
Also, how would I modify the code so that it downloads a non-ending byte stream (but with occasional pauses between each burst of bytes) asynchronously instead of a string? How would I integrate reading this byte stream as it comes through? I realize this is more than one question, but since they are are all closely related I figured one question would save some time.
Thanks in advance,
Bob
P.S. As I am still new to F# please feel free to make any alterations/suggestions to my code which shows how its done in a more functional style. I'm really trying to get out of my C# mindset, so I appreciate any pointers anyone may wish to share.
Edit: I accidentally pasted in the wrong snippet I was using. I did make an alteration to Tomas' snippet and forgot about it.
When I attempt to run your code downloadPage(url, postData) doesn't work as downloadPage expects two seperate strings. downloadPage url postData is what is expected.
If you changed the let binding to tuple form, or let downloadPage(url:string, postData:string) your call would have worked as well.
To explain why you got the error you got is more complicated. Curried form creates a function that returns a function or string -> string -> Async<string> in your case. The compiler therefore saw you passing a single parameter (tuples are single items after all) and saw that the result would have to be a string -> Async<string> which is not compatible with Async<string>. Another error it could have found (and did in my case) is that string * string is not compatible with string. The exact error being Expected string but found 'a * 'b.
This is what I had:
Async.RunSynchronously (downloadPage(url, postData));;
this is what worked after continued random guessing:
Async.RunSynchronously (downloadPage url postData);;
Although, I'm not sure why this change fixed the problem. Thoughts?

Resources