I would like to split a stream in Rust, but I'd like to do it idiomatically without side effects in my stream functions.
This is what I have now where I pipe the stream result into a for_each
let (sender, receiver) tokio::sync::broadcast::channel(1);
let mut base_stream = inf_async_stream_gen().await
.map(function1)
.for_each(|x| {
let inner_sender = sender.clone();
async move { inner_sender.send(x); }
});
let stream_1 = BroadcastStream::new(receiver.resubscribe());
let stream_2 = BroadcastStream::new(receiver.resubscribe());
But this feels a little bit odd. Isn't receiver conceptually a Sink (despite not implementing the trait)?
Related
I am quite new to Rust and have some problems with async and lifetimes.
I am trying to read a USB cam in a loop using rscam crate.
The problem is that after some time the camera driver tends to freeze and returns no data.
Rscam's capture has no builtin timeout (at least the official release) - so I am trying to wrap it in tokio's timeout.
Because all runs in a loop I do not move the camera struct, but pass a reference. And here is my difficulty, as sending into future requires a static lifetime (which I cannot figure out how to do).
Or maybe there is a completely different solution to adding timeouts for blocking io?
let mut camera = Camera::new(&settings.device).unwrap();
camera.set_control(CID_JPEG_COMPRESSION_QUALITY, &95);
camera.start(&Config {
interval: (1, 30),
resolution: (settings.width, settings.height),
format: settings.format.as_bytes(),
..Default::default()
}).unwrap();
let cam_ref = &camera;
let duration = time::Duration::from_millis(settings.timeout);
loop {
let loop_start = time::Instant::now();
let frame;
let future = task::spawn_blocking(|| {
cam_ref.capture()
});
if let Ok(result) = tokio::time::timeout(duration, future).await {
frame = result.unwrap();
} else {
println!("Restarting camera");
break;
}
let buf = decode_jpeg(&frame.unwrap());
// etc
}
I am trying to make a echo server that capitalize a String when it replies, to practice with tokio as an exercise. I used an array as a buffer which is annoying because what if the string overflows the buffer?
I would like to know if there is a better way to this without using an array, ideally just using a String or a vector without needing to create the buffer array.
I tried read_from_string() but is not async and ends up blocking the socket.
extern crate tokio;
use tokio::net::TcpListener;
use tokio::prelude::*;
fn main() {
let addr = "127.0.0.1:6142".parse().unwrap();
let listener = TcpListener::bind(&addr).unwrap();
let server = listener
.incoming()
.for_each(|socket| {
let (mut reader, mut writer) = socket.split();
let mut buffer = [0; 16];
reader.poll_read(&mut buffer)?;
let s = std::str::from_utf8(&buffer).unwrap();
s.to_uppercase();
writer.poll_write(&mut s.as_bytes())?;
Ok(())
})
.map_err(|e| {
eprintln!("something went wrong {}", e);
});
tokio::run(server);
}
Results:
"012345678901234567890" becomes -> "0123456789012345"
I could increase the buffer of course but it would just kick the can down the road.
I believe tokio_codec is a right tool for such tasks. Tokio documentation: https://tokio.rs/docs/going-deeper/frames/
It uses Bytes / BytesMut as its buffer - very powerful structure which will allow you to process your data however you want and avoid unnecessary copies
This question already has answers here:
How to read subprocess output asynchronously
(2 answers)
How do I read the output of a child process without blocking in Rust?
(4 answers)
Closed 3 years ago.
I have a subprocess, which may or may not write something to it's stdout in a specific amount of time, e.g. 3 seconds.
If a new line in the subprocess stdout starts with the correct thing, I want to return the line.
Optimally I would like to realize something like this:
use std::io::{BufRead, BufReader};
use std::thread;
use std::time::Duration;
pub fn wait_for_or_exit(
reader: &BufReader<&mut std::process::ChildStdout>,
wait_time: u64,
cmd: &str,
) -> Option<String> {
let signal: Arc<AtomicBool> = Arc::new(AtomicBool::new(false));
let signal_clone = signal.clone();
let child = thread::spawn(move || {
thread::sleep(Duration::from_millis(wait_time));
signal_clone.store(true, Ordering::Relaxed);
});
let mut line = String::new();
while !signal.load(Ordering::Relaxed) {
//Sleep a really small amount of time not to block cpu
thread::sleep(Duration::from_millis(10));
//This line is obviously invalid!
if reader.has_input() {
line.clear();
reader.read_line(&mut line).unwrap();
if line.starts_with(cmd) {
return Some(line);
}
}
}
None
}
The only line not working here is reader.has_input().
Obviously, if the subprocess answers much faster than the wait_time for a repeated amount of times, there will be a lot of sleeping threads, but I can take care of that with channels.
There are two approaches.
You can spin up a separate thread, and then use some mechanism (probably a channel) to signal success or failure to your waiting thread.
You can use async IO as you mentioned, such as the futures and tokio lib.
I'll demo both. I prefer the futures/Tokio approach, but if you're not familiar with the futures model, then option one might be better.
The Rust stdlib has a Channels API, and this channel actually features a recv_timeout which can help us out quite a bit.
use std::thread;
use std::time::Duration;
use std::sync::mpsc;
// this spins up a separate thread in which to wait for stuff to read
// from the BufReader<ChildStdout>
// If we successfully read, we send the string over the Channel.
// Back in the original thread, we wait for an answer over the channel
// or timeout in wait_time secs.
pub fn wait_for_or_exit(
reader: &BufReader<&mut std::process::ChildStdout>,
wait_time: u64,
cmd: &str,
) -> Option<String> {
let (sender, receiver) = mpsc::channel();
thread::spawn(move || {
let line = reader.read_line();
sender.send(line);
});
match receiver.recv_timeout(Duration::from_secs(wait_time)) {
Ok(line) => if line.starts_with(cmd)
{ Some(line) } else
{ None },
Err(mpsc::RecvTimeoutError::Timeout) => None,
Err(mpsc::RecvTimeoutError::Disconnected) => None
}
}
Option two assumes that you're building a future's based app. In order to accomplish what you want using Async IO is a file descriptor that will let us set NON_BLOCKING. Luckily we don't have to do that ourselves. The Futures and Tokio APIs handle this nicely. The trade-off, is that you have to compose your code out of non-blocking futures.
The code below was taken almost entirely from Tokio Process with a Futures timeout that comes from the Tokio API.
extern crate futures;
extern crate tokio;
extern crate tokio_process;
use std::process::Command;
use std::time::{Duration};
use futures::Future;
use tokio_process::CommandExt;
use tokio::prelude::*;
const TIMEOUT_SECS: u64 = 3;
fn main() {
// Like above, but use `output_async` which returns a future instead of
// immediately returning the `Child`.
let output = Command::new("echo").arg("hello").arg("world")
.output_async();
let future = output.map_err(|e| panic!("failed to collect output: {}", e))
.map(|output| {
assert!(output.status.success());
assert_eq!(output.stdout, b"hello world\n");
println!("received output: {}", String::from_utf8(output.stdout).unwrap());
})
.timeout(Duration::from_secs(TIMEOUT_SECS)) // here is where we say we only want to wait TIMETOUT seconds
.map_err(|_e| { println!("Timed out waiting for data"); });
tokio::run(future);
}
I am using tokio to implement a server which communicates with messages serialized with serde (bincode). Without asynchronous and futures I would do
extern crate tokio_io;
extern crate bincode;
extern crate serde;
extern crate bytes;
extern crate futures;
#[macro_use] extern crate serde_derive;
use tokio_io::{AsyncRead, AsyncWrite};
use tokio_io::io::{read_exact, write_all};
use bincode::{serialize, deserialize, deserialize_from, Infinite, serialized_size};
use std::io::Read;
use std::io::Cursor;
use futures::future::Future;
type Item = String; // Dummy, this is a complex struct with derived Serizalize
type Error = bincode::Error;
// This works
fn decode<R>(reader: &mut R) -> Result<Item, Error> where R: Read {
let message: Item = deserialize_from(reader, Infinite)?;
Ok(message)
}
fn main() {
let ser = serialize("Test", Infinite).unwrap();
let buf = Cursor::new(ser);
let mut reader = std::io::BufReader::new(buf);
println!("{:?}", decode(&mut reader))
}
But what I need is a decode function which can work with an asyncronous socket as
// I need this since I get the reader from a (tokio) socket as
// let socket = TcpListener::bind(&addr, &handle).unwrap();
// let (reader, writer) = socket.split();
fn decode_async<R>(reader: R) -> Result<Item, Error> where R: AsyncRead {
// Does not work:
let message: Item = deserialize_from(reader, Infinite)?;
Ok(message)
}
The only idea I have is to manually write the length into the buffer during encoding and then use read_exact:
// Encode with size
fn encode_async(item: &Item) -> Result<Vec<u8>, Error>{
let size = serialized_size(item);
let mut buf = serialize(&size, Infinite).unwrap();
let ser = serialize(item, Infinite).unwrap();
buf.extend(ser);
Ok(buf)
}
// Decode with size
fn decode_async<R>(reader: R) -> Box<Future<Item = Item, Error = std::io::Error>>
where R: AsyncRead + 'static {
let read = read_exact(reader, vec![0u8; 8]).and_then(|(reader, buf)| {
let size = deserialize::<u64>(&mut &buf[..]).unwrap();
Ok((reader, size as usize))
}).and_then(|(reader, size)| {
read_exact(reader, vec![0u8; size])
}).and_then(|(reader, buf)| {
let item = deserialize(&mut &buf[..]).unwrap();
Ok(item)
});
Box::new(read)
}
fn main() {
let ser = encode_async(&String::from("Test")).unwrap();
let buf = Cursor::new(ser);
let mut reader = std::io::BufReader::new(buf);
let dec = decode_async(reader).wait();
println!("{:?}", dec)
}
Is there a better way to implement the decoding?
deserialize_from can't handle IO errors, especially not of the kind WouldBlock which is returned by async (non-blocking) Readers when they are waiting for more data. That is limited by the interface: deserialize_from doesn't return a Future or a partial state, it returns the full decoded Result and wouldn't know how to combine the Reader with an event loop to handle WouldBlock without busy looping.
Theoretically, it is possible to implement an async_deserialize_from, but not by using the interfaces provided by serde unless you read the full data to decode in advance, which would defeat the purpose.
You need to read the full data using tokio_io::io::read_to_end or tokio_io::io::read_exact (what you're currently using), if you know the size of the encoded data in an "endless" stream (or in a stream followed by other data).
Stefan's answer is correct, however you might be interested in looking at the tokio-serde-* family of crates which do this for you, specifically tokio-serde-bincode. From the readme:
Utilities needed to easily implement a Tokio Bincode transport using serde for serialization and deserialization of frame values. Based on tokio-serde.
The crate has several examples of how to use it.
I want to read data from a TCP stream but it results in an empty Vec:
extern crate net2;
use net2::TcpBuilder;
use std::io::Read;
use std::io::Write;
use std::io::BufReader;
let tcp = TcpBuilder::new_v4().unwrap();
let mut stream = tcp.connect("127.0.0.1:3306").unwrap();
let mut buf = Vec::with_capacity(1024);
stream.read(&mut buf);
println!("{:?}", buf); // prints []
When I use stream.read_to_end the buffer is filled but this takes way too long.
In Python I can do something like
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 3306
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
#s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
s.close()
print "received data:", data
How can I achieve this in Rust?
The two methods you tried don't work for different reasons:
read(): "does not provide any guarantees about whether it blocks waiting for data". In general, read() is unreliable from a users perspective and should only be used as a building block for higher level functions, like read_to_end().
But maybe more importantly, you have a bug in your code: you create your vector via with_capacity() which reserves memory internally, but doesn't change the length of the vector. It is still empty! When you now slice it like &buf, you pass an empty slice to read(), thus read() cannot read any actual data. To fix that, the elements of your vector need to be initialized: let mut buf = vec![0; 1024] or something like that.
read_to_end(): calls read() repeatedly until EOF is encountered. This doesn't really make sense in most TCP stream situations.
So what should you use instead? In your Python code you read a specific number of bytes into a buffer. You can do that in Rust, too: read_exact(). It works like this:
const BUFFER_SIZE: usize = 1024;
let mut stream = ...;
let mut buf = [0; BUFFER_SIZE];
stream.read_exact(&mut buf);
println!("{:?}", buf);
You could also use take(). That way you can use read_to_end():
const BUFFER_SIZE: usize = 1024;
let mut stream = ...;
let mut buf = Vec::with_capacity(BUFFER_SIZE);
stream.take(BUFFER_SIZE).read_to_end(&mut buf);
println!("{:?}", buf);
If you want to use the stream multiple times, you probably want to use by_ref() before calling take().
The two code snippets are not equivalent though! Please read the documentation for more details.