I'm working on an IRC bot using TcpStream from the standard library.
I'm able to read all the lines that come in, but the IRC server doesn't seem to respond to my identify requests. I thought I was sending the request too soon so I tried sleeping before sending the IDENT but that doesn't work. I write using both BufReader, BufWriter and calling read and write directly on the stream to no avail.
use std::net::TcpStream;
use std::io::{BufReader, BufWriter, BufRead, Write, Read};
use std::{thread, time};
struct Rusty {
name: String,
stream: TcpStream,
reader: BufReader<TcpStream>,
writer: BufWriter<TcpStream>,
}
impl Rusty {
fn new(name: &str, address: &str) -> Rusty {
let stream = TcpStream::connect(address).expect("Couldn't connect to server");
let reader = BufReader::new(stream.try_clone().unwrap());
let writer = BufWriter::new(stream.try_clone().unwrap());
Rusty {
name: String::from(name),
reader: reader,
writer: writer,
stream: stream,
}
}
fn write_line(&mut self, string: String) {
let line = format!("{}\r\n", string);
&self.writer.write(line.as_bytes());
}
fn identify(&mut self) {
let nick = &self.name.clone();
self.write_line(format!("USER {} {} {} : {}", nick, nick, nick, nick));
self.write_line(format!("NICK {}", nick));
}
fn read_lines(&mut self) {
let mut line = String::new();
loop {
self.reader.read_line(&mut line);
println!("{}", line);
}
}
}
fn main() {
let mut bot = Rusty::new("rustyrusty", "irc.rizon.net:6667");
thread::sleep_ms(5000);
bot.identify();
bot.read_lines();
}
It's very important to read the documentation for the components we use when programming. For example, the docs for BufWriter states (emphasis mine):
Wraps a writer and buffers its output.
It can be excessively inefficient to work directly with something that
implements Write. For example, every call to write on TcpStream
results in a system call. A BufWriter keeps an in-memory buffer of
data and writes it to an underlying writer in large, infrequent
batches.
The buffer will be written out when the writer is dropped.
Said another way, the entire purpose of a buffered reader or writer is that read or write requests don't have a one-to-one mapping to the underlying stream.
That means when you call write, you are only writing to the buffer. You also need to call flush if you need to ensure that the bytes are written to the underlying stream.
Additionally, you should:
Handle the errors that can arise from read, write, and flush.
Re-familiarize yourself with what each function does. For example, read and write don't guarantee that they read or write as much data as you ask them to. They may perform a partial read or write, and it's up to you to handle that. That's why there are helper methods like read_to_end or write_all.
Clear your String that you are reading into. Otherwise the output just repeats every time the loop cycles.
Use write! instead of building up a string that is immediately thrown away.
fn write_line(&mut self, string: &str) {
write!(self.writer, "{}\r\n", string).unwrap();
self.writer.flush().unwrap();
}
With these changes, I was able to get a PING message from the server.
Related
I was wondering if anyone could give me any pointers on the best way to go about handling wake ups when writing a wrapper for a Stream.
For context I've got a Byte stream coming in via a HTTP request (using reqwest) and I'm doing some filtering and mapping on that stream to handle validation and deserialization. Effectively whenever the inner stream produces a value I want this stream to (potentially) emit a value.
** Edit **
An additional caveat is the stream needs to also hold a small amount of state (A Vec<String>) that it needs to be able to reference on each poll - (the columns property)
The Solution
This turned out to be me just not understanding how the stream was working under the hood. Rodrigo's answer below was completely correct. I did just need to return Poll::Pending from the inner stream, however I was making the mistake of matching on that and returning my own Poll::Pending which was why the stream wasn't being appropriately woken up.
If it's useful to anyone, instead of matching on the output of inner_stream.poll_next(), I ended up just mapping the Some value and returning that to ensure that I was building off the Polls of the inner stream eg:
return Pin::new(&mut this.stream).poll_next(cx).map(|data| { ... })
Thanks for everyone who commented and helped out!
Context for the original question
The wrapper type:
pin_project! {
#[derive(Default)]
struct QueryStream<T, S> where S: Stream, T: DeserializeOwned {
columns: Vec<String>,
#[pin]
stream: S,
has_closed: bool,
_marker: PhantomData<T>
}
}
The only implementation of Stream that I've managed to get to work on the wrapper type is one that spins on the inner stream when it returns Poll::Pending. This doesn't seem ideal though as I believe it would block until a value is emitted?
impl<T, S> Stream for QueryStream<T, S>
where
T: DeserializeOwned,
S: Stream<Item = std::result::Result<Bytes, reqwest::Error>>,
{
type Item = Result<T>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let mut this = self.project();
loop {
if *this.has_closed {
return Poll::Ready(None);
}
match Pin::new(&mut this.stream).poll_next(cx) {
Poll::Ready(Some(data)) => {
// Parsing Logic Here
return Poll::Ready(Some(Ok::<_, Error>(resp)));
}
Poll::Ready(None) => return Poll::Ready(None),
Poll::Pending => {}
}
}
}
}
Trying to remove the loop (and changing the Poll::Pending match arm to Poll::Pending => Poll::Pending) generally results in poll only being called once before hanging, from my very rough understanding of why this is, it's because I'm dropping the reference to the waker when I return from this function, as it's not stored anywhere.
However I'm struggling to work out how to arrange my struct/code to enable the storage of that reference or alternatively what the best way to enable the use of that waker is? Is anyone able to explain how this problem can be solved?
Many thanks in advance!
I'm quite new to Rust, I'm mainly a C#, javascript and python developer, so I like to approach things in a OOP way, however I still can't wrap my head around ownership in rust. Especially when it comes to OOP.
I'm writing a TCP server. I have a struct that contains connections (streams) and I read the sockets asynchronously using the mio crate. I understand what the error is telling me, but I have no clue how to fix it. I tried changing the read_message method into a function (without the reference to self), which worked, but the problem with this is that I'll need to access the connections and whatnot from the struct (to relay messages between sockets for example), so this workaround won't be plausible in later versions. Is there an easy fix for this, or is the design inherently flawed?
Here's a snippet that shows what my problem is:
let sock = self.connections.get_mut(&token).unwrap();
loop {
match sock.read(&mut msg_type) {
Ok(_) => {
self.read_message(msg_type[0], token);
}
}
}
fn read_message(&mut self, msg_type: u8, token: Token) {
let sock = self.connections.get_mut(&token).unwrap();
let msg_type = num::FromPrimitive::from_u8(msg_type);
match msg_type {
Some(MsgType::RequestIps) => {
let decoded: MsgTypes::Announce = bincode::deserialize_from(sock).unwrap();
println!("Public Key: {}", decoded.public_key);
}
_ => unreachable!()
}
}
And the error I'm getting is the following:
You are holding a mutable borrow on sock, which is part of self, at the moment you try to call self.read_message. Since you indicated that read_message needs mutable access to all of self, you need to make sure you don't have a mutable borrow on sock anymore at that point.
Fortunately, thanks to non-lexical lifetimes in Rust 2018, that's not hard to do; simply fetch sock inside the loop:
loop {
let sock = self.connections.get_mut(&token).unwrap();
match sock.read(&mut msg_type) {
Ok(_) => {
self.read_message(msg_type[0], token);
}
}
}
Assuming sock.read doesn't return anything that holds a borrow on sock, this should let the mutable borrow on sock be released before calling self.read_message. It needs to be re-acquired in the next iteration, but seeing as you're doing network I/O, the relative performance penalty of a single HashMap (?) access should be negligible.
(Due to lack of a minimal, compileable example, I wasn't able to test this.)
In my RSS reader project, I want to read my RSS feeds asynchronously. Currently, they're read synchronously thanks to this code block
self.feeds = self
.feeds
.iter()
.map(|f| f.read(&self.settings))
.collect::<Vec<Feed>>();
I want to make that code asynchronous, because it will allow me to better handle poor web server responses.
I understand I can use a Stream that I can create from my Vec using stream::from_iter(...) which transforms the code into something like
self.feeds = stream::from_iter(self.feeds.iter())
.map(|f| f.read(&self.settings))
// ???
.collect::<Vec<Feed>>()
}
But then, I have two questions
How to have results joined into a Vec (which is a synchronous struct)?
How to execute that stream? I was thinking about using task::spawn but it doesn't seems to work ...
How to execute that stream? I was thinking about using task::spawn but it doesn't seems to work
In the async/await world, asynchronous code is meant to be executed by an executor, which is not part of the standard library but provided by third-party crates such as tokio. task::spawn only schedules one instance of async fn to run, not actually running it.
How to have results joined into a vec (which is a sync struct)
The bread and butter of your rss reader seems to be f.read. It should be turned into an asynchronous function. Then the vector of feeds will be mapped into a vector of futures, which need to be polled to completion.
The futures crate has futures::stream::futures_unordered::FuturesUnordered to help you do that. FuturesUnordered itself implements Stream trait. This stream is then collected into the result vector and awaited to completion like so:
//# tokio = { version = "0.2.4", features = ["full"] }
//# futures = "0.3.1"
use tokio::time::delay_for;
use futures::stream::StreamExt;
use futures::stream::futures_unordered::FuturesUnordered;
use std::error::Error;
use std::time::{Duration, Instant};
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let start = Instant::now();
let feeds = (0..10).collect::<Vec<_>>();
let res = read_feeds(feeds).await;
dbg!(res);
dbg!(start.elapsed());
Ok(())
}
async fn read_feeds(feeds: Vec<u32>) -> Vec<u32> {
feeds.iter()
.map(read_feed)
.collect::<FuturesUnordered<_>>()
.collect::<Vec<_>>()
.await
}
async fn read_feed(feed: &u32) -> u32 {
delay_for(Duration::from_millis(500)).await;
feed * 2
}
delay_for is to simulate the potentially expensive operation. It also helps to demonstrate that these readings indeed happen concurrently without any explicit thread related logic.
One nuance here. Unlike its synchronous counterpart, the results of reading rss feeds aren't in the same order of feeds themselves any more, whichever returns the first will be at the front. You need to deal with that somehow.
I am attempting to create simplest possible example that can get async fn hello() to eventually print out Hello World!. This should happen without any external dependency like tokio, just plain Rust and std. Bonus points if we can get it done without ever using unsafe.
#![feature(async_await)]
async fn hello() {
println!("Hello, World!");
}
fn main() {
let task = hello();
// Something beautiful happens here, and `Hello, World!` is printed on screen.
}
I know async/await is still a nightly feature, and it is subject to change in the foreseeable future.
I know there is a whole lot of Future implementations, I am aware of the existence of tokio.
I am just trying to educate myself on the inner workings of standard library futures.
My helpless, clumsy endeavours
My vague understanding is that, first off, I need to Pin task down. So I went ahead and
let pinned_task = Pin::new(&mut task);
but
the trait `std::marker::Unpin` is not implemented for `std::future::GenFuture<[static generator#src/main.rs:7:18: 9:2 {}]>`
so I thought, of course, I probably need to Box it, so I'm sure it won't move around in memory. Somewhat surprisingly, I get the same error.
What I could get so far is
let pinned_task = unsafe {
Pin::new_unchecked(&mut task)
};
which is obviously not something I should do. Even so, let's say I got my hands on the Pinned Future. Now I need to poll() it somehow. For that, I need a Waker.
So I tried to look around on how to get my hands on a Waker. On the doc it kinda looks like the only way to get a Waker is with another new_unchecked that accepts a RawWaker. From there I got here and from there here, where I just curled up on the floor and started crying.
This part of the futures stack is not intended to be implemented by many people. The rough estimate that I have seen in that maybe there will be 10 or so actual implementations.
That said, you can fill in the basic aspects of an executor that is extremely limited by following the function signatures needed:
async fn hello() {
println!("Hello, World!");
}
fn main() {
drive_to_completion(hello());
}
use std::{
future::Future,
ptr,
task::{Context, Poll, RawWaker, RawWakerVTable, Waker},
};
fn drive_to_completion<F>(f: F) -> F::Output
where
F: Future,
{
let waker = my_waker();
let mut context = Context::from_waker(&waker);
let mut t = Box::pin(f);
let t = t.as_mut();
loop {
match t.poll(&mut context) {
Poll::Ready(v) => return v,
Poll::Pending => panic!("This executor does not support futures that are not ready"),
}
}
}
type WakerData = *const ();
unsafe fn clone(_: WakerData) -> RawWaker {
my_raw_waker()
}
unsafe fn wake(_: WakerData) {}
unsafe fn wake_by_ref(_: WakerData) {}
unsafe fn drop(_: WakerData) {}
static MY_VTABLE: RawWakerVTable = RawWakerVTable::new(clone, wake, wake_by_ref, drop);
fn my_raw_waker() -> RawWaker {
RawWaker::new(ptr::null(), &MY_VTABLE)
}
fn my_waker() -> Waker {
unsafe { Waker::from_raw(my_raw_waker()) }
}
Starting at Future::poll, we see we need a Pinned future and a Context. Context is created from a Waker which needs a RawWaker. A RawWaker needs a RawWakerVTable. We create all of those pieces in the simplest possible ways:
Since we aren't trying to support NotReady cases, we never need to actually do anything for that case and can instead panic. This also means that the implementations of wake can be no-ops.
Since we aren't trying to be efficient, we don't need to store any data for our waker, so clone and drop can basically be no-ops as well.
The easiest way to pin the future is to Box it, but this isn't the most efficient possibility.
If you wanted to support NotReady, the simplest extension is to have a busy loop, polling forever. A slightly more efficient solution is to have a global variable that indicates that someone has called wake and block on that becoming true.
I am working on an echo server which takes data from TCP and applies some logic to that data. For example, if the client data comes in as hello I want to respond it ashello from server.
I am able to forward the input data using the copy function, but this is not useful in my case.
Here is the starting code that I am working on:
extern crate futures;
extern crate tokio_core;
extern crate tokio_io;
use futures::stream::Stream;
use futures::Future;
use std::net::SocketAddr;
use tokio_core::net::TcpListener;
use tokio_core::reactor::Core;
use tokio_io::io::copy;
use tokio_io::AsyncRead;
fn main() {
let addr = "127.0.0.1:15000".parse::<SocketAddr>().unwrap();
let mut core = Core::new().unwrap();
let handle = core.handle();
let socket = TcpListener::bind(&addr, &handle).unwrap();
println!("Listening on: {}", addr);
let done = socket.incoming().for_each(move |(socket, addr)| {
let (reader, writer) = socket.split();
let amt = copy(reader, writer);
let msg = amt.then(move |result| {
match result {
Ok((amt, _, _)) => println!("wrote {} bytes to {}", amt, addr),
Err(e) => println!("error on {}: {}", addr, e),
}
Ok(())
});
handle.spawn(msg);
Ok(())
});
core.run(done).unwrap();
}
I know that I need to add some logic instead of this copy function but how?
let amt = copy(reader, writer);
An echo server is kind of special in a sense, that exactly one "request" from a client is followed by exactly one response from the server. A very nice example for such a use-case is tokio's TinyDB example.
One thing that should be considered, however, is that while UDP is based on packets, that hit the other side in the exact form that you sent them with, TCP is not. TCP is a stream protocol - it has strong guarantees relating that a packet was received by the other side and that the data sent is received in exactly the order it was sent in. However, what is not guaranteed is, that one call to "send" on the one side leads to exactly one "receive" call on the other side, returning the exact same chunk of data that was sent. This is especially of interest when sending very long chunks of data, where one send maps to multiple receives. Thus you should settle for a delimiter that the server can wait for before trying to send a response to the client. In Telnet, that delimiter would be "\r\n".
That is where tokio's Decoder/Encoder infrastructure comes to play. An example implementation of such a codec is LinesCodec. If you want to have
Telnet, this does exactly what you want. It will give you exactly one message at a time and allow you to send exactly one such message at a time as response:
extern crate tokio;
use tokio::codec::Decoder;
use tokio::net::TcpListener;
use tokio::prelude::*;
use tokio::codec::LinesCodec;
use std::net::SocketAddr;
fn main() {
let addr = "127.0.0.1:15000".parse::<SocketAddr>().unwrap();
let socket = TcpListener::bind(&addr).unwrap();
println!("Listening on: {}", addr);
let done = socket.incoming()
.map_err(|e| println!("failed to accept socket; error = {:?}", e))
.for_each(move |socket| {
// Fit the line-based codec on top of the socket. This will take on the task of
// parsing incomming messages, as well as formatting outgoing ones (appending \r\n).
let (lines_tx, lines_rx) = LinesCodec::new().framed(socket).split();
// This takes every incomming message and allows to create one outgoing message for it,
// essentially generating a stream of responses.
let responses = lines_rx.map(|incomming_message| {
// Implement whatever transform rules here
if incomming_message == "hello" {
return String::from("hello from server");
}
return incomming_message;
});
// At this point `responses` is a stream of `Response` types which we
// now want to write back out to the client. To do that we use
// `Stream::fold` to perform a loop here, serializing each response and
// then writing it out to the client.
let writes = responses.fold(lines_tx, |writer, response| {
//Return the future that handles to send the response to the socket
writer.send(response)
});
// Run this request/response loop until the client closes the connection
// Then return Ok(()), ignoring all eventual errors.
tokio::spawn(
writes.then(move |_| Ok(()))
);
return Ok(());
});
tokio::run(done);
}