I am working on a simple chat server and client in golang. I am having some trouble with reading messages from the net.Conn. So far this is what I have been doing:
bufio.NewReader(conn).ReadString('\n')
Since the user presses enter to send the message I only have to read until '\n'. But I am now working on encryption and when sending the public keys between client and server the key sometimes contains '\n', which makes it hard to get the whole key. I am just wondering how I can read the whole message instead of stopping at a specific character. Thanks!
A simple option for sending binary data is to use a length prefix. Encode the data size as a 32bit big endian integer, then read that amount of data.
// create the length prefix
prefix := make([]byte, 4)
binary.BigEndian.PutUint32(prefix, uint32(len(message)))
// write the prefix and the data to the stream (checking errors)
_, err := conn.Write(prefix)
_, err = conn.Write(message)
And to read the message
// read the length prefix
prefix := make([]byte, 4)
_, err = io.ReadFull(conn, prefix)
length := binary.BigEndian.Uint32(prefix)
// verify length if there are restrictions
message = make([]byte, int(length))
_, err = io.ReadFull(conn, message)
See also Golang: TCP client/server data delimiter
You can also of course use an existing, well test protocol, like HTTP, IRC, etc. for your messaging needs. The go std library comes with a simple textproto package, or you could opt to enclose the messages in a uniform encoding, like JSON.
Related
I'm writing a very simple golang http server app to generate http traffic for some performance testing. There are 4 files containing static content, each ~1mb in size, to be sent in response to http requests from clients on the same network.
In order to keep the response packets sent by the server < 1500 bytes, during setup i'm breaking the files up into chunks of 1420 bytes, and placing the array of chunks in a map with 4 elements, keyed by file. The request handler uses this map to construct responses.
From the documentation, it seems an http.Flusher can be used to force a packet send. The code to do this is below.
I'm seeing expected behavior > 99% of the time, but a handful of packets are way over 1500 bytes. Oversize frames always occur immediately following a client ACK packet, and only a single oversized packet.
So, have I screwed up or possibly seeing a bug?
func createHttpServer(cfg common.Config, dir string, port int, cmap *map[string][][]byte) *http.Server {
mux := http.NewServeMux()
// for each of the 4 static content files, iterate over the chunks and send a packet for each
for i := 0; i < len(cfg.Content); i++ {
location := cfg.Content[i].Location
mux.HandleFunc(location, func(w http.ResponseWriter, req *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
panic("createHttpServer(): expected http.ResponseWriter to be an http.Flusher")
}
w.Header().Add("From: joeblow#test.com", "Date: "+utcNow())
w.WriteHeader(200)
flusher.Flush()
for _, chunk := range (*cmap)[location] {
if len(chunk) > 1420 {
panic("unexpected oversized chunk")
}
w.Write(chunk)
flusher.Flush()
}
})
}
srv := &http.Server{
Addr: ":" + strconv.Itoa(port),
Handler: mux,
}
return srv
}
Additional Info: the original issue applies to testing done on windows. Under linux (same code), there are many large packets. Docs say that by default Nagle's is disabled in golang, and i've verified mtu is set to 1500. what else to check in order to figure out why results of Flush() are being coalesced into jumbo packets?
based on helpful comments by Cerise Limon and JimB, it is not possible to control the TCP packet size as I attempted to do, so the question is based on wrong assumptions
My mail goal is to read data from a TCP socket connected to a HTTP server, and parse
the HTTP response chunk by chunk (Transfer-Encoding: chunked) - the server sends a chunk every 30 seconds on the same connection
I attached my code. It seems like io.Copy reads the first chunk and then waits for the next one before returning, so it blocks me.
How can I read the chunks when they arrive, without waiting for the next one?
func readHttpChunk(server net.conn) {
buf := bufio.NewReader(server)
for {
resp, err := http.ReadResponse(buf, nil)
b := new(bytes.Buffer)
io.Copy(b, resp.Body)
fmt.Printf("Got me a chunk!")
}
}
Instead of using io.Copy use resp.Body.Read(buf) where buf is an allocated byte slice. Use a big enough buffer so that the chunk doesn’t get truncated. The read should fill buf with a chunk.
Now, i do something like this with Golang:
//read all bytes from body
bytes, err := ioutil.ReadAll(request.Body)
//set the bytes as NewReader to new request.body
request, err := http.NewRequest(http.MethodPut, url, bytes.NewReader(bytes))
but i want to streaming read from original body(io.Reader) to the new, instead of read all bytes to memory by ReadAll，and then copy to NewRequest.
How can i implement this?
thx.
Check the io package if you want specialized reader - writer handling, the Pipe or Tee structs might be useful.
In this case, though,
request, err := http.NewRequest(http.MethodPut, url, request.Body)
should actually just work fine.
How to send a message of the form [Integer, String] in erlang using gen_tcp.
Eg: I am looking to send messages of the form [25, "Hello"] or [50, "Hi"] over a socket using gen_tcp:send/2.
I tried to do [ID | Msg] but that doesn't help.
Thanks
In Erlang, a string is just a list of Integers, so the list [25, "Hello"]
is actually represented like [25, [72,101,108,108,111]]
The question remains, how to send that information over a Socket in Erlang.
According to the documentation, in Erlang you can send the data as binary or as a list, which you can set either as {mode, binary} or {mode, list} when you create the Socket.
In the case that you are working with binary data (which is advice since sending and receiving binaries is faster than lists), you can do something like:
{ok, Socket} = gen_tcp:connect(Host, Port, [{mode, binary}]).
Data = list_to_binary([25, "Hello"]).
gen_tcp:send(Socket, Data).
Now, if you use your socket in list mode, you just send the list directly:
{ok, Socket} = gen_tcp:connect(Host, Port, [{mode, binary}]).
Data = [25, "Hello"].
gen_tcp:send(Socket, Data).
At the server side, if you receive the data in a Socket which is in list mode, you convert back to your original format with:
[Integer|String] = ListReceived.
If you receive the data in a Socket with binary mode, then you have to transform the data from a Binary to a List, like:
[Integer|String] = binary_to_list(BinaryReceived).
I have a device that sends data to a server.
Data
[ Client ] == > [ Server ]
After the validation on the server I want to return a message:
OK
[ Client ] < == [ Server ]
Is there a standard "OK" message to return? And an "ERROR" message? How does it looks like? (e.g. ":0011", ":110F")
You've got to design an application-level protocol. TCP is a byte stream, so even the definition of "Data" in your client->server piece needs some protocol so that the receiver can know what bytes make up the data (when to stop reading).
A couple of common types of protocols are...
Length-delimited chunks. Every message starts with a 16 or 32-bit length prefix. Then that many bytes follow. The length needs to be in a defined byte order (see htons, ntohs, etc). Everyone who uses this protocol knows to read the length prefix then read that many bytes. Having defined that "chunk" on the network, you might put a header on the contents of the chunk. Maybe a message type (ACK, NAK, Data, etc) followed by some contents.
ASCII newline delimited. Each message is a line of ASCII (or UTF8, etc) text. It ends at a newline. Newline endings for the lines play the same role as the length prefix for the chunks above. You then define what's in each line (like space or comma-delimited ASCII/UTF8/whatever fields). Somewhere in that you'd define what data looks like, ACK, etc.
I'm sure you could come up with other ideas, but that's the basic job: defining your application-level protocol on top of TCP's byte stream.