My mail goal is to read data from a TCP socket connected to a HTTP server, and parse
the HTTP response chunk by chunk (Transfer-Encoding: chunked) - the server sends a chunk every 30 seconds on the same connection
I attached my code. It seems like io.Copy reads the first chunk and then waits for the next one before returning, so it blocks me.
How can I read the chunks when they arrive, without waiting for the next one?
func readHttpChunk(server net.conn) {
buf := bufio.NewReader(server)
for {
resp, err := http.ReadResponse(buf, nil)
b := new(bytes.Buffer)
io.Copy(b, resp.Body)
fmt.Printf("Got me a chunk!")
}
}
Instead of using io.Copy use resp.Body.Read(buf) where buf is an allocated byte slice. Use a big enough buffer so that the chunk doesn’t get truncated. The read should fill buf with a chunk.
Related
I'm writing a very simple golang http server app to generate http traffic for some performance testing. There are 4 files containing static content, each ~1mb in size, to be sent in response to http requests from clients on the same network.
In order to keep the response packets sent by the server < 1500 bytes, during setup i'm breaking the files up into chunks of 1420 bytes, and placing the array of chunks in a map with 4 elements, keyed by file. The request handler uses this map to construct responses.
From the documentation, it seems an http.Flusher can be used to force a packet send. The code to do this is below.
I'm seeing expected behavior > 99% of the time, but a handful of packets are way over 1500 bytes. Oversize frames always occur immediately following a client ACK packet, and only a single oversized packet.
So, have I screwed up or possibly seeing a bug?
func createHttpServer(cfg common.Config, dir string, port int, cmap *map[string][][]byte) *http.Server {
mux := http.NewServeMux()
// for each of the 4 static content files, iterate over the chunks and send a packet for each
for i := 0; i < len(cfg.Content); i++ {
location := cfg.Content[i].Location
mux.HandleFunc(location, func(w http.ResponseWriter, req *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
panic("createHttpServer(): expected http.ResponseWriter to be an http.Flusher")
}
w.Header().Add("From: joeblow#test.com", "Date: "+utcNow())
w.WriteHeader(200)
flusher.Flush()
for _, chunk := range (*cmap)[location] {
if len(chunk) > 1420 {
panic("unexpected oversized chunk")
}
w.Write(chunk)
flusher.Flush()
}
})
}
srv := &http.Server{
Addr: ":" + strconv.Itoa(port),
Handler: mux,
}
return srv
}
Additional Info: the original issue applies to testing done on windows. Under linux (same code), there are many large packets. Docs say that by default Nagle's is disabled in golang, and i've verified mtu is set to 1500. what else to check in order to figure out why results of Flush() are being coalesced into jumbo packets?
based on helpful comments by Cerise Limon and JimB, it is not possible to control the TCP packet size as I attempted to do, so the question is based on wrong assumptions
Now, i do something like this with Golang:
//read all bytes from body
bytes, err := ioutil.ReadAll(request.Body)
//set the bytes as NewReader to new request.body
request, err := http.NewRequest(http.MethodPut, url, bytes.NewReader(bytes))
but i want to streaming read from original body(io.Reader) to the new, instead of read all bytes to memory by ReadAll，and then copy to NewRequest.
How can i implement this?
thx.
Check the io package if you want specialized reader - writer handling, the Pipe or Tee structs might be useful.
In this case, though,
request, err := http.NewRequest(http.MethodPut, url, request.Body)
should actually just work fine.
I am working on a simple chat server and client in golang. I am having some trouble with reading messages from the net.Conn. So far this is what I have been doing:
bufio.NewReader(conn).ReadString('\n')
Since the user presses enter to send the message I only have to read until '\n'. But I am now working on encryption and when sending the public keys between client and server the key sometimes contains '\n', which makes it hard to get the whole key. I am just wondering how I can read the whole message instead of stopping at a specific character. Thanks!
A simple option for sending binary data is to use a length prefix. Encode the data size as a 32bit big endian integer, then read that amount of data.
// create the length prefix
prefix := make([]byte, 4)
binary.BigEndian.PutUint32(prefix, uint32(len(message)))
// write the prefix and the data to the stream (checking errors)
_, err := conn.Write(prefix)
_, err = conn.Write(message)
And to read the message
// read the length prefix
prefix := make([]byte, 4)
_, err = io.ReadFull(conn, prefix)
length := binary.BigEndian.Uint32(prefix)
// verify length if there are restrictions
message = make([]byte, int(length))
_, err = io.ReadFull(conn, message)
See also Golang: TCP client/server data delimiter
You can also of course use an existing, well test protocol, like HTTP, IRC, etc. for your messaging needs. The go std library comes with a simple textproto package, or you could opt to enclose the messages in a uniform encoding, like JSON.
In my simple TCP client server application, server send repetitively 1 kB message to the client and client send a reply acknowledgement (just send 'ACK') for each packet. Just think this scenario like client and server passing 1 kB messages here and there in a infinite loop.
I send the same message every time and the fist byte (first char) is always 1. But while testing this client and server application in the same machine for a long time, I noticed first character of some of the received messages are something else in the receive buffer and recv function also returned 1024 (1 kB). This is not happen frequently.
This is the how I receive.
char recvBuff[DEFAULT_BUFFER_SIZE];
int iResult = SOCKET_ERROR;
iResult = recv(curSocket, recvBuff, DEFAULT_BUFFER_SIZE, 0);
if (iResult == SOCKET_ERROR)
{
return iResult;
}
if (recvBuff[0] != 1)
{
//malformed receive
}
MessageHeader *q = (MessageHeader*)recvBuff;
message.header = *q; q++;
std::string temp((char*)q, message.header.fragmentSize);
message.message = temp;
Actually the problem is in constructing the temp string. It breaks since the correct fragment size not received. I tried to drop these kind of malformed data. But the problem is there is a gap between last successfully received fragment ID and first successfully received fragment ID after malformed receives. Any idea why these malformed receives happen?
You’re assuming that you’ve received a complete message when the recv() call completes. If this is a TCP connection (as opposed to UDP), it is byte-oriented, and that means that recv() will return whenever there are any bytes available.
Put more explicitly, there is no reason that doing
send (toServerSocket, someMessage, 1024, 0);
on the client side will cause
recv (fromClientSocket, myBuffer, 1024, 0);
to receive 1,024 bytes. It could just as well receive 27 bytes, with the remaining 997 coming from future calls to recv().
What’s happening in your program, then, is that you’re getting one of these short returns, and it’s causing your program to lose sync. with the message stream. How to fix it? Use recv() to read enough of your message that you know the length (or set a fixed length, though that’s inefficient in many cases). Then continue calling recv() into your buffer until you have read at least that many bytes. Note that you might read more bytes than the length of your message — that is, you may read some bytes that belong to the next message, so you will need to keep those in the buffer after processing the current message.
I've been looking into Mochiweb, but I can't find a way to read the body while I'm receiving the request through the socket.
I'm not limited to Mochiweb, any other HTTP library would be good for me.
I also tried gen_tcp:listen(Port, [{packet, http}]), this way I can read the body/headers while I'm receiving the HTTP request, but I must handle manually the responses and keeping the socket open for more requests, so I prefer not to use this solution.
My intention is to receive request with large bodies and not to wait to receive the full body in order to start reading/processing them.
With mochiweb you can fold over chunks of the request body using Req:stream_body/3.
It expects a chunk handler function as the second argument. This handler is called with
{ChunkSize, BinaryData} and your state for every chunk, as it is received from the socket.
Example (retrieving a [reversed] list of chunks):
MaxChunkSize = 100,
InitialState = [],
ChunkHandler = fun ({_Size, Bin}, State) -> [Bin | State] end,
List = Req:stream_body(MaxChunkSize, ChunkHandler, InitialState),
...