In a TcpClient/TcpListener set up, is there any difference from the receiving end point of view between:
// Will sending a prefixed length before the data...
client.GetStream().Write(data, 0, 4); // Int32 payload size = 80000
client.GetStream().Write(data, 0, 80000); // payload
// Appear as 80004 bytes in the stream?
// i.e. there is no end of stream to demarcate the first Write() from the second?
client.GetStream().Write(data, 0, 80004);
// Which means I can potentially read more than 4 bytes on the first read
var read = client.GetStream().Read(buffer, 0, 4082); // read could be any value from 0 to 4082?
I noticed that DataAvailable and return value of GetStream().Read() does not reliably tell whether there are incoming data on the way. Do I always need to write a Read() loop to exactly read the first 4 bytes?
// Read() loop
var ms = new MemoryStream()
while(ms.length < 4)
{
read = client.GetStream().Read(buffer, 0, 4 - ms.length);
if(read > 0)
ms.Write(buffer, 0, read);
}
The answer seems to be yes, we have to always be responsible for reading the same number of bytes that was sent. In other words, there has to be an application level protocol to read exactly what was written on to the underlying stream because it does not know when a new message start or ends.
Related
I have a TCP Client in rust, which should communicate with a Java Server. I got the basics working and can send bytearrays between them.
But for the bytearray buffer, I need to know the length of the bytearray. But I don't know I should obtain it. At the moment, I only have a fixed size for the buffer right now.
My Rust code looks like this:
loop {
let mut buffer = vec![0; 12]; //fixed buffer length
let n = stream.read(&mut buffer).await;
let text = from_utf8(&buffer).unwrap();
println!("{}", text);
}
In Java, you can send the size of the buffer directly as an Integer with DataInputStream. Is there any option to do that in rust?
For example, this is how I'm doing it in Java:
public String readMsg(Socket socket) throws IOException {
DataInputStream in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
byte[] bytes = new byte[in.readInt()]; //dynamic buffer length
in.readFully(bytes);
return new String(bytes, StandardCharsets.US_ASCII);
}
What you want to know is a property of the protocol that you are using. It's not a property of the programming language you use. Based on your Java code it seems like you are using a protocol which sends a 4 byte length field before the message data (signed/unsigned?).
If that is the case you can handle reading the message the same way in Rust:
1. Read the 4 bytes in order to obtain the length information
2. Read the remaining data
3. Deserialize the data
fn read_message(stream: Read) -> io::Result<String> {
let mut buffer = [0u8; 4];
// Read the length information
stream.read_exact(&mut buffer[..])?;
// Deserialize the length
let size = u32::from_be_bytes(buffer);
// Allocate a buffer for the message
// Be sure to check against a maximum size before doing this in production
let mut payload = vec![0; size];
stream.read_exact(&mut payload[..]).await;
// Convert the buffer into a string
let text = String::from_utf8(payload).map_err(/* omitted */)?;
println!("{}", text);
Ok(text)
}
This obviously is only correct if your protocol uses length prefixed messages with a 4byte unsigned int prefix. This is something that you need to check.
My goal is to read some bytes from a TcpStream in order to parse the data in each message and build a struct from it.
loop {
let mut buf: Vec<u8> = Vec::new();
let len = stream.read(&mut buf)?;
if 0 == len {
//Disconnected
}
println!("read() -> {}", len);
}
Like in Python, I thought the stream.read() would block until it received some data.
So I've set up a server that calls the loop you see above for each incoming connection. I've then tried to connect to the server with netcat; netcat connects successfully to the server and blocks on the stream.read(), which is what I want; but as soon as I send some data, read() returns 0.
I've also tried doing something similar with stream.read_to_end() but it only appears to only return when the connection is closed.
How can I read from the TcpStream, message per message, knowing that each message can have a different, unknown, size ?
You're getting caught with your pants down by an underlying technicality of Vec more than by std::io::Read, although they both interact in this particular case.
The definition and documentation of Read states:
If the return value of this method is Ok(n), then it must be guaranteed that 0 <= n <= buf.len(). A nonzero n value indicates that the buffer buf has been filled in with n bytes of data from this source. If n is 0, then it can indicate one of two scenarios:
The important part is bolded.
When you define a new Vec the way you did, it starts with a capacity of zero. This means that the underlying slice (that you will use as a buffer) has a length of zero. As a result, since it must be guaranteed that 0 <= n <= buf.len() and since buf.len() is zero, your read() call immediately returns with 0 bytes read.
To "fix" this, you can either assign a default set of elements to your Vec (Vec::new().resize(1024, 0)), or just use an array from the get-go (let mut buffer:[u8; 1024] = [0; 1024])
The processing of writing data in the ByteArrayOutputStream from InputStream obtained by urlConnection.getInputStream() is taking more than 1 minute.
CODE SNIPPET
URL requestUrl= new URL(_sampleurl_);
HttpURLConnection urlConnection=(HttpURLConnection)requestUrl.openConnection();
urlConnection.setConnectTimeout(10000);
urlConnection.setReadTimeout(10000);
urlConnection.setRequestMethod("GET");
urlConnection.connect();
int statusCode=urlConnection.getResponseCode(); //200
long contentLengthByTen = urlConnection.getHeaderFieldLong("Content-Length", _defaultSize_); //7631029
InputStream inputStream = urlConnection.getInputStream();
final byte[] byteArray = new byte[16384];
int length;
ByteArrayOutputStream byteArrOutStrm = new ByteArrayOutputStream();
int k = 0;
while ((length = inputStream.read(byteArray)) != -1)
{
byteArrOutStrm.write(byteArray, 0, length);
k++;
}
Some of the observations are:
The while loop alone executing for more than one minute and it is iterated for around 2650 times.
The HttpURLConnection response code is 200, so the entire content is available in the InputStream.
The Content-Length of the file is 7631029 (bytes).
I have two questions:
Though the byte array size is 16384 and the status code is 200, the inputStream.read method reads only 2800 bytes averagely. Why and which factors decide these bytes?
Proper solution to reduce the processing time?
Though the byte array size is 16384 and the status code is 200, the inputStream.read method reads only 2800 bytes averagely. Why and which factors decide these bytes?
The number of bytes available in the socket receive buffer, which in turn is a function of the speed of the sender, the frequency of reads at the receiver, the network bandwidth, and the path MTU. What you're seen isn't surprising: it indicates that you're keeping up with the sender pretty well.
Proper solution to reduce the processing time?
Have the sender send faster. There is nothing wrong with your receiving code, although I wonder why you're collecting the entire response in a ByteArrayOutputStream before doing anything with it. This just wastes memory and adds latency.
I am tring to send a screenshot of a desktop over winsock.
As such, there are four tasks:
Save bitmap to buffer
Write data across wire using a socket
Read data from wire using a socket
Load a bitmap from a buffer
I have saved the bitmap to a char array using GetDIBits.
Writing the data to the server, I have done but I have questions.
For writing data over from server to the client, do I need to use only 1 recv() call (I am using TCP), or do i need to split it up into multiple parts? Ive read that TCP is stream concept and that I wouldnt have to worry about packets because that is abstracted for me?
How would I go about loading the information from GetDIBits into a bitmap and displaying it on the main window?
I am guessing I have to use SetDIBits, but into which device contexts do i use?
The Server screenshot capturer is here:
HDC handle_ScreenDC = GetDC(NULL);
HDC handle_MemoryDC = CreateCompatibleDC(handle_ScreenDC);
BITMAP bitmap;
int x = GetDeviceCaps(handle_ScreenDC, HORZRES);
int y = GetDeviceCaps(handle_ScreenDC, VERTRES);
HBITMAP handle_Bitmap = CreateCompatibleBitmap(handle_ScreenDC, x, y);
SelectObject(handle_MemoryDC, handle_Bitmap);
BitBlt(handle_MemoryDC, 0, 0, x, y, handle_ScreenDC, 0, 0, SRCCOPY);
GetObject(handle_Bitmap, sizeof(BITMAP), &bitmap);
BITMAPFILEHEADER bmfHeader;
BITMAPINFOHEADER bi;
bi.biSize = sizeof(BITMAPINFOHEADER);
bi.biWidth = bitmap.bmWidth;
bi.biHeight = bitmap.bmHeight;
bi.biPlanes = 1;
bi.biBitCount = 16;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
//std::cout<< bitmap.bmWidth;
DWORD dwBmpSize =((bitmap.bmWidth * bi.biBitCount + 5) / 32) * 4 * bitmap.bmHeight;
//int i = bitmap.bmWidth;
//DWORD dwBmpSize = 99;
HANDLE hDIB = GlobalAlloc(GHND, dwBmpSize);
char* bufptr = (char *)GlobalLock(hDIB);
GetDIBits(handle_ScreenDC, handle_Bitmap, 0, (UINT)bitmap.bmHeight, bufptr, (BITMAPINFO *)&bi, DIB_RGB_COLORS);
send(clientsock, bufptr , GlobalSize((char *)GlobalLock(hDIB)), 0);
/*Do i need to packetize/split it up? Or 1 send() is good for the matching Recv on the client?*/
/*I am assuming i must send bi structure over winsock also correct?*/
And The receiveing client code:
case WM_PAINT:{
//Im a Gdi beginner so I dont have a clue what im doing here as far as blitting the recved bits, this is just some stuff i tried myself before asking for help
PAINTSTRUCT paintstruct;
HDC handle_WindowDC = BeginPaint(hwnd, &paintstruct);
handle_MemoryDC = CreateCompatibleDC(handle_WindowDC);
handle_Bitmap = CreateCompatibleBitmap(handle_WindowDC, 640, 360);
std::cout << SetDIBits(handle_MemoryDC, handle_Bitmap, 0, bi.biHeight, buffer, (BITMAPINFO *)&bi, DIB_RGB_COLORS);
SelectObject(handle_MemoryDC, handle_Bitmap);
StretchBlt(handle_WindowDC, 50, 50, 640, 360, handle_MemoryDC, 0, 0, x, y, SRCCOPY);
EndPaint(hwnd, &paintstruct);
}
Sockets do have limited buffer sizes at both ends, typically around 4000 bytes. So if you dump a large block of data (like a full screendump) in one call to a non-blocking send, you will likely get errors, and you will need to manage your own buffers, calling multiple sends. However, if you are using non-blocking socket, you should be OK, as send() will simply block until all the data is sent.
On the receiving side, it is a similar case - a blocking receive can just keep waiting until it has the full data size that you asked for, but a non-blocking receive will return with whatever data is available at that time, which will result in the data filtering through bit by bit, and you will need to reassemble the data from multiple recv() calls.
I have heard of issues with sending really large blocks of data in one hit, so if you are sending 5 megabytes of data in one hit, be aware there might be other issues coming into play as well.
Using .NET 4.0, IIS 7.5 (Windows Server 2008 R2). I would like to stream out a binary content of about 10 MB. The content is already in a MemoryStream. I wonder if IIS7 automatically chunks the output stream. From the client receiving the stream, is there any difference between these two approaches:
//#1: Output the entire stream in 1 single chunks
Response.OutputStream.Write(memoryStr.ToArray(), 0, (int) memoryStr.Length);
Response.Flush();
//#2: Output by 4K chunks
byte[] buffer = new byte[4096];
int byteReadCount;
while ((byteReadCount = memoryStr.Read(buffer, 0, buffer.Length)) > 0)
{
Response.OutputStream.Write(buffer, 0, byteReadCount);
Response.Flush();
}
Thanks in advance for any help.
I didn't try your 2nd suggestion passing the original data stream. The memory stream was indeed populated from a Response Stream of a Web Request. Here is the code,
HttpWebRequest webreq = (HttpWebRequest) WebRequest.Create(this._targetUri);
using (HttpWebResponse httpResponse = (HttpWebResponse)webreq.GetResponse())
{
using (Stream responseStream = httpResponse.GetResponseStream())
{
byte[] buffer = new byte[4096];
int byteReadCount = 0;
MemoryStream memoryStr = new MemoryStream(4096);
while ((byteReadCount = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
memoryStr.Write(buffer, 0, byteReadCount);
}
// ... etc ... //
}
}
Do you think it can safely pass the responseStream to Response.OutputStream.Write() ? If yes, can you suggest an economic way of doing so? How to send ByteArray + exact stream length to Response.OutputStream.Write()?
The second option is the best one as ToArray will in fact create a copy of the internal array stored in the MemoryStream.
But, you can also preferably use memoryStr.GetBuffer() that will return a reference to this internal array. In this case, you need to use the memoryStr.Length property because the buffer returned by GetBuffer() is in general bigger than the stored actual data (it's allocated chunk by chunk, not byte by byte).
Note that it would be best to pass the original data as a stream directly to the ASP.NET outputstream, instead of using an intermediary MemoryStream. It depends on how you get your binary data in the first place.
Another option, if you serve the exact same content often, is to save this MemoryStream to a physical file (using a FileStream), and use Response.TransmitFile on all subsequent requests. Response.TransmitFile is using low level Windows socket layers and there's nothing faster to send a file.