Truncated DNS PTR query response in Netty 4.1.13.Final - pointers

I'm using Netty to do forward and reverse lookups and it's working well. One thing I've run into though when doing a PTR lookup of an hp.com IP address (15.73.104.147) is that a large result set is truncated.
When using dig -x 15.73.104.147, I can get a result but I noticed that it too has a truncated response and so successfully switches to TCP mode -
;; Truncated, retrying in TCP mode.
If there any way of doing something similar with Netty given that the DnsNameResolverBuilder needs a DatagramChannel instance?
thanks,
Matt

Netty itself not support DNS over TCP atm. What you could try is increase the datagram packet size that is expected via DnsResolverBuilder. maxPayloadSize(....).
https://github.com/netty/netty/blob/4.1/resolver-dns/src/main/java/io/netty/resolver/dns/DnsNameResolverBuilder.java#L249

Related

Netty and TCP - How to properly send an empty message

We have a simple TCP server behind an AWS Network ELB (similar to Echo server with long-lived connections) written in Netty and I'm trying to implement a keep-alive mechanism similar to TCP keep-alive mechanism to keep our idle connections open. Unfortunately we cannot rely on TCP keep-alive mechanism since NELBs do not forward keep-alive TCP packets to the other side of the loadbalancer.
What I'm thinking to do is to watch for idle connections and send an empty string (empty byte array) to clients. What I did so far in the code is:
Add a IdleStateHandler with some timeout values
Register a GprsKeepAliveHandler, a sub class of ChannelDuplexHandler, overriding userEventTriggered method sending (ctx.writeAndFlush) the Unpooled.EMPTY_BUFFER.
This way, I expect to receive a RST packet if the connection is gone. Otherwise the connection will become active again.
The problem is Netty does not do anything with the empty message, it does not send any packets to the client (monitored with Wireshark). If I change the message to Unpooled.wrappedBuffer(new byte[]{0}) I see what I'm expect to see.
Questions
I couldn't find a better way to achieve my objective (keep connections alive and detect dead connections). If there's a better way please let me know.
What is the proper way to send an empty message in Netty? (I saw this question but it didn't help)
If the issue is because of OS TCP stack behavior, is there a way to solve this problem?
from my perspective you need to send something meaningful, because you try to do (e.g. ping/pong, heartbeating behavior). Also see Is it is possible to force TCP socket to send 0 bytes in case of packet losses - python
It seems that Netty does not make any syscall in case of empty messages. (see this)

QUdpSocket problem

I try to send data using UDP protocol. Is it possible to understand when UDP dont send data?
Thanks a lot.
I try to a servis which run into client. And they send their IP an port number in one second. Server listen them and if they dont send this message it understand that client is not connected. I do this but I cant understand when they dont send? Do you have any suggestion
You can check the result of writeDatagram
Sends the datagram at data of size size to the host address address at port port. Returns the number of bytes sent on success; otherwise returns -1.
Then just check the return number to make sure the number of bytes sent was what you expected
Of course it's possible, but it might be hard.
I would recommend:
Verify that you don't get errors from your calls to send data (perhaps you're specifying a bad address, or the socket is in a bad state or something).
Try sending more seldom, perhaps your packets are getting dropped by your local network stack.
Make sure you really listen properly at the receiving end, perhaps the packets make it but you fail to read them properly.
Consider firewall/NAT issues, as usual with UDP. Protocol-wise, never include connection information as application data in packets, since then it's invisible to NAT-machines.
The next step might be digging down and trying to get some feedback from the local network stack, or maybe sniffing the network to see if the packets make it some way at least.

how to know which is the last TCP segment received by the server when data is transferring?

When transferring data in TCP, and given all the incoming and outcoming packets, how will one know if the packet received is the last of the data?
TCP packets are fragmented into smaller parts. I'm transferring over the HTTP protocol.
When the FIN flag is set by one end of the connection, it indicates that that end will not be sending anymore data.
If the connection is not being closed after the last of the data, then there must be an application-layer method of determining it. For HTTP, the rules are reasonably complicated.
You might use PSH flag of TCP protocol. It should be set to 1 in last packet.
To confirm this just start tracing, make HTTP GET and filter session. You will find that last packet for each response on your HTTP GET is marked by this flag.
I'm assuming you're using some sort of socket library. You can tell a TCP connection is finished because a read() on the socket will return 0. This will happen when the other side closes the connection (which it never has to do).

TCP Connection Persistent State

Is there any field/option/anything that I can put in a TCP packet (be it a syn or an ack or just plain data) that I can be sure will be returned by the other end intact?
For eg. I want to "tag" a particular connection (src, srcport, dst, dstport) with a number that I can always read from a packet belonging to that connection. That means I can identify the connection without using the 4-tuple (as given above).
Yes: it is called a Client protocol encapsulated in the TCP server protocol.
In other words: define the Client protocol to meet your needs. Don't try to "shove" extra bits in the TCP overhead.
There are of course the 'options' overhead in TCP but I doubt you'll find an easy way to access these... and in any case, you shouldn't.
You could possibly abuse the TCP Timestamp option for this. It does not seem like a great idea, though.
You can have a lookup table in your application where you associate your tag with the socket.
No, there isn't any facility for what you describe.
Typically what you would do if you're writing a socket application with multiple connections to other systems, is keep track of the socket handle that belongs to each remote system. When receiving data, you are using the socket handle (in some form, don't know which OS or language you're using) so you can take appropriate action based on whichever socket handle that is.
I've never seen a server application that keeps track of connections based on the 4-tuple of address/ports. That seems like way too much work.
On rereading your question, it seems like you may be asking this from the point of view of the TCP driver level. What sort of software are you writing here?
In UDP, destination IP and destination port number are used to demultiplex the packets, but in TCP destination IP, source IP, destination port number and source port numbers (4-tuple) all needed to distinguish between the connections why reasoning for this usage.

Working with persistent HTTP connections

We are trying to implement a proxy proof of concept but have encountered an interesting question: Since a single HTTP connection can, and indeed should, make multiple requests, and the HTTP transactions are sent via multiple packets due to TCP's magic, is it possible for a HTTP request to begin in the middle of a packet?
Bear in mind that this is not a theoretical question regarding possible optimization of the browser, but whether it actually happens in real life. It would be even better if someone could point me to a written reference on whether or not this is possible and if so how often it can occur.
Clarification update: We know that if we work in the HTTP layer alone we would not need to bother with this question, however we're trying to figure out if some advanced technique could be applied by working on the TCP layer first.
Assuming that you are talking about IP packets: Yes, it is possible that HTTP request starts middle of IP packet.
When you are using persistent HTTP connections, that is, using same TCP connection for several HTTP requests, it is fully possible that request boundary is middle of IP packet.
Also there is a TCP protocol between IP and HTTP. TCP contains also some headers so a IP packet may start with some TCP headers and rest of the packet consists of HTTP request.
HTTP request may also consist of several IP packets (in case of file uploads, transmission errors and following retransmissions etc).
However, I wonder why you are interested in packets if you are working at HTTP level. TCP should hide the IP packet details.
First of all, TCP is a stream based protocol and has no concept of packets. HTTP itself might have some kind of message or record delimiter, but TCP doesn't.
This page might be helpful: Structure of HTTP Transactions
From your question it sounds like you think that each read from a TCP socket is a "packet" of data. In reality, each read simply reads as many bytes as are in the buffer up to the maximum that you requested, without any concept of records or packets.
So for instance, lets say you read 2048 bytes from the socket, you could have the tail end of one transaction, followed by the beginning of a second response half way through the data you read, and only get the remainder of your second response on your next read from the socket.
If you're here in Jerusalem or near by maybe I could help you out.
Unless you are implementing your own TCP stack, you should not need to worry about the packets, but rather about the API that the TCP provides, in case of POSIX interfaces it would be the recv() or read(). So I treat the question then as "Can more than one HTTP requests come into a single read(), and can the HTTP request be split between multiple read() requests?" -- The answer to both would be "yes, it is possible".
An example of where this can happen is HTTP pipelining. This not frequent in real life (ironically, at least some of the browsers disable it by default because of "buggy proxies" :-) - but when it happens, can be a bit of a problem for the users to diagnose - especially if they have no access to the proxy.
One very notable place where it does happen by default apt-get in Debian-derived linux systems. Just install a Debian or Ubuntu server and try to use it through your proxy. You can do that by editing the /etc/apt/apt.conf.d/proxy file and placing the following there:
Acquire::http::Proxy "http://your.proxy.address:8080";
Depends of which abstraction layer of a packet you are talking about: there are many layers underneath HTTP.
HTTP --> TCP (byte stream) --> IP (packet) --> (possibly something else) Ethernet (frame) --> (possibly) some other transport
If you are talking about the IP layer, then yes the HTTP layer would start later on... Note that TCP presents a "byte stream interface" to its Client layer hence, no concept of packet here.
I think I understand where you are trying to go with this question.
If you don't use persistent HTTP connections, the HTTP GET request header is always the very first thing which is sent over the TCP connection, so we can be sure that the start of the HTTP GET request header does "not start in the middle of some TCP packet". But keep in mind that there may be one or more TCP packets without any user data, e.g. only a SYN, which may preceed the TCP packet with the start of the HTTP GET request header. And also keep in mind that the HTTP GET request header may not be contained in a single TCP packet.
If you do use persistent HTTP connections, the start of the HTTP GET request header for request number N+1 can start in the middle of a TCP packet, namely after the end of HTTP GET request body of request number N.
If you are asking these questions you are possibly "doing it wrong". As several other responders have already pointed out, in the vast majority of cases you should probably just be a TCP client and deal with a TCP stream of data and let the TCP code worry about the TCP packets. (Unless, of course, you are working on some special hardware which is looking at individual IP packets as they fly by and try to do some processing at the HTTP layer.)

Resources