I'm trying to implement a TCP stack over TUN device according to RFC 793 in Linux. By default, my program is in the LISTEN state and is waiting for an SYN packet to establish a connection. I use nc to send an SYN:
$ nc 192.168.20.99 20
My program responds with SYN, ACK, but nc doesn't send an ACK at the end. This is the flow:
# tshark -i tun0 -z flow,tcp,network
1 0.000000000 192.168.20.1 → 192.168.20.99 TCP 60 39284 → 20 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1691638570 TSecr=0 WS=128
2 0.000112185 192.168.20.99 → 192.168.20.1 TCP 40 20 → 39284 [SYN, ACK] Seq=0 Ack=1 Win=10 Len=0
3 1.001056784 192.168.20.1 → 192.168.20.99 TCP 60 [TCP Retransmission] [TCP Port numbers reused] 39284 → 20 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1691639571 TSecr=0 WS=128
|Time | 192.168.20.1 |
| | | 192.168.20.99 |
|0.000000000| SYN | |Seq = 0
| |(39284) ------------------> (20) |
|0.000112185| SYN, ACK | |Seq = 0 Ack = 1
| |(39284) <------------------ (20) |
|1.001056784| SYN | |Seq = 0
| |(39284) ------------------> (20) |
More info about my TCP header:
Frame 2: 40 bytes on wire (320 bits), 40 bytes captured (320 bits) on interface tun0, id 0
Raw packet data
Internet Protocol Version 4, Src: 192.168.20.99, Dst: 192.168.20.1
Transmission Control Protocol, Src Port: 20, Dst Port: 39310, Seq: 0, Ack: 1, Len: 0
Source Port: 20
Destination Port: 39310
[Stream index: 0]
[Conversation completeness: Incomplete, CLIENT_ESTABLISHED (3)]
[TCP Segment Len: 0]
Sequence Number: 0 (relative sequence number)
Sequence Number (raw): 0
[Next Sequence Number: 1 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 645383655
0101 .... = Header Length: 20 bytes (5)
Flags: 0x012 (SYN, ACK)
Window: 10
[Calculated window size: 10]
Checksum: 0x99b0 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
NOTE: I'm aware of the ISN prediction attack, but this is just a test, and 0 for the sequence number is just as random as any other number in this case.
UPDATE: This is the output of tcpdump which says I'm calculating checksum wrong:
# tcpdump -i tun0 -vv -n
...
IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40, bad cksum 16f3 (->911b)!)
192.168.20.99.20 > 192.168.20.1.39308: Flags [S.], cksum 0x9bb0 (incorrect -> 0x1822), seq 0, ack 274285560, win 10, length 0
...
Here is my checksum calculator (From RFC 1071):
uint16_t checksum(void *addr, int count)
{
uint32_t sum = 0;
uint16_t *ptr = addr;
while (count > 1) {
sum += *ptr++;
count -= 2;
}
if (count > 0)
sum += *(uint8_t *)ptr;
while (sum >> 16)
sum = (sum & 0xffff) + (sum >> 16);
return ~sum;
}
And I'm passing the combination of pseudo-header with the TCP segment for TCP checksum. (in big-endian order):
uint16_t tcp_checksum(struct tcp_header *tcph, uint8_t *pseudo_header)
{
size_t len = PSEUDO_HEADER_SIZE + (tcph->data_offset * 4);
uint8_t combination[len];
memcpy(combination, pseudo_header, PSEUDO_HEADER_SIZE);
dump_tcp_header(tcph, combination, PSEUDO_HEADER_SIZE);
return checksum(combination, len / 2);
}
What am I doing wrong here?
Problem solved by calculating checksums via in_cksum.c from tcpdump source code, which is a line-by-line implementation of the RFC 1071. I also had to set IFF_NO_PI for the tun device. For this case, using a tap device instead of a tun device is probably a better choice to handle EtherType.
I have a redis server on my host(macOS), it's port is 6378, first I execute this command:
sudo tcpdump -vvvn -i lo0 port 6378
Then execute this in another tab
redis-cli -h 127.0.0.1 -p 6378
And here is the results from tcpdump after redis-cli connected to redis-server
21:29:05.866610 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->3cb6)!)
127.0.0.1.64020 > 127.0.0.1.6378: Flags [S], cksum 0xfe34 (incorrect -> 0xf8d2), seq 1870296365, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 3029686726 ecr 0,sackOK,eol], length 0
21:29:05.866682 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->3cb6)!)
127.0.0.1.6378 > 127.0.0.1.64020: Flags [S.], cksum 0xfe34 (incorrect -> 0x4dad), seq 3099403233, ack 1870296366, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 962237723 ecr 3029686726,sackOK,eol], length 0
21:29:05.866693 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->3cc2)!)
127.0.0.1.64020 > 127.0.0.1.6378: Flags [.], cksum 0xfe28 (incorrect -> 0xaeb6), seq 1, ack 1, win 6379, options [nop,nop,TS val 3029686726 ecr 962237723], length 0
21:29:05.866701 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->3cc2)!)
127.0.0.1.6378 > 127.0.0.1.64020: Flags [.], cksum 0xfe28 (incorrect -> 0xaeb6), seq 1, ack 1, win 6379, options [nop,nop,TS val 962237723 ecr 3029686726], length 0
21:29:05.866949 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 69, bad cksum 0 (->3cb1)!)
127.0.0.1.64020 > 127.0.0.1.6378: Flags [P.], cksum 0xfe39 (incorrect -> 0x2629), seq 1:18, ack 1, win 6379, options [nop,nop,TS val 3029686726 ecr 962237723], length 17
21:29:05.866967 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->3cc2)!)
127.0.0.1.6378 > 127.0.0.1.64020: Flags [.], cksum 0xfe28 (incorrect -> 0xaea5), seq 1, ack 18, win 6379, options [nop,nop,TS val 962237723 ecr 3029686726], length 0
21:29:05.907727 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 86, bad cksum 0 (->3ca0)!)
127.0.0.1.6378 > 127.0.0.1.64020: Flags [P.], cksum 0xfe4a (incorrect -> 0xde76), seq 1:35, ack 18, win 6379, options [nop,nop,TS val 962237762 ecr 3029686726], length 34
21:29:05.907757 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->3cc2)!)
127.0.0.1.64020 > 127.0.0.1.6378: Flags [.], cksum 0xfe28 (incorrect -> 0xae35), seq 18, ack 35, win 6379, options [nop,nop,TS val 3029686765 ecr 962237762], length 0
But in wireshark, it has a sequence number
And from here we can know that every packet of TCP must have a 32bit "Sequence Number", so, should I add some args to tcpdump so it can show the seq number of lines that has Flags [.]?
According to the Transmission_Control_Protocol on Wikipedia:
Sequence number (32 bits)
Has a dual role:
If the SYN flag is set (1), then this is the initial sequence number. The sequence number of the actual first data byte and the acknowledged number in the corresponding ACK are then this sequence number plus 1.
If the SYN flag is clear (0), then this is the accumulated sequence number of the first data byte of this segment for the current session.
And according to the Stevens's TCP IP Illustrated, Volume 1:
Every TCP segment (except those exchanged during connection establishment)
includes a valid Sequence Number field, an ACK Number or Acknowledgment field, and a Window Size field (containing the window advertisement).
Now let's construct a scenario and use tcpdump to trace the tcp segments:
Initialize a HTTP (i.e. TCP) request with curl:
$ curl -iIL https://blog.codefarm.me/
HTTP/2 200
server: GitHub.com
.....
Meanwhile, executing tcpdump and write the dump date to a file as below:
tcpdump -n port 443 -r /tmp/https.pcap
Reading the dump data with the following command:
$ tcpdump --number -ntS port 443 -r /tmp/https.pcap
reading from file /tmp/https.pcap, link-type EN10MB (Ethernet), snapshot length 262144
1 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [S], seq 2427498844, win 64240, options [mss 1460,sackOK,TS val 2733586448 ecr 0,nop,wscale 7], length 0
2 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [S.], seq 1574645920, ack 2427498845, win 64240, options [mss 1460], length 0
3 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [.], ack 1574645921, win 64240, length 0
4 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427498845:2427499362, ack 1574645921, win 64240, length 517
5 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499362, win 64240, length 0
6 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [P.], seq 1574645921:1574650508, ack 2427499362, win 64240, length 4587
7 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [.], ack 1574650508, win 61320, length 0
8 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499362:2427499442, ack 1574650508, win 62780, length 80
9 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499442:2427499488, ack 1574650508, win 62780, length 46
10 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499488:2427499537, ack 1574650508, win 62780, length 49
11 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499442, win 64240, length 0
12 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499488, win 64240, length 0
13 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499537, win 64240, length 0
14 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499537:2427499640, ack 1574650508, win 62780, length 103
15 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499640, win 64240, length 0
16 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [P.], seq 1574650508:1574651050, ack 2427499640, win 64240, length 542
17 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [P.], seq 1574651050:1574651109, ack 2427499640, win 64240, length 59
18 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [.], ack 1574651109, win 62780, length 0
19 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499640:2427499671, ack 1574651109, win 62780, length 31
20 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499671, win 64240, length 0
21 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [P.], seq 1574651109:1574651502, ack 2427499671, win 64240, length 393
22 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [P.], seq 2427499671:2427499695, ack 1574651502, win 62780, length 24
23 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499695, win 64240, length 0
24 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [F.], seq 2427499695, ack 1574651502, win 62780, length 0
25 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499696, win 64239, length 0
26 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [FP.], seq 1574651502:1574651526, ack 2427499696, win 64239, length 24
27 IP 192.168.91.128.32868 > 185.199.108.153.443: Flags [R], seq 2427499696, win 0, length 0
The packets at the line 11-13 are all ACK segments without palyload. And tcpdump also doesn't show the seq field which is should be 1574650508 as the last sending segment with payload from server (i.e. packet 6 at line number 6).
Why?
Now let's run tcpdump with the following options:
$ tcpdump --number -ntSxx port 443 -r /tmp/https.pcap
.....
6 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [P.], seq 1574645921:1574650508, ack 2427499362, win 64240, length 4587
.....
11 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499442, win 64240, length 0
0x0000: 000c 298c df3f 0050 56e9 f627 0800 4500
0x0010: 0028 5a0e 0000 8006 9e38 b9c7 6c99 c0a8
0x0020: 5b80 01bb 8064 5ddb 428c 90b0 b3b2 5010
0x0030: faf0 0b70 0000 0000 0000 0000
12 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499488, win 64240, length 0
0x0000: 000c 298c df3f 0050 56e9 f627 0800 4500
0x0010: 0028 5a0f 0000 8006 9e37 b9c7 6c99 c0a8
0x0020: 5b80 01bb 8064 5ddb 428c 90b0 b3e0 5010
0x0030: faf0 0b42 0000 0000 0000 0000
13 IP 185.199.108.153.443 > 192.168.91.128.32868: Flags [.], ack 2427499537, win 64240, length 0
0x0000: 000c 298c df3f 0050 56e9 f627 0800 4500
0x0010: 0028 5a10 0000 8006 9e36 b9c7 6c99 c0a8
0x0020: 5b80 01bb 8064 5ddb 428c 90b0 b411 5010
0x0030: faf0 0b11 0000 0000 0000 0000
.....
Actually, with the xx option, we can analyze the 32bit sequence number (i.e. 5ddb 428c) of the above TCP segments (11-13) and convert it to decimal number as below:
$ echo $((16#5ddb428c))
1574650508
Here, we can see the sequence number that repeated in ACK segment three times is 1574650508 which is same as the WireShark (with absolute sequence number option).
I use tcpdump to catch some tcp packets, when analyzing them according to ip/tcp packet schema, the packet seems to be broken. Here is a sample packet I got from tcpdump output. Is any one familiar with them?
Should not the first 4 bit of ip packet always be 0100 in under ipv4?
ip packet: https://en.wikipedia.org/wiki/IPv4
some examples: http://mike.passwall.com/networking/samplepacket.html
13:11:43.330397 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.16.0.14.16668 > 36.24.146.114.64853: Flags [S.], cksum 0xdc0f (correct), seq 3029391223, ack 129060479, win 14480, options [mss 1460,sackOK,TS val 1254469916 ecr 1492278057,nop,wscale 6], length 0
0x0000: feee 809f 3247 5254 0054 aa9f 0800 4500 ....2GRT.T....E.
0x0010: 003c 0000 4000 4006 d813 ac10 000e 2418 .<..#.#.......$.
0x0020: 9272 411c fd55 b490 d777 07b1 4e7f a012 .rA..U...w..N...
0x0030: 3890 dc0f 0000 0204 05b4 0402 080a 4ac5 8.............J.
uname -a
# Linux VM_0_14_centos 2.6.32-754.2.1.el6.x86_64 #1 SMP Fri Jul 13 12:50:12 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# tcpdump
tcpdump tcp -vv -XX -n -i eth0 port 16668
0x0000: feee 809f 3247 5254 0054 aa9f 0800 4500 ....2GRT.T....E.
The first 14 bytes are from the link layer (EN10MB). The IP layer only starts with 4500, where the 4 (binary 0100) are the first 4 bits which describe the version number, i.e. IP version 4.
These link layer data are explicitly requested by the -XX option which is used by the OP as pointed out in a comment by David Hoelzer. To cite from the documentation of tcpdump:
-XX When parsing and printing, in addition to printing the headers of each packet, print the data of each packet, including its link level header, in hex and ASCII.
I am using TCPDUMP to intercept the network traffic on an Android device.
The command I use is
./tcpdump -tt -nn -vv tcp > tcp.log
In the result file (tcp.log), I found that there are not only TCP packets, but also some others with Ethernet address. (Some IP and Ethernet addresses are obfuscated due to privacy purpose).
1410451676.980763 IP (tos 0x0, ttl 41, id 0, offset 0, flags [DF], proto TCP (6), length 60)
54.204.ABC.144.80 > 172.31.DEF.178.59949: Flags [S.], seq 572870324, ack 3778403975, win 17898, options [mss 1380,sackOK,TS[|tcp]>
1410451676.980916 IP (tos 0x0, ttl 64, id 44656, offset 0, flags [DF], proto TCP (6), length 52)
172.31.DEF.178.59949 > 54.204.ABC.144.80: Flags [.], seq 1, ack 1, win 1369, options [nop,nop,TS[|tcp]>
1410451676.982167 IP (tos 0x0, ttl 64, id 44657, offset 0, flags [DF], proto TCP (6), length 701)
172.31.DEF.178.59949 > 54.204.ABC.144.80: Flags [P.], seq 1:650, ack 1, win 1369, options [nop,nop,TS[|tcp]>
1410451676.996114 00:24:f9:c3:XX:00 > d8:50:e6:2b:YY:6a, ethertype IPv6 (0x86dd), length 86:
0x0000: 6000 0000 0020 0634 2607 f8b0 400d 0c03 `......4&...#...
0x0010: 0000 0000 0000 00bc 2001 0468 0c80 4340 ...........h..C#
0x0020: b040 b100 7831 4228 146c c1cc ceb8 fc7a .#..x1B(.l.....z
1410451677.000783 00:24:f9:c3:XX:00 > d8:50:e6:2b:YY:6a, ethertype IPv6 (0x86dd), length 535:
0x0000: 6000 0000 01e1 0634 2607 f8b0 400d 0c03 `......4&...#...
0x0010: 0000 0000 0000 00bc 2001 0468 0c80 4340 ...........h..C#
0x0020: b040 b100 7831 4228 146c c1cc ceb8 fc7a .#..x1B(.l.....z
1410451677.000935 d8:50:e6:2b:YY:6a > 00:24:f9:c3:XX:00, ethertype IPv6 (0x86dd), length 86:
0x0000: 6000 0000 0020 0640 2001 0468 0c80 4340 `......#...h..C#
0x0010: b040 b100 7831 4228 2607 f8b0 400d 0c03 .#..x1B(&...#...
0x0020: 0000 0000 0000 00bc c1cc 146c 3b74 2fa9 ...........l;t/.
1410451677.011006 IP (tos 0x0, ttl 41, id 62137, offset 0, flags [DF], proto TCP (6), length 52)
54.204.ABC.144.80 > 172.31.DEF.178.59949: Flags [.], seq 1, ack 650, win 75, options [nop,nop,TS[|tcp]>
In a regular TCP packet header, the result includes its timestamp (UNIX time), IP packet flag+options, source IP address+port, destination IP address+port, and TCP flags.
BUT, what I do not understand is that there are some Ethernet packets in between and according to the results, these packets include their MAC address, instead of IP address.
Why is the case? Are they TCP packets?
Thanks for your answers/insights.
All those packets are Ethernet packets.
The version of tcpdump you're using was apparently not built with IPv6 support, so, while it can recognize IPv4-over-Ethernet packets and printout the IP information, it can't recognize IPv6-over-Ethernet packets, and just prints out the Ethernet-layer information.
You should ask whoever built that version of tcpdump why it doesn't include IPv6 support.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
This question exists because it has
historical significance, but it is not
considered a good, on-topic question
for this site, so please do not use it
as evidence that you can ask similar
questions here.
More info: https://stackoverflow.com/faq
I am having a weird TCP issue that makes no sense at all. I am tearing my hair out.
When downloading a file from Amazon S3 (and only Amazon S3- every other site works fine) some percentage of the time the connections dies. This only happens from our servers in Seattle with a web server behind a NAT router. Connecting directly from the router works fine. From our servers here in Victoria everything works fine, and we can't replicate this at all.
Web server in Seattle (tried 10 different servers) -> SNAT router (2 different routers; old kernels and new kernels) -> Amazon S3 = doesn't work ~10% of the time
Web server in Seattle -> SNAT router -> any other website = works
Router box -> Amazon S3 = works
Web server in Victoria -> SNAT router in Victoria -> Amazon S3 = works
Things I've tried:
Disabling window scaling
Lowering the window size
upgrading the router to the newest kernel
Disabling the firewall entirely
In the failure case Amazon (or something along the way) sends us a 868 length packet instead of the expected 1500 byte packet. The server upon seeing the packets replies with a RST packet (error has occurred) and the connection is torn down.
Any help would be greatly appreciated!
Below are two TCP dumps-
------------ Begin Error Case TCP Dump -------------
tcpdump -r /tmp/s3doesntwork-died
reading from file /tmp/s3doesntwork-died, link-type EN10MB (Ethernet)
19:58:42.542189 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: S 193799772:193799772(0) win 5840 <mss 1460,sackOK,timestamp 760821159 0,nop,wscale 5>
19:58:42.544115 IP 189-81.amazon.com.www > locum.sparklit.com.39491: S 3148664267:3148664267(0) ack 193799773 win 8190 <mss 1460,nop,wscale 6,nop,nop,sackOK>
19:58:42.545176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1 win 183
19:58:42.545184 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: P 1:212(211) ack 1 win 183
19:58:42.548113 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . ack 212 win 916
19:58:42.558108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1:1461(1460) ack 212 win 916
19:58:42.558117 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1461:2921(1460) ack 212 win 916
19:58:42.558123 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 2921:4381(1460) ack 212 win 916
19:58:42.558128 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 4381:5841(1460) ack 212 win 916
19:58:42.559108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 5841:7301(1460) ack 212 win 916
19:58:42.559118 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 7301:8129(828) ack 212 win 916
19:58:42.559138 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: R 193799984:193799984(0) win 0
19:58:42.559169 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1461 win 274
19:58:42.559176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 2921 win 365
19:58:42.559180 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 4381 win 457
19:58:42.559188 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 5841 win 548
19:58:42.560167 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 7301 win 639
19:58:45.308618 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: F 212:212(0) ack 7301 win 639
19:58:45.310512 IP 189-81.amazon.com.www > locum.sparklit.com.39491: R 3148671568:3148671568(0) win 8201
Update: Traceroute to Amazon
1 64.34.33.195 (64.34.33.195) 0.795 ms 0.775 ms 0.762 ms
2 six01-sea4.amazon.com (206.81.80.147) 0.746 ms 0.732 ms 0.715 ms
3 72.21.222.183 (72.21.222.183) 2.657 ms 2.651 ms 2.638 ms
4 72.21.222.179 (72.21.222.179) 2.618 ms 2.604 ms 2.591 ms
5 * * *
6 * * *
7 * * *
Update: The offending incoming packet that causes the RST to be sent
19:58:42.559118 IP (tos 0x0, ttl 57, id 18905, offset 0, flags [DF], proto TCP (6), length 868) 189-81.amazon.com.www > locum.sparklit.com.39491: ., cksum 0x9f6e (correct), 7301:8129(828) ack 212 win 916
0x0000: 4500 0364 49d9 4000 3906 05d7 cfab bd51 E..dI.#.9......Q
0x0010: 4022 21c5 0050 9a43 bbac ea50 0b8d 2730 #"!..P.C...P..'0
0x0020: 5010 0394 9f6e 0000 7c75 a901 cd57 0718 P....n..|u...W..
0x0030: a786 e954 4160 3734 f563 5029 e7ad 48a7 ...TA`74.cP)..H.
0x0040: 34c0 b11b 75a2 a341 c1e6 8aab a03c 31ee 4...u..A.....<1.
0x0050: 1496 c9ef df22 aadc b87e 8431 fc2a dcd6 ....."...~.1.*..
0x0060: e72d 8cf7 aa92 5b12 4923 3f51 50cd 5195 .-....[.I#?QP.Q.
0x0070: 7910 6ce4 0fc0 d63c f115 276b b7e7 5bf7 y.l....<..'k..[.
0x0080: 508e d8fa d655 c5b3 1638 3cd6 6cd1 198c P....U...8<.l...
0x0090: 1c7f 1b7e 59a4 4370 9c87 523d 0ae2 adb4 ...~Y.Cp..R=....
0x00a0: 4d8e 7ad5 7954 c6ac 79e3 9e05 4148 6c97 M.z.yT..y...AHl.
0x00b0: b711 f262 47fa 363f 9d52 bcd4 a58e 177f ...bG.6?.R......
0x00c0: b33b 1033 b530 7351 d8eb 29a9 dbcf 2c6b .;.3.0sQ..)...,k
0x00d0: b161 99e0 3e67 192b 8c9f a735 89ad f886 .a..>g.+...5....
0x00e0: 4d3a ff00 462e b0ad 8dec 8f04 9bdb 9121 M:..F..........!
0x00f0: 4263 0ac4 81c9 18e7 ae6a 1a65 d8db d3ee Bc.......j.e....
0x0100: 3722 b608 cf24 1182 2ba1 b39b 728a f29b 7"...$..+...r...
0x0110: df1e db68 9af5 b69a e51e 5923 2ed0 29fd ...h......Y#..).
0x0120: f0e4 6303 f8bb f4ae ef44 d5e3 bc67 8f70 ..c......D...g.p
0x0130: 5b88 8299 a30d 931e e1c0 38fc 6bca a917 [.........8.k...
0x0140: 7b9d 8a2d 2bb3 5ef7 9e0d 625e c59c d6dc {..-+.^...b^....
0x0150: e778 cd67 5c26 ecd2 a64b 6737 345b 643d .x.g\&...Kg74[d=
0x0160: 6a3d b57e f22f 98d5 455e b5da b546 0c60 j=.~./..E^...F.`
0x0170: 5c51 5281 8a69 4e69 888c af15 0b0a b240 \QR..iNi.......#
0x0180: c540 c334 0101 1935 04b5 6996 a091 7ad0 .#.4...5..i...z.
0x0190: 2676 ba0d 8bbc 30cc 3385 c7f2 a835 ab19 &v....0.3....5..
0x01a0: 2e0b 003a d75b e17b 453a 03b6 070a a7f4 ...:.[.{E:......
0x01b0: a6cf 671e 7270 6b59 ae5e 5f43 28eb 73cd ..g.rpkY.^_C(.s.
0x01c0: 53c2 8f34 9c8e 3e95 d268 fe0f 4864 4661 S..4..>..h..HdFa
0x01d0: 9008 2702 b75e 782d 81e0 0fc2 b3ae bc51 ..'..^x-.......Q
0x01e0: 0c00 e080 477a 9bb6 3b1f 08fc 7efd 943e ....Gz..;...~..>
0x01f0: 26eb ff00 163c 4bad 695e 1b97 51d2 2f6e &....<K.i^..Q./n
0x0200: 4cb6 f35b dca3 7c98 1d46 ec8f a62b c675 L..[..|..F...+.u
0x0210: 4fd9 dfc7 1a4b b2df e812 da63 fe7e 0b2f O....K.....c.~./
0x0220: eb8a fd38 bff1 d055 386f c735 cf5f f8fa ...8...U8o.5._..
0x0230: 7705 44ac 54f6 2723 f2ae d862 2a25 6b1c w.D.T.'#...b*%k.
0x0240: f2a5 06ee 7e6b a7c1 bd79 f2a2 2b1d c3b1 ....~k...y..+...
0x0250: ba5c d759 e03f 849a e787 3564 d6af 2d6d .\.Y.?....5d..-m
0x0260: db4e b7fd d48f 0ca1 8866 1c0e dcd7 db37 .N.......f.....7
0x0270: b2d9 eb2e 4dd6 9767 704f 5692 d949 fcf1 ....M..gpOV..I..
0x0280: 53da 7c3c d0fc 45a3 ea3a 4cf6 3f67 b379 S.|<..E..:L.?g.y
0x0290: a197 cbb5 7311 0dcf 20d6 8f11 2d99 2a94 ....s.......-.*.
0x02a0: 7a1e 69e1 9fda 77c5 7e0d d3ad 74f8 eeac z.i...w.~...t...
0x02b0: e6b3 b641 1c71 df5b 6085 1db7 a907 f3cd ...A.q.[`.......
0x02c0: 7a3e 81fb 6b4c 42ff 0069 6831 4ebd dec6 z>..kLB..ih1N...
0x02d0: e837 fe3a c335 3d9f ec75 e01d 576b 5c47 .7.:.5=..u..Wk\G
0x02e0: ab12 c3a7 f683 8fe9 5a29 fb07 7c31 3f32 ........Z)..|1?2
0x02f0: dbea 81fd 4ea7 2e7f 98ae 37ec 99aa 53e8 ....N.....7...S.
0x0300: cb5a 97ed cfe0 ad1f 4d6b 99f4 bd56 79c1 .Z......Mk...Vy.
0x0310: ff00 8f48 d151 c8ef 8627 15f4 1681 acc5 ...H.Q...'......
0x0320: e21d 0b4e d56d e378 adef add2 e238 e5c6 ...N.m.x.....8..
0x0330: f556 1900 e38c d781 d97e c3bf 0ce0 71f6 .V.......~....q.
0x0340: 9b1d 42ea 31fc 0fa9 cf8f fd0a be80 d2b4 ..B.1...........
0x0350: f834 8d3a d2c2 d13c ab4b 5896 1850 9270 .4.:...<.KX..P.p
0x0360: 8a30 073e .0.>
So we finally figured it out-
It turns out that our router was injecting the RST packets because Netfilter declared the packet invalid. There is a setting that makes netfitler more liberal so I tried enabling it (inet.ipv4.netfilter.ip_conntrack_tcp_be_liberal=1) and the connection is no longer torn down.
However, I still haven't figured out why netfilter declared that packet as invalid. Enabling "ip_conntrack_log_invalid" doesn't cause anything to be printed in the log.
This is a guess. Maybe running some traceroutes to the target target and collecting the hops. Then pinging each of the hops with a 1500 byte packet configured for a 100 count would help identify problems links.
The sender is entitled to send any length packet it likes between 1 and 1460 (in this case, being your apparent MTU). The question is why does your server send the RST?