TCP ACK ignored, retransmission SYN ACK, why? - tcp

We don't understand this TCP behavior showing that a redhat linux 5 TCP stack (HTTP server, this is where this dump is from) received an ACK for a SYN,ACK but continues ignoring that and repeats duplicated SYN, ACK 5 times. Finally the server sends a RST for a HTTP GET on this 'connection'.
Time Source Destination Port Protocol Length Info
2015-01-30 08:42:18.387260000 81.74.146.89 124.219.82.236 80 TCP 74 64866 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 WS=8 SACK_PERM=1 TSval=988669132 TSecr=0
2015-01-30 08:42:18.387309000 124.219.82.236 81.74.146.89 64866 TCP 62 http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:42:18.387354000 81.74.146.89 124.219.82.236 80 TCP 60 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0
2015-01-30 08:42:21.386871000 124.219.82.236 81.74.146.89 64866 TCP 62 [TCP Retransmission] http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:42:21.387118000 81.74.146.89 124.219.82.236 80 TCP 66 [TCP Dup ACK 3#1] 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0 SLE=0 SRE=1
2015-01-30 08:42:27.386919000 124.219.82.236 81.74.146.89 64866 TCP 62 [TCP Retransmission] http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:42:27.387165000 81.74.146.89 124.219.82.236 80 TCP 66 [TCP Dup ACK 3#2] 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0 SLE=0 SRE=1
2015-01-30 08:42:39.387130000 124.219.82.236 81.74.146.89 64866 TCP 62 [TCP Retransmission] http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:42:39.387376000 81.74.146.89 124.219.82.236 80 TCP 66 [TCP Dup ACK 3#3] 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0 SLE=0 SRE=1
2015-01-30 08:43:03.387486000 124.219.82.236 81.74.146.89 64866 TCP 62 [TCP Retransmission] http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:43:03.387709000 81.74.146.89 124.219.82.236 80 TCP 66 [TCP Dup ACK 3#4] 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0 SLE=0 SRE=1
2015-01-30 08:43:51.588227000 124.219.82.236 81.74.146.89 64866 TCP 62 [TCP Retransmission] http > 64866 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1
2015-01-30 08:43:51.588449000 81.74.146.89 124.219.82.236 80 TCP 66 [TCP Dup ACK 3#5] 64866 > http [ACK] Seq=1 Ack=1 Win=5840 Len=0 SLE=0 SRE=1
2015-01-30 08:57:13.679727000 81.74.146.89 124.219.82.236 80 TCP 353 64866 > http [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=299
2015-01-30 08:57:13.679740000 124.219.82.236 81.74.146.89 64866 TCP 54 http > 64866 [RST] Seq=1 Win=0 Len=0
http://i.stack.imgur.com/5F2ZO.png
pcap:
https://www.dropbox.com/s/ave4sctozc5m2a4/TcpAckIgnored.pcapng?dl=0
What is the reason for this TCP Spurious Retransmission? Are there ways to analyze this? Any help is deeply appreciated.
Thomas


After further investigation we came to the conclusion that this is an apache behavior we can reproduce any time with:
"telnet serveraddress 80"
If we start a simple webserver with "nc -l 80" there is no retransmittion.
So the apache needs at least a TCP PSH or any datapacket to open the connection. You can monitor that with netstat.


I have the same issue on a reloading database, so I found this question. But it looks like we are in different reason, so I will post my result here and other people who reach this problem can see here.
In my case, if I have a listening port and you can't accept the new comming connection fast enough, you will get this retransmission problem when the listen queue is full. I have test code below, you can use it to test by yourself. I've tested it on a ubuntu12.04, I am not sure if you will get the same behaviour or not on other linux version.
Test code
BTW, the apache behaviour might be related with TCP_DEFER_ACCEPT, it's referred in this question: real-world-use-of-tcp-defer-accept


It happened to me, and I found that the subnet mask of the target host was misconfigured (24 instead of 16).

Related

Why QWebSocketServer sends TCP [FIN] message on client connection attempt

I use a QWebSocketServer with Qt5.8. Most of the WebSocket clients have no troubles connecting to it, but one of them provokes a singular behaviour.
Below is the WireShark capture I get with this client (client 10.8.0.1, server 10.8.0.8).
10.8.0.1 10.8.0.8 TCP 44 51331 → 8888 [SYN] Seq=0 Win=2920 Len=0 MSS=1355
10.8.0.8 10.8.0.1 TCP 44 8888 → 51331 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460
10.8.0.1 10.8.0.8 TCP 40 51331 → 8888 [ACK] Seq=1 Ack=1 Win=2920 Len=0
10.8.0.1 10.8.0.8 TCP 64 51331 → 8888 [PSH, ACK] Seq=1 Ack=1 Win=2920 Len=24 [TCP segment of a reassembled PDU]
10.8.0.8 10.8.0.1 TCP 40 8888 → 51331 [ACK] Seq=1 Ack=25 Win=29200 Len=0
10.8.0.8 10.8.0.1 TCP 40 8888 → 51331 [FIN, ACK] Seq=1 Ack=25 Win=29200 Len=0
10.8.0.1 10.8.0.8 HTTP 209 GET /example HTTP/1.1
10.8.0.8 10.8.0.1 TCP 40 8888 → 51331 [RST] Seq=1 Win=0 Len=0
10.8.0.1 10.8.0.8 TCP 40 51331 → 8888 [ACK] Seq=194 Ack=2 Win=2919 Len=0
10.8.0.8 10.8.0.1 TCP 40 8888 → 51331 [RST] Seq=2 Win=0 Len=0
10.8.0.1 10.8.0.8 TCP 40 51331 → 8888 [FIN, ACK] Seq=194 Ack=2 Win=2919 Len=0
10.8.0.8 10.8.0.1 TCP 40 8888 → 51331 [RST] Seq=2 Win=0 Len=0
Below is the WireShark capture I get from a working client connecting to the same server (client 192.168.1.48, server 192.168.1.111).
192.168.1.48 192.168.1.111 TCP 74 37556 → 8888 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=7880001 TSecr=0 WS=16
192.168.1.111 192.168.1.48 TCP 74 8888 → 37556 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=3996923817 TSecr=7880001 WS=128
192.168.1.48 192.168.1.111 TCP 66 37556 → 8888 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=7880013 TSecr=3996923817
192.168.1.48 192.168.1.111 HTTP 264 GET /example HTTP/1.1
192.168.1.111 192.168.1.48 TCP 66 8888 → 37556 [ACK] Seq=1 Ack=199 Win=30080 Len=0 TSval=3996923820 TSecr=7880013
192.168.1.111 192.168.1.48 HTTP 540 HTTP/1.1 101 Switching Protocols Continuation
192.168.1.48 192.168.1.111 TCP 66 37556 → 8888 [ACK] Seq=199 Ack=475 Win=30272 Len=0 TSval=7880034 TSecr=3996924025
192.168.1.48 192.168.1.111 WebSocket 136 WebSocket Text [FIN] [MASKED]
As far as I understand, in the first case, my QWebSocketServer implementation sends a [FIN] message for a reason I don't understand, before the client can send its HTTP Upgrade Get Request.
Could that be caused by the fact the first client sends [PSH,ACK] TCP message whereas the second does not ? Who is not compliant with the WebSocket protocol here ?

I just found this Qt bug report that explains my situation. As stated by Timur Pocheptsov, this appears to be a known bug that would require a major QWebSockets refactoring to fix.
Unfortunately, it's by design so: QWebSocketHandshakeRequest tries to parse a client's handshake and sees nothing except this line, it considers such a request invalid and then QWebsocketServerPrivate::handshakeReceived closes the connection. A major re-write/re-design in qtwebsockets is needed to handle this correctly.

curl with client certificate authentication

We like to access a webserver using client certificate authentication instead of basic authentication.
Certificate is a PEM cert and the key file is a separate file.
The curl call looks like this:
curl -v --cert cert.crt --key key.key --pass foobar https://testserver/soap/request
From the debug log:
HTTP/1.1 401 Unauthorized
X-message-code: PWD_WRONG
WWW-Authenticate: Basic realm="Test Platform"
Transfer-Encoding: chunked
Date: Wed, 30 May 2018 10:26:51 GMT
Server: TEST
Set-Cookie: ...
I learned that WWW-Authenticate: Basic is a request from the server to the client to provide a Basic auth. However this may happen for an unauthenticated request, so it does not prove the server does not provide certificate authentication.
Reading https://medium.com/#sevcsik/authentication-using-https-client-certificates-3c9d270e8326
...as double-check I tried with a Firefox Browser by merging both files to a PKCS#12 format and importing that into the browser. It works, so the server supports certificate authentication.
What is wrong about the failing curl call?
Is there any test server supporting certificate auth? https://www.httpbin.org/ does not seem to provide that setup.
The same error occurs when using the PKCS#12 version with curl like this:
curl -v --cert cert.p12 --cert-type p12 --pass foobar
https://testserver/soap/request...
Complete debug log:
* Uses proxy env variable no_proxy == '10.0.0.0/8,127.0.0.1,172.16.0.0/12,192.168.20.0/24,'
* Uses proxy env variable https_proxy == 'http://gateway01:8080'
* Trying 10.190.224.23...
* TCP_NODELAY set
* Connected to gateway01 (10.190.224.23) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to testserver:443
> CONNECT testserver:443 HTTP/1.1
> Host: testserver:443
> User-Agent: curl/7.60.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Baden-Wurttemberg; L=...; O=...; OU=...; CN=...
* start date: Aug 15 07:54:38 2016 GMT
* expire date: Aug 15 07:54:38 2019 GMT
* subjectAltName: host "testserver" matched cert's "*.testserver"
* issuer: C=NL; L=Amsterdam; O=Verizon Enterprise Solutions; OU=Cybertrust; CN=Verizon Public SureServer CA G14-SHA2
* SSL certificate verify ok.
> GET /soap/request HTTP/1.1
> Host: testserver
> User-Agent: curl/7.60.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< X-message-code: PWD_WRONG
< WWW-Authenticate: Basic realm="Test Platform"
< Transfer-Encoding: chunked
< Date: Wed, 06 Jun 2018 05:10:56 GMT
< Server: TEST
< Set-Cookie: ...; path=/; httponly; secure
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
* Connection #0 to host gateway01 left intact
As Firefox is able to authenticate, I analyzed both requests with Wireshark. I find the certificate data by filtering for "tcp contains " and the known CN of my certificate in both network dumps, but Firefox and curl are placing the certs on different packets. Having a SOCKS5 proxy on localhost:8081 I capture loopback interface and filter on that tcp port 8081 which you can see in the dump:
Firefox:
$ tshark -r firefox_request.pcapng
1 0.000000000 127.0.0.1 → 127.0.0.1 TCP 74 51254 → 8081 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1945282596 TSecr=0 WS=128
2 0.000013339 127.0.0.1 → 127.0.0.1 TCP 74 8081 → 51254 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1945282596 TSecr=1945282596 WS=128
3 0.000025766 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=1945282597 TSecr=1945282596
4 0.000052377 127.0.0.1 → 127.0.0.1 TCP 69 51254 → 8081 [PSH, ACK] Seq=1 Ack=1 Win=43776 Len=3 TSval=1945282597 TSecr=1945282596
5 0.000058018 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51254 [ACK] Seq=1 Ack=4 Win=43776 Len=0 TSval=1945282597 TSecr=1945282597
6 0.000110299 127.0.0.1 → 127.0.0.1 TCP 68 8081 → 51254 [PSH, ACK] Seq=1 Ack=4 Win=43776 Len=2 TSval=1945282597 TSecr=1945282597
7 0.000127024 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=4 Ack=3 Win=43776 Len=0 TSval=1945282597 TSecr=1945282597
8 0.000145980 127.0.0.1 → 127.0.0.1 TCP 112 51254 → 8081 [PSH, ACK] Seq=4 Ack=3 Win=43776 Len=46 TSval=1945282597 TSecr=1945282597 [TCP segment of a reassembled PDU]
9 0.049958725 127.0.0.1 → 127.0.0.1 TCP 76 8081 → 51254 [PSH, ACK] Seq=3 Ack=50 Win=43776 Len=10 TSval=1945282646 TSecr=1945282597
10 0.050239394 127.0.0.1 → 127.0.0.1 TCP 583 51254 → 8081 [PSH, ACK] Seq=50 Ack=13 Win=43776 Len=517 TSval=1945282647 TSecr=1945282646 [TCP segment of a reassembled PDU]
11 0.100158230 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51254 [ACK] Seq=13 Ack=567 Win=44800 Len=0 TSval=1945282697 TSecr=1945282647
12 0.107621822 127.0.0.1 → 127.0.0.1 TLSv1.2 2786 Server Hello
13 0.107672270 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=567 Ack=2733 Win=174720 Len=0 TSval=1945282704 TSecr=1945282704
14 0.107832838 127.0.0.1 → 127.0.0.1 TLSv1.2 1426 Certificate, Server Key Exchange
15 0.108417953 127.0.0.1 → 127.0.0.1 TCP 1426 8081 → 51254 [PSH, ACK] Seq=4093 Ack=567 Win=44800 Len=1360 TSval=1945282705 TSecr=1945282704 [TCP segment of a reassembled PDU]
16 0.108423407 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=567 Ack=5453 Win=436608 Len=0 TSval=1945282705 TSecr=1945282704
17 0.109263897 127.0.0.1 → 127.0.0.1 TLSv1.2 2674 Certificate Request, Server Hello Done
18 0.118830308 127.0.0.1 → 127.0.0.1 TCP 3613 51254 → 8081 [PSH, ACK] Seq=567 Ack=8061 Win=464896 Len=3547 TSval=1945282715 TSecr=1945282706 [TCP segment of a reassembled PDU]
19 0.118851470 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51254 [ACK] Seq=8061 Ack=4114 Win=175744 Len=0 TSval=1945282715 TSecr=1945282715
20 0.173415105 127.0.0.1 → 127.0.0.1 TLSv1.2 117 Change Cipher Spec, Client Hello[Malformed Packet]
21 0.173932334 127.0.0.1 → 127.0.0.1 TCP 798 51254 → 8081 [PSH, ACK] Seq=4114 Ack=8112 Win=464896 Len=732 TSval=1945282770 TSecr=1945282770 [TCP segment of a reassembled PDU]
22 0.173943923 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51254 [ACK] Seq=8112 Ack=4846 Win=182912 Len=0 TSval=1945282770 TSecr=1945282770
23 0.250413758 127.0.0.1 → 127.0.0.1 TLSv1.2 844 Application Data
24 0.300115138 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=4846 Ack=8890 Win=464896 Len=0 TSval=1945282897 TSecr=1945282847
25 2.490166002 127.0.0.1 → 127.0.0.1 TCP 798 51254 → 8081 [PSH, ACK] Seq=4846 Ack=8890 Win=464896 Len=732 TSval=1945285087 TSecr=1945282847 [TCP segment of a reassembled PDU]
26 2.490201268 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51254 [ACK] Seq=8890 Ack=5578 Win=184320 Len=0 TSval=1945285087 TSecr=1945285087
27 2.554148656 127.0.0.1 → 127.0.0.1 TLSv1.2 844 Application Data
28 2.554155780 127.0.0.1 → 127.0.0.1 TCP 66 51254 → 8081 [ACK] Seq=5578 Ack=9668 Win=464896 Len=0 TSval=1945285151 TSecr=1945285151
29 12.610141935 127.0.0.1 → 127.0.0.1 TCP 66 [TCP Keep-Alive] 51254 → 8081 [ACK] Seq=5577 Ack=9668 Win=464896 Len=0 TSval=1945295207 TSecr=1945285151
30 12.610162448 127.0.0.1 → 127.0.0.1 TCP 66 [TCP Keep-Alive ACK] 8081 → 51254 [ACK] Seq=9668 Ack=5578 Win=184320 Len=0 TSval=1945295207 TSecr=1945285151
Curl:
$ tshark -r curl_request.pcapng
1 0.000000000 127.0.0.1 → 127.0.0.1 TCP 74 51196 → 8081 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1944382941 TSecr=0 WS=128
2 0.000014517 127.0.0.1 → 127.0.0.1 TCP 74 8081 → 51196 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1944382941 TSecr=1944382941 WS=128
3 0.000026917 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
4 0.000068294 127.0.0.1 → 127.0.0.1 TCP 69 51196 → 8081 [PSH, ACK] Seq=1 Ack=1 Win=43776 Len=3 TSval=1944382941 TSecr=1944382941
5 0.000076382 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=1 Ack=4 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
6 0.000105767 127.0.0.1 → 127.0.0.1 TCP 68 8081 → 51196 [PSH, ACK] Seq=1 Ack=4 Win=43776 Len=2 TSval=1944382941 TSecr=1944382941
7 0.000114276 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=4 Ack=3 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
8 0.012753479 127.0.0.1 → 127.0.0.1 TCP 76 51196 → 8081 [PSH, ACK] Seq=4 Ack=3 Win=43776 Len=10 TSval=1944382954 TSecr=1944382941
9 0.056073156 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=3 Ack=14 Win=43776 Len=0 TSval=1944382997 TSecr=1944382954
10 0.060558849 127.0.0.1 → 127.0.0.1 TCP 76 8081 → 51196 [PSH, ACK] Seq=3 Ack=14 Win=43776 Len=10 TSval=1944383002 TSecr=1944382954
11 0.106077184 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=14 Ack=13 Win=43776 Len=0 TSval=1944383047 TSecr=1944383002
12 0.156827822 127.0.0.1 → 127.0.0.1 TLSv1 583 Client Hello
13 0.156838865 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=13 Ack=531 Win=44800 Len=0 TSval=1944383098 TSecr=1944383098
14 0.204829368 127.0.0.1 → 127.0.0.1 TLSv1.2 2786 Server Hello
15 0.204837221 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=531 Ack=2733 Win=174720 Len=0 TSval=1944383146 TSecr=1944383146
16 0.204942723 127.0.0.1 → 127.0.0.1 TLSv1.2 4146 Certificate, Server Key Exchange
17 0.204950957 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=531 Ack=6813 Win=305664 Len=0 TSval=1944383146 TSecr=1944383146
18 0.204981363 127.0.0.1 → 127.0.0.1 TLSv1.2 1314 Certificate Request, Server Hello Done
19 0.204987399 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=531 Ack=8061 Win=436608 Len=0 TSval=1944383146 TSecr=1944383146
20 0.208331980 127.0.0.1 → 127.0.0.1 TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
21 0.208341083 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=8061 Ack=2531 Win=175744 Len=0 TSval=1944383149 TSecr=1944383149
22 0.255529146 127.0.0.1 → 127.0.0.1 TLSv1.2 117 Change Cipher Spec, Client Hello[Malformed Packet]
23 0.255792019 127.0.0.1 → 127.0.0.1 TLSv1.2 220 Application Data
24 0.255799639 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=8112 Ack=2685 Win=179840 Len=0 TSval=1944383197 TSecr=1944383197
25 2.312629664 127.0.0.1 → 127.0.0.1 TLSv1.2 535 Application Data
26 2.312809015 127.0.0.1 → 127.0.0.1 TLSv1.2 97 Encrypted Alert
27 2.312821190 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=8581 Ack=2716 Win=179840 Len=0 TSval=1944385254 TSecr=1944385254
28 2.313444679 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [FIN, ACK] Seq=2716 Ack=8581 Win=444800 Len=0 TSval=1944385255 TSecr=1944385254
29 2.356079624 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [ACK] Seq=8581 Ack=2717 Win=179840 Len=0 TSval=1944385297 TSecr=1944385255
30 2.357676736 127.0.0.1 → 127.0.0.1 TCP 66 8081 → 51196 [FIN, ACK] Seq=8581 Ack=2717 Win=179840 Len=0 TSval=1944385299 TSecr=1944385255
31 2.357690888 127.0.0.1 → 127.0.0.1 TCP 66 51196 → 8081 [ACK] Seq=2717 Ack=8582 Win=444800 Len=0 TSval=1944385299 TSecr=1944385299
Filtering for the special CN of my local certificate I can find this in both dumps:
$ for file in {firefox,curl}_request.pcapng;do tshark -r "$file" -Y "tcp contains FOOBAR";done
18 0.118830308 127.0.0.1 → 127.0.0.1 TCP 3613 51254 → 8081 [PSH, ACK] Seq=567 Ack=8061 Win=464896 Len=3547 TSval=1945282715 TSecr=1945282706 [TCP segment of a reassembled PDU]
20 0.208331980 127.0.0.1 → 127.0.0.1 TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
However it is detected as part of the TLS handshake only for curl:
$ for file in {firefox,curl}_request.pcapng;do tshark -r "$file" -Y "ssl contains FOOBAR";done
20 0.208331980 127.0.0.1 → 127.0.0.1 TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
What is different in the authentication of Firefox (successful) and Curl (failing)?

The server includes a list of acceptable certificate authorities in its CertificateRequest message. The client should then send a certificate chain that is acceptable according to those criteria.
Based on the fact that your client certificate is included in a "TCP segment of a reassembled PDU" in Firefox, I guess that it additionally included intermediate certificate authorities to complete the certification path. This enlarges the handshake message such that it no longer fits in a single TLS record (or TCP segment).
With cURL, you have probably not included this intermediate certificate which results in a smaller Certificate message from the client to server that fits in a single TLS record. This then results in Wireshark displaying something for ssl contains YOUR_CN. (The reason for this display issue is actually a missing feature in Wireshark, reassembly of handshake messages across different TLS records, bug 3303.)
To fix this, you can try appending the intermediate CAs to your local client certificate.
(I could be mistaken, but this is one possible reason I can think of based on the text trace alone.)

Thanks Lekensteyn, you were right, it was about CA intermediate certs.
Looking inside the Certificate, Client Key Exchange of Firefox, it was adding another intermediate CA from Comodo, I just don't know where it got that from.
However merging the 2 certs into one file didn't help in curl, neither helped adding 2 --cert. But --cacert worked which is interesting as --cacert is actually meant to set the CA cert to check the server certs on client side.

In the traces you can see Firefox sending 2 certs and curl sending only one cert.
Question is, is this a common approach to send the CA certificate as well? Where is this documented as TLS doesn't describe that.
Why behaves Firefox that way and how does it know to only send one intermediate in a chain of 4 certificates in total up to the root cert?

TCP destination port is not showing 80

when I direct to a website the destination port(website server port) must be 80 as I read but on wireshark I could not get tcp port 80.
1 0.000000 23.57.209.221 192.168.0.102 TCP 60 443 → 49856 [ACK] Seq=1 Ack=1 Win=29200 Len=0

A websocket via nginx proxy?

Hi I have a question ~ I want to use nginx proxy to distribute websocket connect to other port. like this .
client ====> proxy =====> ws server
but my question is the websocket tunnel is build on the client and ws server or client and proxy and ws server.
client === proxy ==== ws server
Or
client ===== ws server
Thanks ~~~

You don't skip proxy as such. The proxy makes the connection on behalf of you. See the below excerpt from https://www.nginx.com/blog/websocket-nginx/
A WebSocket application keeps a long‑running connection open between the client and the server, facilitating the development of real‑time applications. The HTTP Upgrade mechanism used to upgrade the connection from HTTP to WebSocket uses the Upgrade and Connection headers. There are some challenges that a reverse proxy server faces in supporting WebSocket. One is that WebSocket is a hop‑by‑hop protocol, so when a proxy server intercepts an Upgrade request from a client it needs to send its own Upgrade request to the backend server, including the appropriate headers. Also, since WebSocket connections are long lived, as opposed to the typical short‑lived connections used by HTTP, the reverse proxy needs to allow these connections to remain open, rather than closing them because they seem to be idle.
Edit-1 - Connections Comparisons between Nginx and direct NodeJS
So finally I did some test to make sure. Started a socket.io app on 3000 and nginx on 80 to forward to request to the Socket.io
Nginx
When I access the app using http://IP/ in two browser windows, the open connection are as below
$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 4443 vagrant 11u IPv6 25730 0t0 TCP *:3000 (LISTEN)
node 4443 vagrant 13u IPv6 28591 0t0 TCP localhost:3000->localhost:42698 (ESTABLISHED)
node 4443 vagrant 14u IPv6 28626 0t0 TCP localhost:3000->localhost:42712 (ESTABLISHED)
nginx 5144 vagrant 6u IPv4 28402 0t0 TCP *:http (LISTEN)
nginx 5144 vagrant 13u IPv4 28589 0t0 TCP 192.168.33.100:http->192.168.33.1:64799 (ESTABLISHED)
nginx 5144 vagrant 14u IPv4 28590 0t0 TCP localhost:42698->localhost:3000 (ESTABLISHED)
nginx 5144 vagrant 15u IPv4 28625 0t0 TCP localhost:42712->localhost:3000 (ESTABLISHED)
nginx 5144 vagrant 16u IPv4 28624 0t0 TCP 192.168.33.100:http->192.168.33.1:64826 (ESTABLISHED)
Now after closing both the tabs
$ lsof
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 4443 vagrant 11u IPv6 25730 0t0 TCP *:3000 (LISTEN)
nginx 5144 vagrant 6u IPv4 28402 0t0 TCP *:http (LISTEN)
NodeJS
Opening two browser tabs with direct connection to NodeJS http://<IP>:3000. Then the results are as below
$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 4443 vagrant 11u IPv6 25730 0t0 TCP *:3000 (LISTEN)
node 4443 vagrant 13u IPv6 30014 0t0 TCP 192.168.33.100:3000->192.168.33.1:52550 (ESTABLISHED)
node 4443 vagrant 14u IPv6 30015 0t0 TCP 192.168.33.100:3000->192.168.33.1:52551 (ESTABLISHED)
node 4443 vagrant 15u IPv6 30016 0t0 TCP 192.168.33.100:3000->192.168.33.1:52552 (ESTABLISHED)
node 4443 vagrant 16u IPv6 30017 0t0 TCP 192.168.33.100:3000->192.168.33.1:52553 (ESTABLISHED)
node 4443 vagrant 17u IPv6 30018 0t0 TCP 192.168.33.100:3000->192.168.33.1:52554 (ESTABLISHED)
node 4443 vagrant 18u IPv6 30020 0t0 TCP 192.168.33.100:3000->192.168.33.1:52556 (ESTABLISHED)
node 4443 vagrant 19u IPv6 30024 0t0 TCP 192.168.33.100:3000->192.168.33.1:52578 (ESTABLISHED)
nginx 5144 vagrant 6u IPv4 28402 0t0 TCP *:http (LISTEN)
This may look a bit strange, but this is because the connections used to open the site are also kept alive for few seconds. So after waiting for sometime
$ lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 4443 vagrant 11u IPv6 25730 0t0 TCP *:3000 (LISTEN)
node 4443 vagrant 18u IPv6 30020 0t0 TCP 192.168.33.100:3000->192.168.33.1:52556 (ESTABLISHED)
node 4443 vagrant 19u IPv6 30024 0t0 TCP 192.168.33.100:3000->192.168.33.1:52578 (ESTABLISHED)
nginx 5144 vagrant 6u IPv4 28402 0t0 TCP *:http (LISTEN)
As you can see no. of connections will be always be higher when you use Nginx

Unexpected TCP Dup ACK

I write a simple TCP server and client myself. After I run them on localhost, I can always capture 'Dup ACK' by Wireshark.
2147 is the Dup ACK. The server port is 12093.
Can anyone tell me why this Dup Ack happens?
690 54.831381000 127.0.0.1 127.0.0.1 TCP 68 62301→12093 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=32 TSval=418815085 TSecr=0 SACK_PERM=1
691 54.831549000 127.0.0.1 127.0.0.1 TCP 68 12093→62301 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=16344 WS=32 TSval=418815085 TSecr=418815085 SACK_PERM=1
692 54.831558000 127.0.0.1 127.0.0.1 TCP 56 62301→12093 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=418815085 TSecr=418815085
693 54.831564000 127.0.0.1 127.0.0.1 TCP 56 [TCP Window Update] 12093→62301 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=418815085 TSecr=418815085
694 54.831652000 127.0.0.1 127.0.0.1 TCP 112 62301→12093 [PSH, ACK] Seq=1 Ack=1 Win=408288 Len=56 TSval=418815085 TSecr=418815085
695 54.831663000 127.0.0.1 127.0.0.1 TCP 56 12093→62301 [ACK] Seq=1 Ack=57 Win=408224 Len=0 TSval=418815085 TSecr=418815085
2140 60.517240000 127.0.0.1 127.0.0.1 TCP 59 12093→62301 [PSH, ACK] Seq=1 Ack=57 Win=408224 Len=3 TSval=418820717 TSecr=418815085
2142 60.517260000 127.0.0.1 127.0.0.1 TCP 56 62301→12093 [ACK] Seq=57 Ack=4 Win=408288 Len=0 TSval=418820717 TSecr=418820717
2145 60.517377000 127.0.0.1 127.0.0.1 TCP 56 62301→12093 [FIN, ACK] Seq=57 Ack=4 Win=408288 Len=0 TSval=418820717 TSecr=418820717
"2146","60.517394000","127.0.0.1","127.0.0.1","TCP","56","12093→62301 [ACK] Seq=4 Ack=58 Win=408224 Len=0 TSval=418820717 TSecr=418820717"
2147 60.517401000 127.0.0.1 127.0.0.1 TCP 56 [TCP Dup ACK 2145#1] 62301→12093 [ACK] Seq=58 Ack=4 Win=408288 Len=0 TSval=418820717 TSecr=418820717
2152 60.517682000 127.0.0.1 127.0.0.1 TCP 56 12093→62301 [FIN, ACK] Seq=4 Ack=58 Win=408224 Len=0 TSval=418820717 TSecr=418820717
2153 60.517704000 127.0.0.1 127.0.0.1 TCP 56 62301→12093 [ACK] Seq=58 Ack=5 Win=408288 Len=0 TSval=418820717 TSecr=418820717

Duplicate ACKs are perfectly normal behaviour as the receiver processes more data and updates its window size. Basically the receiver is saying to its peer: "hey, I know you haven't sent me anything new but when you do I've got more space available now so feel free to send me up to 408288 bytes".

Resources