I am trying to access a remote webservice from an asp.net application in an iis 6 server to a remote webserver.
As I wasn't able to connect, I used MS Network Monitor to look over what was happening.
This is the TLS progress
2 15:26:31 14.03.2013 0.0000000 yyy.yyy.yyy.yyy XXX.XXX.XXX.XXX TLS TLS:TLS Rec Layer-1 HandShake: Client Hello. {TLS:4, SSLVersionSelector:3, TCP:2, IPv4:1}
3 15:26:31 14.03.2013 0.0312500 XXX.XXX.XXX.XXX yyy.yyy.yyy.yyy TLS TLS:TLS Rec Layer-1 HandShake: Server Hello.; TLS Rec Layer-2 HandShake: Certificate. {TLS:4, SSLVersionSelector:3, TCP:2, IPv4:1}
4 15:26:31 14.03.2013 0.0312500 yyy.yyy.yyy.yyy XXX.XXX.XXX.XXX TLS TLS:TLS Rec Layer-1 HandShake: Certificate. Client Key Exchange. Certificate Verify.; TLS Rec Layer-2 Cipher Change Spec; TLS Rec Layer-3 HandShake: Encrypted Handshake Message. {TLS:4, SSLVersionSelector:3, TCP:2, IPv4:1}
5 15:26:31 14.03.2013 0.0468750 XXX.XXX.XXX.XXX yyy.yyy.yyy.yyy TLS TLS:TLS Rec Layer-1 Cipher Change Spec; TLS Rec Layer-2 HandShake: Encrypted Handshake Message. {TLS:4, SSLVersionSelector:3, TCP:2, IPv4:1}
6 15:26:31 14.03.2013 0.0781250 XXX.XXX.XXX.XXX yyy.yyy.yyy.yyy TLS TLS:TLS Rec Layer-1 Encrypted Alert {TLS:4, SSLVersionSelector:3, TCP:2, IPv4:1}
And this is final TLS transaction with the alert
- TlsRecordLayer: TLS Rec Layer-1 Encrypted Alert
ContentType: Encrypted Alert
- Version: TLS 1.0
Major: 3 (0x3)
Minor: 1 (0x1)
Length: 18 (0x12)
EncryptedData: Binary Large Object (18 Bytes)
And finaly this is the hex dump for the alert part.
15 03 01 00 12 CE CA A3 1C C9 44 5C AC A2 3C BB 3C F3 F9 8A A6 4F 22
According to http://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol I tried to decode the meaning
15 = 21 dec <- always 21
03 01 = major and minor Version
00 12 = lentgh
And now it gets weird. Normaly the next two bytes should be the level and description of the alert. But they make no sense. Beside that, they differ every trace I run. While the front part stays constantly the same.
Any ideas how i can decrypt the alert?
My first thought was that the other side may not allow TLS 1, while IIS 6 only supports TLS 1 (atleast that is what i thougt).
Thanks alot in advanced
The alert itself is encrypted, because the handshake was successful. So before you know what it says, you have to decrypt it - you could use ssldump, in case you have the server's private key.
Found this here: http://www.wireshark.org/lists/ethereal-users/200110/msg00055.html - after I puzzled about the same problem for a while.
This is about ssldump: http://www.rtfm.com/ssldump/
Using Wireshark, you can decrypt everything, in case you have the private key that was used for encryption: http://support.citrix.com/article/ctx116557
Related
I use Delphi 2010, newest Indy 10.6.2 (5498) and OpenSSL 1.0.2r (from indy.fulgan.com) and I am not able to force it to use TLS 1.2. When I open a page mentioned later, it returns EIdOSSLUnderlyingCryptoError:
Error connecting with SSL. error:1408F10B:SSL
routines:SSL3_GET_RECORD:wrong version number
Here is the code I use to open web
var
IdHTTP: TIdHTTP;
begin
IdHTTP:=TIdHTTP.Create;
try
IdHTTP.IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);
TIdSSLIOHandlerSocketOpenSSL(IdHTTP.IOHandler).SSLOptions.Method := sslvTLSv1_2;
TIdSSLIOHandlerSocketOpenSSL(IdHTTP.IOHandler).SSLOptions.SSLVersions := [sslvTLSv1_2];
IdHTTP.Get('https://...');
finally
IdHTTP.Free;
end;
end;
When I open https:// www.google.com, then it correctly uses TLS 1.2, here are details from Wireshark
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 512
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 508
Version: TLS 1.2 (0x0303)
But when I open this page https:// www.downloadtb.com, Indy uses TLS 1.0. The site uses TLS 1.2 only and this cipher: ECDHE-RSA-AES256-GCM-SHA384.
Secure Sockets Layer
TLSv1 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 329
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 325
Version: TLS 1.2 (0x0303)
It seems that it mixes TLS 1.0 and 1.2. Is this some kind of fallback? How do I fix it?
EDIT: Something changed on the site and Indy now correctly uses TLS 1.2
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 512
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 508
Version: TLS 1.2 (0x0303)
I captured and decrypted an HTTPS request using fiddler. It looks about like this:
CONNECT qwer1234.com:443 HTTP/1.0
User-Agent: Blabla Example/1.0
Host: qwer1234.com:443
Content-Length: 0
Connection: Keep-Alive
Pragma: no-cache
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.
Version: 3.3 (TLS/1.2)
Random: 13 35 55 0B F2 3C 9D A3 A2 C2 E4 3E 66 FA D3 DD E5 15 27 33 F1 E6 CD A5 E0 C4 10 4E 41 67 32 8E
SessionID: empty
Extensions:
renegotiation_info 00
server_name qwer1234.com
status_request 01 00 00 00 00
elliptic_curves secp256r1 [0x17], secp384r1 [0x18]
ec_point_formats uncompressed [0x0]
signature_algorithms 00 01 06 07 05 08 03 09 04 03 06 04 04 07 01 09
SessionTicket empty
Ciphers:
[003C] TLS_RSA_WITH_AES_128_CBC_SHA256
[002F] TLS_RSA_AES_128_SHA
[003D] TLS_RSA_WITH_AES_256_CBC_SHA256
[0035] TLS_RSA_AES_256_SHA
[0005] SSL_RSA_WITH_RC4_128_SHA
[000A] SSL_RSA_WITH_3DES_EDE_SHA
[C027] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[C013] TLS1_CK_ECDHE_RSA_WITH_AES_128_CBC_SHA
[C014] TLS1_CK_ECDHE_RSA_WITH_AES_256_CBC_SHA
[C02B] TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[C023] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[C02C] TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[C024] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[C009] TLS1_CK_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
[C00A] TLS1_CK_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
[0040] TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
[0032] TLS_DHE_DSS_WITH_AES_128_SHA
[006A] TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
[0038] TLS_DHE_DSS_WITH_AES_256_SHA
[0013] SSL_DHE_DSS_WITH_3DES_EDE_SHA
[0004] SSL_RSA_WITH_RC4_128_MD5
Compression:
[00] NO_COMPRESSION
Is this a special kind of HTTPS request? Why is only half of the request excrypted? What do the random generated numbers mean / what are they for? How would I send this request using a WinHttpRequest? How does it differ from other HTTPS requests?
I'm kind of bad in finding the right places to ask questions like this. Tell me if you think that I should have asked this on a different page and if so, where.
This isn't a HTTPS request at all, it is a CONNECT tunnel through which HTTPS traffic subsequently flows. Client applications first send a CONNECT request to the proxy (like Fiddler) to tell them where to establish a TCP/IP connection to, then after the connection establishes, a HTTPS handshake is performed and the requests are sent.
When showing you the CONNECT tunnel, Fiddler analyzes the parameters and displays them for you to read. The actual HTTPS requests will appear after this tunnel in the Web Sessions list, if and only if you have enabled HTTPS decryption in the Tools > Fiddler Options > HTTPS screen.
I am work on a high load tcp application with Java Netty, which expect to arrive 300k concurrent TCP connections.
It works perfect on test server, arrive 300k connections, but when deploy to production server, it only can support 65387 connections, after arrive this number, client will throw out a "java.io.IOException: Connection reset by peer" exceptions. I try many times, every time, when connections up to 65387, client will can't create connection.
The network capture as bellow, 10.95.196.27 is server, 10.95.196.29 is client :
16822 12:26:12.480238 10.95.196.29 10.95.196.27 TCP 74 can-ferret > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=872641174 TSecr=0 WS=128
16823 12:26:12.480267 10.95.196.27 10.95.196.29 TCP 66 http > can-ferret [SYN, ACK] Seq=0 Ack=1 Win=2920 Len=0 MSS=1460 SACK_PERM=1 WS=1024
16824 12:26:12.480414 10.95.196.29 10.95.196.27 TCP 60 can-ferret > http [ACK] Seq=1 Ack=1 Win=14720 Len=0
16825 12:26:12.480612 10.95.196.27 10.95.196.29 TCP 54 http > can-ferret [FIN, ACK] Seq=1 Ack=1 Win=3072 Len=0
16826 12:26:12.480675 10.95.196.29 10.95.196.27 HTTP 94 Continuation or non-HTTP traffic
16827 12:26:12.480697 10.95.196.27 10.95.196.29 TCP 54 http > can-ferret [RST] Seq=1 Win=0 Len=0
The exception cause by after client 3 handshake to server, server send a RST package to client, and the new connection was broken.
client side exception stack as bellow:
16:42:05.826 [nioEventLoopGroup-1-15] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the end of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_25]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_25]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) ~[na:1.7.0_25]
at sun.nio.ch.IOUtil.read(IOUtil.java:193) ~[na:1.7.0_25]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) ~[na:1.7.0_25]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:259) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:885) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:226) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:72) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:460) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:424) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:360) ~[netty-all-4.0.0.Beta3.jar:na]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:103) ~[netty-all-4.0.0.Beta3.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Sever side have not exceptions.
I had try turning some sysctl item as bellow to support huge connections, but its useless:
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 4096 33554432
net.ipv4.tcp_wmem = 4096 4096 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 4096
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_syn_backlog = 4096
net.netfilter.nf_conntrack_max = 3000000
net.nf_conntrack_max = 3000000
net.core.somaxconn = 327680
The max open fd already set to 999999
linux-152k:~ # ulimit -n
999999
The OS release is SUSE Linux Enterprise Server 11 SP2 with 3.0.13 kernel:
linux-152k:~ # cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2
linux-152k:~ # uname -a
Linux linux-152k 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 x86_64 x86_64 GNU/Linux.
The dmesg have not any error information, CPU and Memory keep low level, every thing looks good, just server reset connection from client.
We have a test server which was SUSE Linux Enterprise Server 11 SP1 with 2.6.32 kernel, it works well, can support up to 300k connections.
I think maybe some kernel or security limit cause this, but I can't find it, any suggestions or any way to get some debug informations of why server send RST? Thanks.
Santal, I've just came across the following link, and it seems it can give an answer to your question:
What is the theoretical maximum number of open TCP connections that a modern Linux box can have
Finally got the root cause. Simply said, it was a JDK bug, please refer to http://mail.openjdk.java.net/pipermail/nio-dev/2013-September/002284.html
which cause NPE when fd > 64 * 1024.
After upgrade to JDK7_45, everything works great now.
I'm writing some code to integrate an in-house app into a DVR to retrieve a video file. This is all reverse engineered as there isn't any official documentation, and I'm having trouble understanding the following sequence of events (captured by playing with the DVR's Android app).
936 72.985204 192.168.0.1 192.168.0.200 HTTP 468 POST /cgi-bin/supervisor/NetworkBk.cgi HTTP/1.1 (application/x-www-form-urlencoded)
937 72.985368 192.168.0.200 192.168.0.1 TCP 54 mit-ml-dev > 41859 [ACK] Seq=1 Ack=415 Win=65535 Len=0
938 73.933676 192.168.0.200 192.168.0.1 HTTP 275 HTTP/1.0 200 OK (video/mpeg4)
939 73.933983 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=222 Win=15544 Len=0
940 74.004433 192.168.0.200 192.168.0.1 TCP 74 [TCP segment of a reassembled PDU]
941 74.004887 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=242 Win=15544 Len=0
942 74.024669 192.168.0.200 192.168.0.1 HTTP 1346 Continuation or non-HTTP traffic
The HTTP POST requests the video file, which then results in an HTTP OK. I get confused as to what happens next. Isn't the request complete when the HTTP 200 is received? Why then is it continuing to receive TCP data and then getting a HTTP Continuation or non-HTTP traffic? The subsequent TCP packets contain the video file I'm intending to download. When I manually craft a HTTP POST I get the HTTP OK response and then I'm stumped.
This is the code I use to simulate the HTTP POST.
import requests
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post("http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi", data=dc, auth=(username, password))
This code gets the HTTP 200 OK response, how do I get to the Continuation or non-HTTP traffic? I'm new to this and so am unsure if I've provided enough details. I can provide the HTTP headers if that will help.
Addendum
This is the RAW response of the HTTP OK reply. As far as I can tell, there is nothing there about expecting extra content.
HTTP/1.0 200 OK
Date: Mon, 01 Jul 2013 15:01:34 GMT
nServer: Linux/2.x UPnP/1.0 Avtech/1.0
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Connection: close
Content-Type: video/mpeg4
Content-Length: 5
0
OK
Why then is it continuing to receive TCP data and then getting a HTTP
Continuation or non-HTTP traffic?
By default, when you make a request, the body of the response is downloaded immediately.
So in this case once the successful POST request is made the DVR will immediately start sending the video data over TCP - most probably as a H.264 Byte Stream. That would account for the non-HTTP traffic you are seeing.
This code gets the HTTP 200 OK response, how do I get to the
Continuation or non-HTTP traffic?
You can override this default behaviour and defer downloading the response body until you access the Response.content attribute with the stream parameter: You could then use something like r.iter_content to iterate over the response data in chunks and then write them to a file. e.g.
import requests
url = "http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi"
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post(url, data=dc, auth=(username, password), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
for chunk in r.iter_content():
f.write(chunk)
I created this small example. I have an EchoServer on Port 8080 and a LogServer on Port 9090 (exemplary in this example). Both are started on the same machine (with Server, which contains the main).
Server started on port 8080
Server started on port 9090
As soon a client -- via telnet -- connects, the EchoServer establishes a connection to the LogServer. Now I am entering a long text, let's say 5000 character (see the long_text in the example), even if bash cannot handle it:
EchoServer Received: 1024
LogServer Received: 1024
EchoServer Received: 2048
LogServer Received: 2048
EchoServer Received: 1025
LogServer Received: 1025
If I enter the text again, I am getting:
EchoServer Received: 2048
LogServer Received: 2048
EchoServer Received: 2049
LogServer Received: 2049
Let's do it again:
EchoServer Received: 3072
EchoServer Received: 1025
LogServer Received: 3072
LogServer Received: 1025
And again:
EchoServer Received: 4096
EchoServer Received: 1
LogServer Received: 4096
LogServer Received: 1
The last time:
EchoServer Received: 4097
LogServer Received: 4097
My observation:
First of all, the data is fragmented. Additionally, each time the fragmends are extended by 1024 bytes (1024,2048,3072,4096,...). I guess the last behavious is because of the TCP slow start.
How can I achive the forwarding to the LogServer without fragmentation, such my text will arrive as one single message? I guess the problem is, how I connect to the LogServer.
[EDIT1]
I changed the logs. It seems, that it's already happening between telnet and the EchoSever. Anyway, I still have the problem in the real environment. The whole message (some Kilobyte) is arriving via WebSockets and the Forwarding to another Connection is fragmented.
[EDIT2]
I did some more research (with wireshark -- the log). I guess it has noting to do with TCP Slow Start. The data (I was sending 4095 times the letter A) arriving on the machine as three correct TCP packets:
Frame 1 (1506 bytes) with 1440 bytes TCP data (41 41 41 ... 41 41 41/HEX)
Frame 2 (1506 bytes) with 1440 bytes TCP data (41 41 41 ... 41 41 41/HEX)
Frame 3 (1283 bytes) with 1217 bytes TCP data (41 41 41 ... 41 0d 0a/HEX)
All 4095 A characters + CRLF arrived as expected.
The EchoServer said:
EchoServer Received: 1024
EchoServer Received: 2048
EchoServer Received: 1025
It also received the 4095 characters + CRLF, but it is different fragmented than the TCP segments (exactly same as the first log above). How can I avoid this Netty behavior?
In non-blocking I/O, there's no practical way to get the number of available bytes in socket receive buffer. Because of that problem, Netty predicts the number of available bytes. It starts from 1024 and then increases the prediction depending the number of read bytes. You can shcnage this behavior by employing a different prediction algorithm.
The default implementation is AdaptiveReceiveBufferSizePredictor and you might want to take a look into its source code to write your own one.
However, no matter what prediction algorithm you choose, you have to keep in mind that TCP/IP is a streaming protocol, which means you can always get messages in a split or merged form. Please refer to the user guide: http://netty.io/docs/stable/guide/html/ (See the 'Dealing with a Stream-based Transport' section.)
You require a FrameDecoder in your pipeline can which assemble bytes from the network into complete frames. In your case I think you need to combine the StringDecoder and DelimiterBasedFrameDecoder. Take a look at the Telnet example and specifically the TelnetServerPipelineFactory