So I've writren a scraper that uses the requests_pkcs12-library and a .p12-cert.
Currently I'm making a lot a requests where I use it as described in the docs
from requests_pkcs12 import get
r = get('https://example.com/test', pkcs12_filename='clientcert.p12', pkcs12_password='correcthorsebatterystaple')
The docs also show that you can use it in a session instead.
from requests import Session
from requests_pkcs12 import Pkcs12Adapter
session = Session()
session.mount('https://example.com', Pkcs12Adapter(pkcs12_filename='clientcert.p12', pkcs12_password='correcthorsebatterystaple'))
r = session.get('https://example.com/test')
So this works fine as well. But what are the advantages of doing so? Does it put less presure on the server which it authenticates with?
It does not seem to be adding an authenticated cookie to the session so I was wondering why the one would be prefered over the other.
Does anyone know this?
Thanks in advance
Yes, using a session may improve performance on client-side and reduce the server-side load as well.
Note that this has almost nothing to do with the requests_pkcs12 library, but is a generic mechanism of the requests library.
The requests manual states:
The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3’s connection pooling. So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).
The linked Wikipedia entry for HTTP persistent connection states:
Advantages
...
Reduced CPU usage and round-trips because of fewer new connections and TLS handshakes.
And the Wikipedia section on TLS handshakes states:
Client-authenticated TLS handshake
...
Negotiation Phase:
...
The server sends a CertificateRequest message, to request a certificate from the client so that the connection can be mutually authenticated.
...
The client responds with a Certificate message, which contains the client's certificate.
To summarize, using a requests session leads to connection pooling which leads to fewer TCP connections and hence fewer TLS handshakes, which in turn means fewer client certificate authentications. Note that this is independent on how the client certificate was made available to requests, whether in PKCS12 format (using requests_pkcs12) or PEM format (using plain requests).
I am using mailR to send emails through R. This is my code
send.mail(from = [from],
to = [to],
subject = "msg",
body = "contents",
html = FALSE,
inline = FALSE,
authenticate = TRUE,
smtp = list(host.name = "smtp.gmail.com",
port = 465,
user.name = [username],
passwd = [password],
ssl = TRUE),
attach.files = "/home/User/outputlog.txt",
send = TRUE)
I am sending sensitive info in the attachment. I am sending it through SSL.
I read this post about how secure SSL is and it looks pretty secure.
Does this message get encrypted in transit?
In theory, yes (for some definition of "transit"), but in practice for "Does this message get encrypted in transit?" the answer is maybe. In short, just ssl = True or equivalent put somewhere does almost not guarantee anything really, for all the reasons explained below.
Hence you are probably not going to like the following detailed response, as it shows basically that nothing is simple and that you have no 100% guarantee even if you do everything right and you have A LOT of things to do right.
Also TLS is the real true name of the feature you are using, SSL is dead since 20 days now, yes everyone use the old name, but that does not make this usage right nevertheless.
First, and very important, TLS provides various guarantees, among which confidentiality (the content is encrypted while in transit), but also authentication which is in your case far more important, and for the following reasons.
You need to make sure that smtp.gmail.com is resolved correctly, otherwise if your server uses lying resolvers, and is inside an hostile network that rewrites the DNS queries or responses, then you can send an encrypted content... to another party than the real "smtp.gmail.com" which makes the content not confidential anymore because you are sending it to a stranger or an active attacker.
To solve that, you need basically DNSSEC, if you are serious.
No, and contrary to what a lot of people seem to believe and convey, TLS alone or even DOH - DNS over HTTPS - do not solve that point.
Why? Because of the following that is not purely theoretical since it
happened recently (https://www.bleepingcomputer.com/news/security/hacker-hijacks-dns-server-of-myetherwallet-to-steal-160-000/), even if it was in the WWW world and not the email, the scenario can be the same:
you manage to grab the IP addresses tied to the name contacted (this can be done by a BGP hijack and it happens, for misconfigurations, "policy" reasons, or active attacks, all the time)
now that you control all communications, you put whatever server you need at the end of it
you contact any CA delivering DV certificates, including those purely automated
since the name now basically resolve to an IP you control, the web (or even DNS) validation that a CA can do will succeed and the CA will give you a certificate for this name (which may continue to work even after the end of the BGP hijack because CAs may not be quick to revoke certificates, and clients may not properly check for that).
hence any TLS stack accepting this CA will happily accept this certificate and your client will send securely content with TLS... to another target than the intended one, hence 0 real security.
In fact, as the link above shows, attackers do not even need to be so smart: even a self signed certificate or an hostname mismatch may go through because users will not care and/or library will have improper default behavior and/or programmer using the library will not use it properly (see this fascinating, albeit a tad old now, paper showing the very sad state of many "SSL" toolkits with incorrect default behavior, confusing APIs and various errors making invalid use of it far more probably than proper sane TLS operations: https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf)
Proper TLS use does not make DNSSEC irrelevant. Both targets and protects against different attacks. You need both to be more secure than just with one, and any of the two (properly used) does not replace the other. Never has and never will.
Now even if the resolution is correct, someone may have hijacked (thanks to BGP) the IP address. Then, again, you are sending to some host some encrypted content except that you do not really authenticate who is this host, so it can be anyone if an attacker managed to hijack IP addresses of smtp.gmail.com (it does not need to do it globally, just locally, "around" where your code execute).
This is where the very important TLS property of authentication kicks in.
This is typically done through X.509 certificates (which will be called - incorrectly - SSL certificates everywhere). Each end of the communication authenticate the other one by looking at the certificate it presented: either it recognizes this certificate as special, or it recognizes the issuing authority of this certificate as trusted.
So you do not just need to connect with TLS on smtp.gmail.com you also need to double check that the certificate then presented:
is for smtp.gmail.com (and not any other name), taking into account wildcards
is issued by a certificate authority you trust
All of this is normally handled by the TLS library you use except that in many cases you need at least to explicitly enable this behaviour (verification) and you need, if you want to be extra sure, to decide clearly with CAs you trust. Otherwise, too many attacks happen as can be seen in the past by rogue, incompetent or other adjectives CAs that issued certificates where they should not (and yes noone is safe against that, even Google and Microsoft got in the past mis-issued certificates with potential devastating consequences).
Now you have another problem more specific to SMTP and SMTP over TLS: the server typically advertises it does TLS and the client seeing this then can start the TLS exchange. Then all is fine (baring all the above).
But in the path between the SMTP server and you someone can rewrite the first part (which is in clear) in order to remove the information that this SMTP server speaks TLS. Then the client will not see TLS and will continue (depending on how it is developed, of course to be secure in such cases the client should abort the communication), then speaking in clear. This is called a downgrade attack. See this detailed explanation for example: https://elie.net/blog/understanding-how-tls-downgrade-attacks-prevent-email-encryption/
As Steffen points out, based on the port you are using this above issue of SMTP STARTTLS and hence the possible downgrade does not exist, because this is for port 25 which you are not using. However I prefer to still warn users about this case because it may not be well known and downgrade attacks are often both hard to detect and hard to defend against (all of this because protocols used nowadays were designed at a time where there was no need to even think about defending one against a malicious actor on the path)
Then of course you have the problem of the TLS version you use, and its parameters. The standard is now TLS version 1.3 but this is still slowly being deployed everywhere. You will find many TLS servers only knowing about 1.2
This can be good enough, if some precautions are taken. But you will also find old stuff speaking TLS 1.1, 1.0 or even worse (that is SSL 3). A secure client code should refuse to continue exchanging packets if it was not able to secure at least a TLS 1.2 connection.
Again this is normally all handled by your "SSL" library, but again you have to check for that, enable the proper settings, etc.
You have also a similar downgrade attack problem: without care, a server first advertise what it offers, in clear, and hence an attacker could modify this to remove the "highest" secure versions to force the client to use a lower versions that has more attacks (there are various attacks against TLS 1.0 and 1.1).
There are solutions, specially in TLS 1.3 and 1.2 (https://www.rfc-editor.org/rfc/rfc7633 : "The purpose of the TLS feature extension is to prevent downgrade
attacks that are not otherwise prevented by the TLS protocol.")
Aside and contrary to Steffen's opinion I do no think that TLS downgrade attacks are purely theoretical. Some examples:
(from 2014): https://p16.praetorian.com/blog/man-in-the-middle-tls-ssl-protocol-downgrade-attack (mostly because web browsers are eager to connect no matter what so typically if an attempt with highest settings fail they will fallback to lower versions until finding a case where the connection happens)
https://www.rfc-editor.org/rfc/rfc7507 specifically offers a protection, stating that: "All unnecessary protocol downgrades are undesirable (e.g., from TLS
1.2 to TLS 1.1, if both the client and the server actually do support
TLS 1.2); they can be particularly harmful when the result is loss of
the TLS extension feature by downgrading to SSL 3.0. This document
defines an SCSV that can be employed to prevent unintended protocol
downgrades between clients and servers that comply with this document
by having the client indicate that the current connection attempt is
merely a fallback and by having the server return a fatal alert if it
detects an inappropriate fallback."
https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/february/downgrade-attack-on-tls-1.3-and-vulnerabilities-in-major-tls-libraries/ discusses not less than 5 CVEs in 2018 that allows TLS attacks: " Two ways exist to attack TLS 1.3. In each attack, the server needs to support an older version of the protocol as well. [..] The second one relies on the fact that both peers support an older version of TLS with a cipher suite supporting an RSA key exchange." and "This prowess is achieved because of the only known downgrade attack on TLS 1.3." and "Besides protocol downgrades, other techniques exist to force browser clients to fallback onto older TLS versions: network glitches, a spoofed TCP RST packet, a lack of response, etc. (see POODLE)".
Even if you are using a correct version, you need to make sure to use correct algorithms, key sizes, etc. Sometimes some server/library enable a "NULL" encryption algorithm, which means in fact no encryption. Silly of course, but that exists, and this is a simple case, there are far more complicated ones.
This other post from Steffen: https://serverfault.com/a/696502/396475 summarizes and touches the various above points, and gives another views on what is most important (we disagree on this, but he answered here as well so anyone is free to take both views into account and make their own opinion).
Hence MTA-STS instead of SMTP STARTTLS, https://www.rfc-editor.org/rfc/rfc8461 with this clear abstract:
SMTP MTA Strict Transport Security (MTA-STS) is a mechanism
enabling mail service providers (SPs) to declare their ability to
receive Transport Layer Security (TLS) secure SMTP connections and
to specify whether sending SMTP servers should refuse to deliver to
MX hosts that do not offer TLS with a trusted server certificate.
Hence you will need to make sure that the host you send your email too does use that feature, and that your client is correctly programmed to handle it.
Again, probably done inside your "SSL Library" but this clearly show you need specific bit in it for SMTP, and you need to contact a webserver to retrieve the remote end SMTP policies, and you need also to do DNS requests, which gets back to you on one of the earlier point about if you trust your resolver or not and if records are protected with DNSSEC.
And with all the above, which already covers many areas and is really hard to do correctly, there are still many other points to cover...
The transit is safe, let us assume. But then how does the content gets retrieved? You may say it is not your problem anymore. Maybe. Maybe not. Do you want to be liable for that? Which means that you should maybe also encrypt the attachment itself, this is in addition (not in replacement) of the transport being secured.
The default mechanisms to secure email contents either use OpenPGP (has a more geek touch to it), or S/MIME (has a more corporate touch to it). This works for everything. Then you have specific solutions depending on the document (but this does not solve the problem of securing the body of the email), like PDF documents can be protected by a password (warning: this has been cracked in the past).
I am sending sensitive info
This is then probably covered by some contract or some norms, depending on your area of business. You may want to dig deeper into those to see exactly what are the requirements forced upon you so that you are not liable for some problems, if you secured everything else correctly.
First, even if SSL/TLS is properly used when delivering the mail from the client it only protects the first step of delivery, i.e. the delivery to the first MTA (mail transfer agent). But mail gets delivered in multiple steps over multiple MTA and then it gets finally retrieved from the client from the last mail server.
Each of these hops (MTA) has access to the plain mail, i.e. TLS is only between hops but not end-to-end between sender and recipient. Additionally the initial client has no control how one hop will deliver the mail to the next hop. This might be also done with TLS but it might be done in plain. Or it might be done with TLS where no certificates get properly checked which means that it is open to MITM attacks. Apart from that each MTA in the delivery chain has access to the mail in plain text.
In addition to that the delivery to the initial MTA might already have problems. While you use port 465 with smtps (TLS from start instead upgrade from plain using a STARTTLS command) the certificate of the server need to be properly checked. I've had a look at the source code of mailR to check how this is done: mailR essentially is using Email from Apache Commons. And while mailR uses setSSL to enable TLS from start it does not use setSSLCheckServerIdentity to enable proper checking of the certificate. Since the default is to not properly check the certificate already the connection to the initial MTA is vulnerable to man in the middle attacks.
In summary: the delivery is not secure, both due to how mail delivery works (hop-by-hop and not end-to-end) and how mailR uses TLS. To have proper end-to-end security you'll to encrypt the mail itself and not just the delivery. PGP and S/MIME are the established methods for this.
For more see also How SSL works in SMTP? and How secure is e-mail landscape right now?.
Recently, one of our ASP.NET ASHX handlers that makes a call to another ASHX handler on another server started to fail with the following error: "Unhandled Exception: System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel." Installing the certificate for the other server did not help (in any case, the other server is public-facing and has a cert signed by a CA).
I wrote a small C# program to test if I could connect to the other server from the command line:
using System.Net;
public class Test
{
const string theUrl = "https://www.example.com/Handler.ashx?ID=123";
public static void Main(string[] args)
{
var req = WebRequest.Create(theUrl);
var resp = req.GetResponse();
}
}
This, too, fails with a System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. However, if I make the call to the handler from a VBScript file:
Const theUrl = "https://www.example.com/Handler.ashx?ID=123"
Dim oXmlHttp
Set oXmlHttp = WScript.CreateObject("Microsoft.XMLHTTP")
With oXmlHttp
.Open "GET", theUrl, False
.Send
WScript.Echo CStr(.Status)
End With
Set oXmlHttp = Nothing
it works; the script returns a status of 200!
I checked the .NET machine.config file as I encountered a similar issue in the past where requests sent from unmanaged code worked but not from .NET because the proxy settings were different, but this is not the case here. Why should a request from VBScript work but fail from .NET?
EDIT: Enabling tracing reveals a similar error to the one reported in this question:
System.Net Information: 0 : [28468] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 1ad5f0:29f3620, targetName = www.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [28468] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=AlgorithmMismatch).
System.Net.Sockets Verbose: 0 : [28468] Socket#3741682::Dispose()
System.Net Error: 0 : [28468] Exception in the HttpWebRequest#2383799:: - The request was aborted: Could not create SSL/TLS secure channel.
System.Net Error: 0 : [28468] Exception in the HttpWebRequest#2383799::GetResponse - The request was aborted: Could not create SSL/TLS secure channel.
"Algorithm mismatch" is a bit vague, especially since TLS errors are well defined. No matter.
Run ssl-enum-ciphers (perl) or nmap's equivalent against the server to see what SSL/TLS versions and ciphers it supports, and prefers. In pinch you could try MS wfetch to do something similar, but it has quite limited protocol and cipher support.
You can do the equivalent for the client using openssl as a server (key+cert required):
openssl s_server -quiet -accept 4443 -key server.key -cert server.crt
optionally mix in -no_xxx or -xxx where xxx is one of ssl2, ssl3, tls1 for variety.
(Browser friendly, but less easy to code/automate is https://www.ssllabs.com/ssltest/viewMyClient.html or see here for more.)
The way the protocol/cipher agreement works is that the client sends (ClientHello) an SSL/TLS version, a list of ciphers (and extensions), then server chooses the protocol version, and a cipher in common (intersecting set) and replies (ServerHello). The (rough) method for choosing is:
use the highest SSL/TLS version supported by the client, or give up (possible handshake failure)
prefer the first common cipher in the client list, i.e. assume it's ordered by preference
-or- prefer the first common cipher in the server list [1]
-or- give up with (no common cipher)
[1] this allows the operator to configure the server prefer ciphers based on other than "strength" (symmetric key-size), e.g. to force 128-bit RC4 over AES-256 while supporting both, as was the style a year or two back.
The default cipher order is usually by descending "strength" (symmetric cipher key size). An SSL/TLS protocol version sets out an expected set of ciphers, but supporting the same cipher (e.g. AES-128) isn't enough.
It can be the case that the client won't talk TLS, and the server won't talk SSL, e.g. https://serverfault.com/questions/208542/what-might-cause-https-failure-when-not-specifying-ssl-protocol
This should cause a protocol or handshake error though.
Since forcing the protocol via SecurityProtocolType.Ssl3 (another case here), it looks like the client is proposing something the server doesn't like -- if you have confirmed both sides support TLS then it might be SNI tripping you up, and falling back to SSL3 side-steps that.
In the unlikely event that the client side is defaulting to SSlv2, the server is quite right to reject it ;-)
We found a workaround but it's not particularly elegant. The problem started to occur when the server to which we are connecting was replaced with a load-balanced configuration. If I edit the hosts file on the client machine to talk to the server directly rather than go through the load-balancer, it works.
In addition, I verified that the code changes in this answer also work. It would still be nice to know the differences are between the managed and unmanaged implementations of SSL/TLS...
I have a server program which listens on a particular port.
I have a requirement where client program that tries to connect to my server must be initiated by a root user.
How do I ensure this in the server program?
How do I ensure [anything about the
client program] in the server program?
You can't. If your security model requires the server to know whether client is root, you don't have security.
Let's consider one possibility: your network protocol includes a notification like this:
My-Uid-Is: 0
Your client, the perfectly secure version that you wrote, might implement this notification like this:
fprintf(socketFd, "My-Uid-Is: %d\n", getuid()); // send server my identity
But, my client, the one what I wrote without your knowledge or consent, will implement the notification like this:
fprintf(socketFd, "My-Uid-Is: 0\n"); // lie to server about my identity
Pop quiz: how can your server know whether it is talking to your truthful client, or my lying client? Answer: it can't. In fact, if you generalize this concept, you realize that the server can't rely upon the validity (whether that means the truthfulness, the format, the range-checking, etc.) of anything the client says.
In this specific case, using the clients source port number is as unreliable as any other choice. Yes, many operating systems require root privileges to bind to low-numbered source ports. But my PC might not be running your favorite operating system. I might be connecting from my own PC running my own OS which doesn't have that feature. Remember: you can't trust anything the client says.
There are techniques involving public-key encryption that can be used to guarantee that the program you are talking to has access to specific secrets. That, assuming that the secrets are adequately protected, can be used to guarantee that a specific person, computer, or account generated the request. I'll let someone else discuss PKI and how it might apply to your situation.
The client should bind to a port below 1024 before connecting. This port range is reserved for root.
I'm using IE9 beta and Fiddler to understand the https session negotiation taking place for the above url (chosen for no paritcular reason other than it's secured).
Some observations made me curious.. does anyone understand what's happening here?
1. When I connect with Fiddler setting: HTTPS decrypt OFF, I see this sequence
5 CONNECTs to fiddler2.com with nothing but headers showing
a) Curious, why more than one?
1 CONNECT to beta.urs.microsoft.com
b) Does this have something to do with asking MS which cert it recognises? I thought this data is supposed to be kept locally? Maybe that only happened because I'm using a beta of IE9?
4 CONNECTs to fiddler2.com with the same SessionID but different Random and the list of ciphers available on the client.
1 CONNECT to beta.urs.microsoft.com with similar content to above 4
c) Why the multiple CONNECTs here with different Random?
2. When I connect with Fiddler setting: HTTPS decrypt ON, I see this sequence
5 CONNECTs to fiddler2.com with nothing but headers in the request only and the response shows a certificate and the chosen cipher. Same in all 5.
a) same question
1 GET with the page contents
d) what happened to the extra CONNECTs this time?
I'm trying to relate what I see here to the negotiation between client and server as it's documented here.
Transport Layer Security
Tyia,
Mick.
You didn't mention what browser you're using and what ciphers you have enabled in that browser.
Sometimes, you'll see multiple CONNECT handshakes because the server immediately closes the connection (ungracefully stating that they don't support the requested protocol version) and the client will retry (fallback) to an older protocol version. You definitely see this happen a lot if you enable TLSv1.1 and TLSv1.2 in IE, for instance.
You also may see multiple CONNECTs if the client aborts a connection and then attempts to open a new one.
urs.microsoft.com and beta.urs.microsoft.com are used for the SmartScreen site-reputation feature.