I'm evaluating setting up SPF for a domain that already has SenderID. We're considering removing the SenderID record entirely and replacing it with just an SPF record, instead of trying to write a SenderID record that tries to encompass SPF and SenderID.
We have two outbound servers, and two inbound servers which relay to internal Exchange machines.
Any bounces received by the two inbound server are delivered directly to the sender. I wasn't sure if that was considered sending and therefore should be included with the SPF record?
There is also a POP/IMAP server on a subdomain of our main domain.
I'm using 192.168.1.10 and 192.168.2.10 as our outbound servers, mail1.mydomain.com and mail2.mydomain.com for privacy here.
I believe the following would be the proper SPF record for our domain:
mydomain.com. 3600 IN TXT "v=spf1 ipv4:192.168.1.10 ipv4:192.168.2.10 ptr:subhost.mydomain.com mx:mail1.mydomain.com mx:mail2.mydomain.com a:subhost.mydomain.com include:constant-contact.com -all"
Is the PTR and A fields for the POP/IMAP host on the network that also sends mail as user#subhost.mydomain.com?
If the marketing folks frequently change who they use to send marketing emails as "user#mydomain.com", would you recommend ~all instead of -all?
We currently have "spf2.0/mfrom,pra ..." as our SenderID record. I would be interested in input on how to adapt that to properly support SPF as well.
It appears that even microsoft.com and live.com don't include "spf2.0" records, but instead just spf. Is anyone even using it anymore?
Thanks,
Alex
I think the simplest SPF record that would fulfill your needs is:
"v=spf1 ip4:192.168.1.10 ip4:192.168.2.10 include:constant-contact.com -all"
You should not use the ptr mechanism, since it can require a lot of DNS queries on the receiving end, and should not be needed in this case. The use of ptr is also discouraged by rfc7208.
I'll guess that mail1 and mail2 are the names for the receiving mail servers. If you want them listed also (I don't think they are needed since bounces normally are delivered internally) you should just use the "mx" mechanism. This will match the ip-address against all mx records for the mydomain.com domain.
Regarding sending as user#subhost.mydomain.com, then this SPF record will not be used for that domain, since the sending domain is subhost.mydomain.com. Instead you will have to add a SPF record for the subdomain. If subhost.mydomain.com is one of the listed sending servers (the ip4s), this record could possible link to the record for mydomain.com:
"v=spf1 redirect=mydomain.com"
And regarding the -all/~all, then I would recommend using ~all if there is a real risk of the marketing department switching mailing list providers without informing IT (It would not be the first time). Just be aware that some receivers may still block mails when SPF returns SoftFail (from ~all), especially of the mailing provider is listed on some of the blacklists.
Related
I am using mailR to send emails through R. This is my code
send.mail(from = [from],
to = [to],
subject = "msg",
body = "contents",
html = FALSE,
inline = FALSE,
authenticate = TRUE,
smtp = list(host.name = "smtp.gmail.com",
port = 465,
user.name = [username],
passwd = [password],
ssl = TRUE),
attach.files = "/home/User/outputlog.txt",
send = TRUE)
I am sending sensitive info in the attachment. I am sending it through SSL.
I read this post about how secure SSL is and it looks pretty secure.
Does this message get encrypted in transit?
In theory, yes (for some definition of "transit"), but in practice for "Does this message get encrypted in transit?" the answer is maybe. In short, just ssl = True or equivalent put somewhere does almost not guarantee anything really, for all the reasons explained below.
Hence you are probably not going to like the following detailed response, as it shows basically that nothing is simple and that you have no 100% guarantee even if you do everything right and you have A LOT of things to do right.
Also TLS is the real true name of the feature you are using, SSL is dead since 20 days now, yes everyone use the old name, but that does not make this usage right nevertheless.
First, and very important, TLS provides various guarantees, among which confidentiality (the content is encrypted while in transit), but also authentication which is in your case far more important, and for the following reasons.
You need to make sure that smtp.gmail.com is resolved correctly, otherwise if your server uses lying resolvers, and is inside an hostile network that rewrites the DNS queries or responses, then you can send an encrypted content... to another party than the real "smtp.gmail.com" which makes the content not confidential anymore because you are sending it to a stranger or an active attacker.
To solve that, you need basically DNSSEC, if you are serious.
No, and contrary to what a lot of people seem to believe and convey, TLS alone or even DOH - DNS over HTTPS - do not solve that point.
Why? Because of the following that is not purely theoretical since it
happened recently (https://www.bleepingcomputer.com/news/security/hacker-hijacks-dns-server-of-myetherwallet-to-steal-160-000/), even if it was in the WWW world and not the email, the scenario can be the same:
you manage to grab the IP addresses tied to the name contacted (this can be done by a BGP hijack and it happens, for misconfigurations, "policy" reasons, or active attacks, all the time)
now that you control all communications, you put whatever server you need at the end of it
you contact any CA delivering DV certificates, including those purely automated
since the name now basically resolve to an IP you control, the web (or even DNS) validation that a CA can do will succeed and the CA will give you a certificate for this name (which may continue to work even after the end of the BGP hijack because CAs may not be quick to revoke certificates, and clients may not properly check for that).
hence any TLS stack accepting this CA will happily accept this certificate and your client will send securely content with TLS... to another target than the intended one, hence 0 real security.
In fact, as the link above shows, attackers do not even need to be so smart: even a self signed certificate or an hostname mismatch may go through because users will not care and/or library will have improper default behavior and/or programmer using the library will not use it properly (see this fascinating, albeit a tad old now, paper showing the very sad state of many "SSL" toolkits with incorrect default behavior, confusing APIs and various errors making invalid use of it far more probably than proper sane TLS operations: https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf)
Proper TLS use does not make DNSSEC irrelevant. Both targets and protects against different attacks. You need both to be more secure than just with one, and any of the two (properly used) does not replace the other. Never has and never will.
Now even if the resolution is correct, someone may have hijacked (thanks to BGP) the IP address. Then, again, you are sending to some host some encrypted content except that you do not really authenticate who is this host, so it can be anyone if an attacker managed to hijack IP addresses of smtp.gmail.com (it does not need to do it globally, just locally, "around" where your code execute).
This is where the very important TLS property of authentication kicks in.
This is typically done through X.509 certificates (which will be called - incorrectly - SSL certificates everywhere). Each end of the communication authenticate the other one by looking at the certificate it presented: either it recognizes this certificate as special, or it recognizes the issuing authority of this certificate as trusted.
So you do not just need to connect with TLS on smtp.gmail.com you also need to double check that the certificate then presented:
is for smtp.gmail.com (and not any other name), taking into account wildcards
is issued by a certificate authority you trust
All of this is normally handled by the TLS library you use except that in many cases you need at least to explicitly enable this behaviour (verification) and you need, if you want to be extra sure, to decide clearly with CAs you trust. Otherwise, too many attacks happen as can be seen in the past by rogue, incompetent or other adjectives CAs that issued certificates where they should not (and yes noone is safe against that, even Google and Microsoft got in the past mis-issued certificates with potential devastating consequences).
Now you have another problem more specific to SMTP and SMTP over TLS: the server typically advertises it does TLS and the client seeing this then can start the TLS exchange. Then all is fine (baring all the above).
But in the path between the SMTP server and you someone can rewrite the first part (which is in clear) in order to remove the information that this SMTP server speaks TLS. Then the client will not see TLS and will continue (depending on how it is developed, of course to be secure in such cases the client should abort the communication), then speaking in clear. This is called a downgrade attack. See this detailed explanation for example: https://elie.net/blog/understanding-how-tls-downgrade-attacks-prevent-email-encryption/
As Steffen points out, based on the port you are using this above issue of SMTP STARTTLS and hence the possible downgrade does not exist, because this is for port 25 which you are not using. However I prefer to still warn users about this case because it may not be well known and downgrade attacks are often both hard to detect and hard to defend against (all of this because protocols used nowadays were designed at a time where there was no need to even think about defending one against a malicious actor on the path)
Then of course you have the problem of the TLS version you use, and its parameters. The standard is now TLS version 1.3 but this is still slowly being deployed everywhere. You will find many TLS servers only knowing about 1.2
This can be good enough, if some precautions are taken. But you will also find old stuff speaking TLS 1.1, 1.0 or even worse (that is SSL 3). A secure client code should refuse to continue exchanging packets if it was not able to secure at least a TLS 1.2 connection.
Again this is normally all handled by your "SSL" library, but again you have to check for that, enable the proper settings, etc.
You have also a similar downgrade attack problem: without care, a server first advertise what it offers, in clear, and hence an attacker could modify this to remove the "highest" secure versions to force the client to use a lower versions that has more attacks (there are various attacks against TLS 1.0 and 1.1).
There are solutions, specially in TLS 1.3 and 1.2 (https://www.rfc-editor.org/rfc/rfc7633 : "The purpose of the TLS feature extension is to prevent downgrade
attacks that are not otherwise prevented by the TLS protocol.")
Aside and contrary to Steffen's opinion I do no think that TLS downgrade attacks are purely theoretical. Some examples:
(from 2014): https://p16.praetorian.com/blog/man-in-the-middle-tls-ssl-protocol-downgrade-attack (mostly because web browsers are eager to connect no matter what so typically if an attempt with highest settings fail they will fallback to lower versions until finding a case where the connection happens)
https://www.rfc-editor.org/rfc/rfc7507 specifically offers a protection, stating that: "All unnecessary protocol downgrades are undesirable (e.g., from TLS
1.2 to TLS 1.1, if both the client and the server actually do support
TLS 1.2); they can be particularly harmful when the result is loss of
the TLS extension feature by downgrading to SSL 3.0. This document
defines an SCSV that can be employed to prevent unintended protocol
downgrades between clients and servers that comply with this document
by having the client indicate that the current connection attempt is
merely a fallback and by having the server return a fatal alert if it
detects an inappropriate fallback."
https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/february/downgrade-attack-on-tls-1.3-and-vulnerabilities-in-major-tls-libraries/ discusses not less than 5 CVEs in 2018 that allows TLS attacks: " Two ways exist to attack TLS 1.3. In each attack, the server needs to support an older version of the protocol as well. [..] The second one relies on the fact that both peers support an older version of TLS with a cipher suite supporting an RSA key exchange." and "This prowess is achieved because of the only known downgrade attack on TLS 1.3." and "Besides protocol downgrades, other techniques exist to force browser clients to fallback onto older TLS versions: network glitches, a spoofed TCP RST packet, a lack of response, etc. (see POODLE)".
Even if you are using a correct version, you need to make sure to use correct algorithms, key sizes, etc. Sometimes some server/library enable a "NULL" encryption algorithm, which means in fact no encryption. Silly of course, but that exists, and this is a simple case, there are far more complicated ones.
This other post from Steffen: https://serverfault.com/a/696502/396475 summarizes and touches the various above points, and gives another views on what is most important (we disagree on this, but he answered here as well so anyone is free to take both views into account and make their own opinion).
Hence MTA-STS instead of SMTP STARTTLS, https://www.rfc-editor.org/rfc/rfc8461 with this clear abstract:
SMTP MTA Strict Transport Security (MTA-STS) is a mechanism
enabling mail service providers (SPs) to declare their ability to
receive Transport Layer Security (TLS) secure SMTP connections and
to specify whether sending SMTP servers should refuse to deliver to
MX hosts that do not offer TLS with a trusted server certificate.
Hence you will need to make sure that the host you send your email too does use that feature, and that your client is correctly programmed to handle it.
Again, probably done inside your "SSL Library" but this clearly show you need specific bit in it for SMTP, and you need to contact a webserver to retrieve the remote end SMTP policies, and you need also to do DNS requests, which gets back to you on one of the earlier point about if you trust your resolver or not and if records are protected with DNSSEC.
And with all the above, which already covers many areas and is really hard to do correctly, there are still many other points to cover...
The transit is safe, let us assume. But then how does the content gets retrieved? You may say it is not your problem anymore. Maybe. Maybe not. Do you want to be liable for that? Which means that you should maybe also encrypt the attachment itself, this is in addition (not in replacement) of the transport being secured.
The default mechanisms to secure email contents either use OpenPGP (has a more geek touch to it), or S/MIME (has a more corporate touch to it). This works for everything. Then you have specific solutions depending on the document (but this does not solve the problem of securing the body of the email), like PDF documents can be protected by a password (warning: this has been cracked in the past).
I am sending sensitive info
This is then probably covered by some contract or some norms, depending on your area of business. You may want to dig deeper into those to see exactly what are the requirements forced upon you so that you are not liable for some problems, if you secured everything else correctly.
First, even if SSL/TLS is properly used when delivering the mail from the client it only protects the first step of delivery, i.e. the delivery to the first MTA (mail transfer agent). But mail gets delivered in multiple steps over multiple MTA and then it gets finally retrieved from the client from the last mail server.
Each of these hops (MTA) has access to the plain mail, i.e. TLS is only between hops but not end-to-end between sender and recipient. Additionally the initial client has no control how one hop will deliver the mail to the next hop. This might be also done with TLS but it might be done in plain. Or it might be done with TLS where no certificates get properly checked which means that it is open to MITM attacks. Apart from that each MTA in the delivery chain has access to the mail in plain text.
In addition to that the delivery to the initial MTA might already have problems. While you use port 465 with smtps (TLS from start instead upgrade from plain using a STARTTLS command) the certificate of the server need to be properly checked. I've had a look at the source code of mailR to check how this is done: mailR essentially is using Email from Apache Commons. And while mailR uses setSSL to enable TLS from start it does not use setSSLCheckServerIdentity to enable proper checking of the certificate. Since the default is to not properly check the certificate already the connection to the initial MTA is vulnerable to man in the middle attacks.
In summary: the delivery is not secure, both due to how mail delivery works (hop-by-hop and not end-to-end) and how mailR uses TLS. To have proper end-to-end security you'll to encrypt the mail itself and not just the delivery. PGP and S/MIME are the established methods for this.
For more see also How SSL works in SMTP? and How secure is e-mail landscape right now?.
Here are my questions followed by some more information.
Is an IP Address considered PII (Personally Identifiable Information)?
We need to filter our measurement protocol traffic via the user's IP address, is there a way to do this?
We are using the Measurement Protocol to send custom event data to our Google Analytics account. All of the data is being sent via PHP cURL from the server. We have 3 different views setup in our account, (View #1) a view that is completely unfiltered, (View #2) another view that is filtering out internal traffic via IP addresses, and a final third view (View #3) for testing purposes.
View #2's filters have stopped working since we moved to this method of sending the event data to Google. I imagine that is because the requests are now coming from the server's IP address instead of each specific user. I have been told about a field that you can use to send the user's IP address to Google, the field is labeled "uip" however Google anonymizes this data and does seem to use it for filtering the views (what would the purpose of this field be then?).
I have a custom dimension setup in which I am sending a hashed IP address (as I am not sure if an IP is considered PII) I am then filtering the reports on those specific hashes ... however this leaves me unable to filter out IP ranges ... certain bot traffic can originate from different ranges of IP addresses and I would be unable to filter them from the reports.
I have been scouring the internet to try to determine whether or not it is a privacy concern for me to simply store the user IP (unhashed) in a custom dimension and setup our filtering rules based on that. This would allow me to create regex that filters out entire ranges of IP's. Most of the articles that say an IP is PII refer to Google's Universal Analytics Guidelines: https://support.google.com/analytics/answer/2795983 - but I have been all over those articles and I cannot see Google specifically stating anywhere whether or not an IP is PII.
Thank you for your time.
For the question of hashed vs. unhashed values - Google has two different policies on the question of hashing (as I only found out when I was researching your question).
For the question if IPs are PIIs - Google at document on "Best practices to avoid sendig PII":
https://support.google.com/analytics/answer/6366371?hl=en&ref_topic=2919631
which does not mention IP addresses. However Google does take some trouble to protect IP addresses (e.g. automatically anonymizing, not exposing them in the interface) so I'd suggest (based on gut feeling, not anything binding) that you do the same and at least hash them with a salted hash 8and filter by the hash).
Also part from the Google TOS there are national laws to consider (don't know where you are doing business, I live in Germany and here IP addresses are definitively PII. I think this is true for the rest of the EU as well).
Last week I started quite a fuss in my Computer Networks class over the need for a mandatory Host clause in the header of HTTP 1.1 GET messages.
The reason I'm provided with, be it written on the Web or shouted at me by my classmates, is always the same: the need to support virtual hosting. However, and I'll try to be as clear as possible, this does not appear to make sense.
I understand that in order to allow two domains to be hosted in a single machine (and by consequence, share the same IP address), there has to exist a way of differentiating both domain names.
What I don't understand is why it isn't possible to achieve this without a Host clause (HTTP 1.0 style) by using an absolute URL (e.g. GET http://www.example.org/index.html) instead of a relative one (e.g. GET /index.html).
When the HTTP message got to the server, it (the server) would redirect the message to the appropriate host, not by looking at the Host clause but, instead, by looking at the hostname in the URL present in the message's request line.
I would be very grateful if any of you hardcore hackers could help me understand what exactly am I missing here.
This was discussed in this thread:
modest suggestions for HTTP/2.0 with their rationale.
Add a header to the client request that indicates the hostname and
port of the URL which the client is accessing.
Rationale: One of the most requested features from commercial server
maintainers is the ability to run a single server on a single port
and have it respond with different top level pages depending on the
hostname in the URL.
Making an absolute request URI required (because there's no way for the client to know on beforehand whether the server homes one or more sites) was suggested:
Re the first proposal, to incorporate the hostname somewhere. This
would be cleanest put into the URL itself :-
GET http://hostname/fred http/2.0
This is the syntax for proxy redirects.
To which this argument was made:
Since there will be a mix of clients, some supporting host name reporting
and some not, it just doesn't matter how this info gets to the server.
Since it doesn't matter, the easier to implement solution is a new HTTP
request header field. It allows all clients and servers to operate as they
do now with NO code changes. Clients and servers that actually need host
name information can have tiny mods made to send the extra header field
containing the URL and process it.
[...]
All I'm suggesting is that there is a better way to
implement the delivery of host name info to the server that doesn't involve
hacking the request syntax and can be backwards compatible with ALL clients
and servers.
Feel free to read on to discover the final decision yourself. But be warned, it's easy to get lost in there.
The reason for adding support for specifying a host in an HTTP request was the limited supply of IP addresses (which was not an issue yet when HTTP 1.0 came out).
If your question is "why specify the host in a Host header as opposed to on the Request-Line", the answer is the need for interopability between HTTP/1.0 and 1.1.
If the question is "why is the Host header mandatory", this has to do with the desire to speed up the transition away from assigned IP addresses.
Here's some background on the Internet address conservation with respect to HTTP/1.1.
The reason for the 'Host' header is to make explicit which host this request refers to. Without 'Host', the server must know ahead of time that it is supposed to route 'http://joesdogs.com/' to Joe's Dogs while it is supposed to route 'http://joscats.com/' to Jo's Cats even though they are on the same webserver. (What if a server has 2 names, like 'joscats.com' and 'joescats.com' that should refer to the same website?)
Having an explicit 'Host' header make these kinds of decisions much easier to program.
I'm pretty sure I remember reading --but cannot find back the links anymore-- about this: on some ISP (including at least one big ISP in the U.S.) it is possible to have a user's GET and POST request appearing to come from different IPs.
(note that this is totally programming related, and I'll give an example below)
I'm not talking about having your IP adress dynamically change between two requests.
I'm talking about this:
IP 1: 123.45.67.89
IP 2: 101.22.33.44
The same user makes a GET, then a POST, then a GET again, then a POST again and the servers see this:
- GET from IP 1
- POST from IP 2
- GET from IP 1
- POST from IP 2
So altough it's the same user, the webserver sees different IPs for the GET and the POSTs.
Surely seen that HTTP is a stateless protocol this is perfectly legit right?
I'd like to find back the explanation as to how/why certain ISP have their networks configured such that this may happen.
I'm asking because someone asked me to implement the following IP filter and I'm pretty sure it is fundamentally broken code (breaking havoc for at least one major american ISP users).
Here's a Java servlet filter that is supposed to protect against some attacks. The reasoning is that:
"For any session filter checks that IP address in the request is the same that was used when session was created. So in this case session ID could not be stolen for forming fake sessions."
http://www.servletsuite.com/servlets/protectsessionsflt.htm
However I'm pretty sure this is inherently broken because there are ISPs where you may see GET and POST coming from different IPs.
Some ISPs (or university networks) operate transparent proxies which relay the request from the outgoing node that is under the least network load.
It would also be possible to configure this on a local machine to use the NIC with the lowest load which could, again, result in this situation.
You are correct that this is a valid state for HTTP and, although it should occur relatively infrequently, this is why validation of a user based on IP is not an appropriate determinate of identity.
For a web server to be seeing this implies that the end user is behind some kind of proxy/gateway. As you say it's perfectly valid given that HTTP is stateless, but I imagine would be unusual. As far as I am aware most ISPs assign home users a real, non-translated IP (albeit usually dynamic).
Of course for corporate/institutional networks they could be doing anything. load balancing could mean that requests come from different IPs, and maybe sometimes request types get farmed out to different gateways (altho I'd be interested to know why, given that N_GET >> N_POST).
In my application, I have to send notification e-mails from time to time. In order to send mail (over SMTP), I have to get the MX server of that particular domain (domain part of e-mail address). This is not a Unix application but an Embedded one.
What I do goes like this ::
1 - Send a DNS query (MX type) containing the domain to the current DNS
2 - If the response contains the MX answer , return success from this function
3 - Read the first NS record and copy its IP address to the current DNS , goto 1
This may loop a few times and this is expected but what I do not expect is that the response contains NS records of servers named like ns1.blahblah.com but not their IP addresses. In this case, I have to send another query to find the IP of this NS. I have seen this for only 1 e-mail address (1 domain), the other addresses worked without any problem.
Is this normal behaviour ? IMHO, it is a misconfig on the DNS records. Any thoughts ?
Thanks in advance...
The authority section in the message, as well as the additional section are optional. Ie, the name servers and their IPs don't have to be in the response to the MX query. It is up to the DNS server to decide to send that extra information even when the server already has the data.
You are stuck having to query for the MX and then query for the IP of the mail server
Short answer to your question: RFC 1035 says,
NS records cause both the usual additional section processing to locate
a type A record, and, when used in a referral, a special search of the
zone in which they reside for glue information.
...the additional records section contains RRs
which relate to the query, but are not strictly answers for the
question.
...When composing a response, RRs which are to be inserted in the
additional section, but duplicate RRs in the answer or authority
sections, may be omitted from the additional section.
So the bottom line in my opinion is that, yes, if the response does not contain the A record matching the NS record it some section, something is likely misconfigured somewhere. But, as the old dodge goes, "be liberal in what you accept;" if you are going to make the queries, you will need to handle situations like this. DNS is awash in these kinds of problems.
The longer answer requires a question: how are you getting the original DNS server where you are starting the MX lookup?
What you are doing is a non-recursive query: if the first server you query does not know the answer, it points you at another server that is "closer" in the DNS hierarchy to the domain you are looking for, and you have to make the subsequent queries to find the MX record. If you are starting your query at one of the root servers, I think you will have to follow the NS pointers yourself like you are.
However, if the starting DNS server is configured in your application (i.e. a manual configuration item or via DHCP), then you should be able to make a recursive request, using the Recusion Desired flag, which will push the repeated lookup off onto the configured DNS server. In that case you would just get the MX record value in your first response. On the other hand, recursive queries are optional, and your local DNS server may not support them (which would be bizarre since, historically, many client libraries relied on recursive lookups).
In any case, I would personally like to thank you for looking MX records. I have had to deal with systems that wanted to send mail but could not do the DNS lookups, and the number and variety of bizarre and unpleasant hacks they have used has left me with emotional scars.
It could be that the domain simply does not have a MX record. I completely take out the MX entry for my unused / parked domains, it saves my mail server a lot of grief (SPAM).
There really is no need to go past step 2. If the system (or ISP) resolver returned no MX entry, its because it already did the extra steps and found nothing. Or, possibly, the system host resolver is too slow (i.e. from an ISP).
Still, I think its appropriate to just bail out if either happened, as its clearly a DNS or ISP issue, not a problem with the function. Just tell the user that you could not resolve a MX record for the domain, and let them investigate it on their end.
Also, is it feasible to make the resolvers configurable in the application itself, so users could get around a bunky NS?