I have created an API endpoint, I have a user of that endpoint requesting from servers at stackoverflow.com. I want to verify that the request was made from stackoverflow.com servers. One way I could verify it came from stackoverflow.com is to ask the developer to sign the request with their let's encrypt domain private key. I can then use their public key to decrypt the message.
I'm not totally sure I can decrypt the privately encrypted message with their public key but even if I could, I would like to avoid having the developer do any special type of encryption. Could I use TLS to verify the origin domain?
TLS supports client authentication, also called '2-way' or 'mutual' authentication. (SSL3 also did, but you should not be using SSL3.) See e.g. TLS1.2
'updated' for ECC and TLS1.3.
How to use this depends on the software (typically library or middleware) being used for TLS, which you didn't indicate; it is even possible some TLS stack doesn't support it at all, though I've never heard of any. Some stacks or use-cases allow client auth to be invoked without any code change, and others with only minimal or localized code change.
Some details that may or may not matter:
this does not sign the request. It authenticates the TLS connection (to be exact, it normally signs a transcript of the handshake) and then the data transferred over the connection is MACed (as well as encrypted) using keys created (and thus authenticated) by the handshake. This provides authentication but not nonrepudiation for the data; you the receiver can reliably determine it came from the sender, but you can't reliably prove this to a third party. For the closely related case of 'proving' the server, see the numerous crossdupes linked at https://security.stackexchange.com/questions/205074/is-it-possible-to-save-a-verifiable-log-of-a-tls-session .
this authenticates the data was sent by the identified client; it says nothing about the origin which as Sam Jason points out is often different.
the client is not necessarily identified by a domain name; it can be a person, organization, or something else. However, many CAs issue a single cert for both TLS server auth and client auth (look at the ExtendedKeyUsage extension in your own or any sample cert(s) to see) and in that case with few exceptions the subject is identified by a domain name or name(s) or at least wildcard(s).
I'm pretty sure you should be using some sort of API key or maybe something similar to how twilio signs its requests
one reason for these patterns is that it's common for HTTP requests to be proxied, with static requests handled by something other than the code/application server. therefore the TLS connection would have been terminated at the proxy server, and the actual application code wouldn't be able to easily see anything about the TLS connection used by the remote server
Related
In case of HTTP requests like HEAD / GET / POST etc, which information of client is received by the server?
I know some of the info includes client IP, which can be used to block a user in case of, lets say, too many requests.
Another information of use would be user-agent, which is different for browsers, scripts, curl, postman etc. (Of course client can change default by setting request headers, but thats alright)
I want to know which other parameters can be used to identify a client (or define some properties)? Does the server get the mac address somehow?
So, is there a possibility that just by the request, it is identifiable that this request is being done by a "bot" (python or java code, eg.) vs a genuine user?
Assume there is no token or any such secret shared between client-server so there is no session...each subsequent request is independent.
The technique you are describing is generally called fingerprinting - the article covers properties and techniques. Depending on the use there are many criticisms of it, as it bypasses a users intention of being anonymous. In all cases it is a statistical technique - like most analytics.
Putting your domain behind a service like cloudflare might help prevent some of those bots from hitting your server. Other than a service like that, setting up a reCAPTCHA would block bots from accessing any pages behind it.
It would be hard to detect bots using solely HTTP because they can send you whatever headers they want. These services use other techniques to try and detect and filter out the bots, while allowing real users to access the site.
I don't think you can rely on any HTTP request header, because a client might not send it to the server, and/or there might be proxies between the client and the server that strip or alter the request headers.
If you just want to associate a unique ID to an HTTP request, you could generate an ID on your backend. For example, the JavaScript framework Hapi.js computes a request ID using this code:
new Date() + '-' + process.pid + '-' + Math.floor(Math.random() * 0x10000)
You might not even need to generate an ID manually. For example, if your app is on AWS and there is an Application Load Balancer in front of your backend, the incoming request will have the custom header X-Amzn-Trace-Id.
As for distinguishing between requests made by human clients and bots, I think you could adopt a "time trap" approach like the one described in this answer about honeypots for spambots.
HTTP request headers are not a good way to track users that use your site. This is because users can edit these headers and the server has no way to verify their authenticy. Also, in the case of the IP Address, it can change during a session if, for example, a user is on a mobile network.
My suggestion is using a cookie with a unique, random id, given to the user the first time they land on a page of your site. Keep in mind that the user can still edit/remove this cookie, so it isn't a perfect method. If you can force the user to login, then you could track the user with their session token.
I know you can input the Username, Password and Domain for a htpasswd protected URL using the following schema :
http://$username:$password#$Domain
eg :
http://sam:1234#example.com
But would this work for an HTTPS Domain ? And if so would the Username and Password be encrypted in transit ?
There are a few issues with HTTP Basic Auth:
The password is sent over the wire in base64 encoding (which can be easily converted to plaintext).
The password is sent repeatedly, for each request. (Larger attack window)
The password is cached by the webbrowser, at a minimum for the length of the window / process. (Can be silently reused by any other request to the server, e.g. CSRF).
The password may be stored permanently in the browser, if the user requests. (Same as previous point, in addition might be stolen by another user on a shared machine).
Of those, using SSL only solves the first. And even with that, SSL only protects until the webserver - any internal routing, server logging, etc, will see the plaintext password.
So, as with anything it's important to look at the whole picture.
Does HTTPS protect the password in transit? Yes.
Is that enough? Usually, no. (I want to say, always no - but it really depends on what your site is and how secure it needs to be.)
Complete credit to below answer (copied word to word)
https://security.stackexchange.com/questions/988/is-basic-auth-secure-if-done-over-https
We will be connecting to a web service over https. This will be triggered in the background if a user performs a certain action.
The link will be between the server and the web-service though - the user will not be aware of it.
As there is no user there to see the certificate come up with an error - because this is server to server - how can we mitigate a man in the middle attack between the two servers? What would happen in the code if one were tried and the certificate failed?
We are using ASP.NET.
Its up to you - you can specify your own validation behavior via
ServicePointManager.ServerCertificateValidationCallback
Reference: https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.servercertificatevalidationcallback(v=vs.110).aspx
Default implementation is very reasonable - throw an exception when validation fails.
If you are using a well-written HTTP client library to consume your web service, the calls will fail if the certificate validation fails. Correct http client library will do full validation including making sure that hostname it connects to matches the subject name of the certificate, that CA is correct etc. I would hope that .NET's implementation is correct in that regard, but you should definitely test and validate that default behavior is correct.
Our company has a SOAP-Based WebService written in C# ASP.Net 4.
The original developer left a while ago and unfortunately left it in an extremely messy state. Maintaining it is an absolute nightmare, so I am in the process of refactoring.
Along the way I have noticed that he had coded the responses to include the original requests including username and passwords for accessing the service (which are the first 2 parameters to all web methods that exist in this API)
Since I am looking into re-doing the infrastructure I wanted to ask if anyone knows whether the practice of returning the original request is normal?
To me it seems like a security issue waiting to happen?
Do other people send back the original request minus the security information?
Note: I am aware that this is legacy technology but unfortunately I am not in a position to re-write the whole thing from the ground up :-(
Thanks,
Gary.
Example (Soap XML omitted for brevity):
REQUEST:
POST our-web-service/Products.asmx/Details
username=TEST_USER&password=TEST_PASSWORD&productId=12345
RESPONSE:
<Response IsValid="True">
<Product id="12345">
<Name>Test Product 1</Name>
<Category>General</Category>
....
</Product>
</Product>
<OriginalRequest>
<Username>TEST_USERNAME</Username>
<Password>TEST_PASSSWORD</Password>
<ProductId>12345</ProductId>
</OriginalRequest>
</Response>
We are not aware of your API funcionality.
So,There may be a case in which your application need original request data to perform some process in stateless environment. I mean, sender or requester don't want to hold on request message untill it get processed and come back in your API.
But exposing username and password like this is highly unlikely.
Even your production runs on https, It doesn't mean it can't be tampered.
If you are using (exchanging) SSL certificates at both client and server end then it's fine.
There is one more option in https at server side, "without client authentication". In this type of transport a self signed certificate is used by client to send and receive http request and response. This can be tampered.
So make sure that you are exchanging SSL certificate. If your client is vey concerned about security.
And there is no need to send password astleast. For one time, you can send username but sending password like this will be blunder for the securit of your application.
If on a login screen user submits a form with their username and password, the password is sent in plain text (even with POST, correct me if I am wrong).
What is the right way to protect the user and his password against the third party who might be eavesdropping on the communication data?
I am aware that HTTPS is a solution to the problem, but is there any way to ensure at least some level of security using the standard HTTP protocol (POST request)? (perhaps using javascript in some way)
What I was about was a page - that is a PHP-generated login page, which is of course sent to users in HTTP GET request as an HTML file. There is no (#Jeremy Powel) connection established between the server and the client so I can't create such a handshaking protocol. And I want the complete process to be transparent to the user - he wants to submit a password, not deal with cryptography.
Using HTTP with SSL will make your life much easier and you can rest at ease. Very smart people (smarter than me at least!) have scrutinized this method of confidential communication for years.
Secure authentication is a broad topic. In a nutshell, as #jeremy-powell mentioned, always favour sending credentials over HTTPS instead of HTTP. It will take away a lot of security related headaches.
TSL/SSL certificates are pretty cheap these days. In fact if you don't want to spend money at all there is a free letsencrypt.org - automated Certificate Authority.
You can go one step further and use caddyserver.com which calls letsencrypt in the background.
Now, once we got HTTPS out of the way...
You shouldn't send login and password via POST payload or GET parameters. Use an Authorization header (Basic access authentication scheme) instead, which is constructed as follows:
The username and password are combined into a string separated by a
colon, e.g.: username:password
The resulting string is encoded using
the RFC2045-MIME variant of Base64, except not limited to 76
char/line.
The authorization method and a space i.e. "Basic " is then
put before the encoded string.
source: Wikipedia: Authorization header
It might seem a bit complicated, but it is not.
There are plenty good libraries out there that will provide this functionality for you out of the box.
There are a few good reasons you should use an Authorization header
It is a standard
It is simple (after you learn how to use them)
It will allow you to login at the URL level, like this: https://user:password#your.domain.com/login (Chrome, for example will automatically convert it into Authorization header)
IMPORTANT:
As pointed out by #zaph in his comment below, sending sensitive info as GET query is not good idea as it will most likely end up in server logs.
The Authorization header value is traditionally a base64-encoded username/password. Base64 is not encryption. The original value can be obtained by an on-path attacked using a simple base64-decode.
You can use a challenge response scheme. Say the client and server both know a secret S. Then the server can be sure that the client knows the password (without giving it away) by:
Server sends a random number, R, to client.
Client sends H(R,S) back to the server (where H is a cryptographic hash function, like SHA-256)
Server computes H(R,S) and compares it to the client's response. If they match, the server knows the client knows the password.
Edit:
There is an issue here with the freshness of R and the fact that HTTP is stateless. This can be handled by having the server create a secret, call it Q, that only the server knows. Then the protocol goes like this:
Server generates random number R. It then sends to the client H(R,Q) (which cannot be forged by the client).
Client sends R, H(R,Q), and computes H(R,S) and sends all of it back to the server (where H is a cryptographic hash function, like SHA-256)
Server computes H(R,S) and compares it to the client's response. Then it takes R and computes (again) H(R,Q). If the client's version of H(R,Q) and H(R,S) match the server's re-computation, the server deems the client authenticated.
To note, since H(R,Q) cannot be forged by the client, H(R,Q) acts as a cookie (and could therefore be implemented actually as a cookie).
Another Edit:
The previous edit to the protocol is incorrect as anyone who has observed H(R,Q) seems to be able to replay it with the correct hash. The server has to remember which R's are no longer fresh. I'm CW'ing this answer so you guys can edit away at this and work out something good.
If your webhost allows it, or you will need to deal with sensitive data, then use HTTPS, period. (It's often required by the law afaik).
Otherwise if you want to do something over HTTP. I would do something like this.
The server embeds its public key into the login page.
The client populates the login form and clicks submit.
An AJAX request gets the current timestamp from the server.
Client side script concatenates the credentials, the timestamp and a salt (hashed from analog data eg. mouse movements, key press events), encrypts it using the public key.
Submits the resulting hash.
Server decrypts the hash
Checks if the timestamp is recent enough (allows a short 5-10 second window only). Rejects the login if the timestamp is too old.
Stores the hash for 20 seconds. Rejects the same hash for login during this interval.
Authenticates the user.
So this way the password is protected and the same authentication hash cannot be replayed.
About the security of the session token. That's a bit harder. But it's possible to make reusing a stolen session token a bit harder.
The server sets an extra session cookie which contains a random string.
The browser sends back this cookie on the next request.
The server checks the value in the cookie, if it's different then it destroys the session, otherwise all is okay.
The server sets the cookie again with different text.
So if the session token got stolen, and a request is sent up by someone else, then on the original user's next request the session will be destroyed. So if the user actively browsing the site, clicking on links often, then the thief won't go far with the stolen token. This scheme can be fortified by requiring another authentication for the sensitive operations (like account deletion).
EDIT: Please note this doesn't prevent MITM attacks if the attacker sets up their own page with a different public key and proxies requests to the server.
To protect against this the public key must be pinned in the browser's local storage or within the app to detect these kind of tricks.
About the implementation: RSA is probably to most known algorithm, but it's quite slow for long keys. I don't know how fast a PHP or Javascript implementation of would be. But probably there are a faster algorithms.
You can use SRP to use secure passwords over an insecure channel. The advantage is that even if an attacker sniffs the traffic, or compromises the server, they can't use the passwords on a different server. https://github.com/alax/jsrp is a javascript library that supports secure passwords over HTTP in the browser, or server side (via node).
I would use a server-side and client-side Diffie-Hellman key exchange system with AJAX or multiple form submits(I recommend the former), although I don't see any good implementations thereof on the internet. Remember that a JS library can always be corrupted or changed by MITM. Local storage can be used to help combat this, to an extent.
HTTPS is so powerful because it uses asymmetric cryptography. This type of cryptography not only allows you to create an encrypted tunnel but you can verify that you are talking to the right person, and not a hacker.
Here is Java source code which uses the asymmetric cipher RSA (used by PGP) to communicate:
http://www.hushmail.com/services/downloads/
you can use ssl for your host there is free project for ssl like letsencrypt
https://letsencrypt.org/
Using https sounds best option here (certificates are not that expensive nowadays). However if http is a requirement, you may use some encription - encript it on server side and decript in users browser (send key separately).
We have used that while implementing safevia.net - encription is done on clients (sender/receiver) sides, so users data are not available on network nor server layer.