I want to post a http document that is password protected.
It has a username and password login page before you can access the actual document.
I tried doing this
curl
-u username:password
"http://localhost:8983/solr/update/extract?literal.id=doc900&commit=true"
-F stream.url=http://somewebsite.com/docs/DOC2609
but it just indexes the login page only.
My guess is that the authentication method this page of yours uses is not one of the HTTP authentication methods that curl supports out-of-the-box.
The HTTP authentication is basic -- some might say trivial -- and it is fundamentally insecure in its most basic form (literally, 'basic' mode HTTP authentication sends the users' password over the wire in plain text, as one of the request headers). I only ever see it used in conjunction with HTTPS, and even that's pretty rare these days.
HTTP authentication is generally eschewed in favor of either an HTTP PUT or POST request over HTTPS, usually fortified with some kind of cross-site request forgery (CSRF) prevention measure, or an OAUTH2 strategy (which typically defers the issue to the authentication broker's implementation).
Your problem is, curl doesn't know about any of this. If you program in python, you may find the requests package helpful (and more modern than curl in its design to boot). Solr won't care either way, if you are calling its HTTP API properly.
Related
I am using Backbone.js and it communicates with a stateless rest API. Some calls require authentication, through HTTP basic.
What I don't understand is, somehow I have to authenticate each request, how could I do this securely? My first thought was to have a cookie, store the username and password but this would be vulnerable?
Can this be done securely?
There are two themes to this question. One is about security and one seems to be about REST rules.
The way to do authentication securely, is to pass that data through an SSL connection. It's the only way to securely transfer data over the wire.
With regards to sending authentication using basic auth over each request (REST), not many people I know do this in reality.
There's always a big long discussion on how much security is enough security and it really depends on your application and what the purpose is. I know this isn't the definitive answer you might be looking for but I'll just give you my take and how I'm going about dealing with the issues you mention.
With RESTful apps, the story is one should authenticate each request but in real practice I find this is more a "guide" than a hard rule. Rare is the fully RESTful application that follows all the rules. I use an encrypted cookie to store the user session data with a standard authentication flow that happens once and expires in a week. Data transfers happen through SSL to prevent MITM attacks and a modified Backbone sync sends a CSRF token along with each POST, PUT, DELETE to prevent cross site request forgeries. Probably "good enough" for the social app that I am working on. Maybe not if you're doing bank wire transfers and stuff. Hope this sort of gives you a point of reference in judging what you might want to do.
Is https://github.com/fiznool/backbone.basicauth something you'd find useful?
This plugin enables access to remote resources which are protected by HTTP Basic Authentication through your Backbone Models and Collections.
How does it work?
A resource protected with HTTP Basic Authentication requires the following HTTP header to be set on every request:
Authorization: Basic
The access token is formed by taking the username and password, concatenating together with a : separator and encoding into Base64.
This plugin handles the Base64 encoding and automatically sets the Authorization header on every request which uses Backbone.sync.
I am successfully able to authenticate Facebook and Google accounts using my Oauth2 servlets. I am using state with a timer and a session cookie to try to verify that it is indeed a legitimate Oauth callback.
Is there any benefit if I also examine the HTTP Referer header to ensure that I was redirected from the provider's OAuth page?
If no benefit, could there be a problem if I also examine the HTTP Referer field?
No.
I can simulate any headers I want as a malicious attacker. I can make it look like I'm coming from http://cia.fbi.gov.vpn/uber1337h4x. This is obvious and well known.
Any pages coming from HTTPS do not send a refer header as per RFC2616 sec15:
Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol.
Breaks usability as per RFC2616 sec15:
Because the source of a link might be private information or might reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent.
In short, you are not given greater security. Your security is not in inspecting a vastly insecure transport protocol, it's in the OAuth layer. You also break usability.
Don't do it.
The answer is:
No, you shouldn't use it, and there is NO valuable benefit of doing it.
Authorization Servers are very aware of this also. And here was stated.
From the mailing list of OAuth-WG:
Callback URL pages SHOULD redirect to a trusted page immediately after receiving the authorization code in the URL. This prevents the authorization code from remaining in the browser history, or from inadvertently leaking in a referer header.
If you are worry about CSRF, you SHOULD NOT use the HTTP Referer as a technique to verify the origin of an authorization, that's why the parameter state is (which sound you're using).
If you worry about an specific security concern of the oauth2 protocol, there is a full section inside the draft.
If you worry about other Security Considerations, this is the source.
I suggest you give all your effort implementing all the validations around the param: state.
Edit:
After reading the nuances of the question, you are really answered your own question. The use of cookies (probably HTML5 local storage) for both cases, is the best solution we know so far.
The first nuance is about CSRF and one of the possible countermeasures available is Checking the HTTP Referer header, and this was already addressed in the protocol.
The second nuance, I'm not completly sure, but is probably a case of Extension Grant, this is because it sounds that you may work as an "auth proxy requester", same as SAML oauth2 extension.
Don't verify the HTTP referer; the "state" parameter (which it sounds you're using) is the approach OAuth 2.0 defines to defend against cross-site request forgery (CSRF) attacks.
You may want to have a look at the new O'Reilly book Getting Started with OAuth 2.0 by Ryan Boyd. It describes this and related security considerations.
Plain security was not a concern of the question because the state parameter is being used.
The main concerns I had in mind were:
Whether it is the same browser that my app sent to Facebook that's coming back to present a candidate token?
Whether the agent (browser-like agent) or agents are repeatedly doing OAuth requests and presenting me with bad OAuth tokens that cause my app to repeatedly contact Facebook with bad tokens leading to potentially adverse treatment by Facebook.
The only possible solution to the first problem is to also set a cookie in addition to using state. referer would help if most providers weren't using https.
The second problem has a nuance. The mis-behaving agents need not be directly controlled by a malicious entity. They may be normal users browsers redirected via some indirect means (a popular hijacked website, social engineering).
Because of the nuance there is a chance that the referer header may not be forged. However, https precludes any meaningful benefit.
Cookies definitely help in the second case also because if you are setting cookies in a POST no third-party website can cause them to be set and you cannot be flooded with bad OAuth responses due to hacked websites redirecting users en masse to OAuth you.
This is not a clear answer (or question) but hopefully this shows the nuances behind the question.
I'm just reading Paypal's API documentation, e.g. Adaptive Accounts API
My question: What's the reason/advantage of using (custom?) HTTP Request Headers for authentication instead of "normal" POST/GET (or even COOKIE) variables?
In the mentioned example PayPal uses the following HTTP Request Headers:
X-PAYPAL-SECURITY-USERID
X-PAYPAL-SECURITY-PASSWORD
X-PAYPAL-SECURITY-SIGNATURE
X-PAYPAL-APPLICATION-ID
X-PAYPAL-DEVICE-IPADDRESS
X-PAYPAL-REQUEST-DATA-FORMAT
Why use HTTP headers rather than something in the body of the request?
By keeping your authentication info separate from the payload (the data you are transmitting) you make it easier to handle authentication at an earlier stage in the request pipeline. For example, a gatekeeper server can receive requests and authenticate them by looking only at the headers, and then pass them along to the module/server/class that does parses the request body and does the real work.
If the request fails authentication, it can be rejected before it even gets near the code that deals with the money.
Of course, you can architect your system this way no matter what, but keeping it in the headers means you don't need to parse the request body or even look at it. You also don't need to worry about adjusting Content-Length: if you wanted to modify the headers before passing it along to another server.
Why use custom HTTP headers rather than WWW-Authenticate or Cookie?
I think this is simply because PayPal wants a more robust scheme than either of these can accommodate. WWW-Authenticate only allows for basic (cleartext) and digest (MD5) authentication, and many better schemes have been developed since the spec for these was written. They also don't allow for more than a username and password.
Cookies are technically opaque bits of data that you receive from a server and pass back to it unchanged. Again, they could tell you how to generate the info that you pass along in a Cookie header, but that wouldn't really be following the spec, so at that point, why not just use some custom headers?
The HTTP access authentication process is described in "HTTP Authentication: Basic and Digest Access Authentication", look here.
User agents are advised to take special care in parsing the WWW-Authenticate field value as it might contain more than one challenge, or if more than one WWW-Authenticate header field is provided, the contents of a challenge itself can contain a comma-separated list of authentication parameters.
If the connection is trusted between two parties and SSL is implemented, HTTP Authentication is a simple to implement way to authenticate between two fixed parties. The connection is simply rejected at the traffic level if the authentication fails.
I don't think Paypal uses HTTP Authentication for payments? It's just for you to access API features to build your own admin interface for Paypal accounts?
I have an API endpoint https://www.example.com/api/authentication which takes username and password as input and returns an authentication token.
In terms of passing username and password, I have two options (at least), namely:
HTTP Basic Authentication (which passes credentials as part of HTTP headers)
HTTP POST parameters
I understand that neither method provides encryption (hence the use of HTTPS/SSL). I also understand why using HTTP GET is is a Bad Idea.
Is there any real difference (aside from the fact that basic authentication feels more idiomatic) between the two methods?
The difference is that basic authentication is a well specified challenge/response scheme that all browsers understand and it is the server that starts it by telling a client that it requires (basic) authentication for a realm. This triggers the browser to show a popup to the user to enter a name/password which it then passes in the headers as you described.
In your second example you have to do all that in your own customized way and create your own login form for the user (etc).
If you deduct this process to the single step of passing the username/password from the client to the server I have to agree that there isn't that much difference but basic authentication implies a bit more than just that.
HTTP Basic authentication implementation is the simplest technique for enforcing access controls to web resources because it doesn't require cookies, session identifiers, or login pages; rather, HTTP Basic authentication uses standard fields in the HTTP header, obviating the need for handshakes.
We're trying to test an API that requires HTTP Basic Access Authentication credentials (http://en.wikipedia.org/wiki/Basic_access_authentication) in the request.
Ideally, we could just test the API using a web browser by putting all API parameters in the URL querystring, but we haven't yet found a way to encode the HTTP Basic Access Authentication credentials (username and password) in the querystring.
Does anyone know a way to do this?
Thus far, we've tried:
https://username:password#mydomain.com/
...without success.
username:password#url authentication has been disabled in many browsers for security reasons.
For example in IE:
Internet Explorer does not support user names and passwords in Web site addresses (HTTP or HTTPS URLs)
As far as I know, there is no way to circumvent this if this is blocked. It's possible that this can be turned of in Firefox using a setting in about:config. Or use some other browser that doesn't block it - I don't know which ones do and which don't.
Alternatively, consider building a quick web form that submits the option to a server-side language (e.g. PHP) that makes the request, or use a command line client like wget to send the requests. The latter might even be easiest