I'm sending a GET request with firefox. Only headers (no cookies within) and query parameters is sent. The website answers with 200.
When I'm trying to replay the request with python.
Even though I'm setting the exact same headers, I'm getting a 403.
So my question is, in the HTTP protocol, if I send a GET request to a domain with the same headers and cookies than another one, what else can make difference between these 2 requests ?
Put aside request body which is not applicable here. I'm also using the same IP.
Related
Due to some decisions in the past, our custom proxy is unable to accept HEAD requests. Changing it to accept HEAD will be quite a lot of work. One approach was to use the code path used by GET when a HEAD request comes. But this would mean the response gets sent back as well. Is it ok to send back a response for a HEAD request? Or does it really depend on the upstream HTTP client? I feel the request body will anyway be ignored so shouldn't be a problem
I'm playing a bit with HTTP requests with fiddler.
Basically the site is my router interface which asks for a password.
The password is then encrypted (with a function i have, branded MD5 of some kind) and passed to the server in the body of the POST request.
Here what i've done:
With fiddler i sniffed the browser GET request (got a redirection 302)
With fiddler i sniffed the server response
With fiddler i sniffed the browser POST request (after typing the password)
Browser successfully logged in.
then:
With fiddler composer i performed a GET request equal to the one sniffed. (got a redirection 302)
With fiddler i sniffed the server response (each time the site provides to the client a different Set-Cookie value which is used in the brand MD5 function for antiXRSF attacks).
With fiddler composer i reproduced the sniffed POST request with a different body content due to the new Set-Cookie value.
The body of the post request is indeed correct because it is calculated by the very same function used by the browser.
What came to my mind now:
I'm using the wrong Set-Cookie value --> not possible since if i try to guess the body content of a browser request with the available parameters, the guess turns out to be correct.
The redirection performed by fiddler is done without the Set-Cookie or with a different one --> i saw i can decide not to follow a redirect, is there a way to decide what to pass in the header during a redirection? I'll test more directly on the redirected url.
Regards,
The redirection performed by fiddler is done without the Set-Cookie
Correct. Fiddler's Composer does not have a cookie jar. If a call returns a cookie via Set-Cookie on a redirect, that cookie is not added to a Cookie header when the redirected request is sent.
I'm trying to retrieve 3 response headers (Rails Devise Auth Headers: uid, client, access-token) in every request to a Rails Server.
Using Postman (http client) it's working.
With OkHttp (java http client) the headers just don't show up in the client (i've checked using Wireshark).
When i'm in debug mode it just work...
The additional headers with postman are due to postman sending an Origin header and the server is replying with CORS headers, i.e. Access-Control-.... These headers are send within the normal HTTP header, i.e. not after the response.
But these access control headers are only relevant when the access is done from a browser because they control the cross origin behavior of XHR. Since you are not inside a browser they should be irrelevant for what you are doing. What is relevant are the body of the response and some of the other headers and here you'll find no differences. Also irrelevant should be if multiple requests are send within the same TCP connection (HTTP keep-alive done by postman) or with multiple connections (OkHttp) because each request is independent from the other and using the same TCP connection is only a performance optimization.
If you really want to get these special headers you should add an Origin header within you OkHttp request. See the OkHttp examples on how to add your own headers. But like I said: these access control headers should be irrelevant for the real task and there should be no need to get to these headers.
There is a property "config.batch_request_buffer_throttle" in the file "config/initializers/devise_token_auth.rb" of the Rails Project. We changed it from 5 seconds to 0 seconds.
It is a property to keep the current token available for that amount of time to the following requests.
As the original documentation: "Sometimes it's necessary to make several requests to the API at the same time. In this case, each request in the batch will need to share the same auth token. This setting determines how far apart the requests can be while still using the same auth token."
So when we did the request using Postman or in Java Debug the 5 seconds was running allowing Devise to generate new tokens then retrieve them to the client.
It's my understanding that caching is one of the main utilities of a proxy server. I'm currently trying to develop a simple one and I would like to know exactly how caching works.
Intuitively I think that it's basically an association between a request and a response. For example: for the following request: "GET google.com" you have the following response: "HTTP/1.0 200 OK..."
That way, whenever the proxy server receives a request for that URL he can reply with the cached response (I'm not really worried right now about when to serve the cached response and when to actually send the request to the real destination).
What I don't understand is how to establish the association between a request and a response since the HTTP response doesn't have any field saying "hey this is the response you get when you request the X URL" (or does it?).
Should I get this information by analyzing the underlying protocols? If so, how?
Your cache proxy server is already putted into play when a request arrives. Therefore you have the requested resource URL. Then you look in your cache and try to find the cached resource for the requested resource URL, if you cannot find the resource in your cache (or the cache is outdated), you fetch the data from the source. Keep in Mind, that you have to invalidate the cached resource if you receive a PUT, POST or DELETE request.
I cannot find any RFC or Standard of HTTP client behavior in case it gets HTTP response with an error 4xx. I know the 401, 407 are the examples when the HTTP headers are parsed, but...
I have the concrete problem for OPTIONS method (HTTP1.1). The server responses 401 Unauthorized, so client tries to authenticate and re-sends the request with an authentication. After that the response has the error 404 Not Found and HTTP header is filled with Set-Cookie HTTP Header. The client use Apache Java HTTPClient/HTTPComponents, which ignores HTTP headers in case of an error in the response.
Should this HTTP Header be accepted by the client? I believe it should not be, but I cannot find the supportive quotation in the RFC.
RFC 2616 does not specify that any headers should be ignored, not for 404 responses and not for 4xx responses in general either.
RFC 6265 allows clients to ignore Set-Cookie headers, but does not specify situations where that might happen; a single example is given, that does not cover your case:
the user agent might wish to block responses to "third-party" requests
from setting cookies
In your case, since your server seems to use HTTP basic access authentication, it does not seem to concern the Set-Cookie header. In HTTP basic authentication, the Authorization header is sent by the client with every request, so there should be no need to keep state in a cookie.
It is not clear from your question if you have a very specific HTTP server that you're talking to, or if you are implementing a general HTTP client that is supposed to work with whatever server you throw it at. If you have such a specific case that the HTTP server you work with sends state with 404 responses, and you're required to honor that state in order to communicate with the server, and you have no control over the server, then it does not matter what the standard says; you will honor the state sent, or you will not be able to talk to the server.
If, on the other hand, you're implementing a general client and need it to work regardless of the remote server, then your best bet is to stick to RFC 1958:
Be strict when sending and tolerant when receiving.
Implementations must follow specifications precisely when sending to
the network, and tolerate faulty input from the network. When in
doubt, discard faulty input silently, without returning an error
message unless this is required by the specification.
Which, to me, would mean that you should honor the full response received, regardless of the status code, unless you have an objective reason making it impossible for you to do so. I don't see a reason to ignore the state, even if it violates the standard (or in this case, your personal perception of the standard, since it does not say anything about accepting or ignoring the state).
Update: RFC 2617 (HTTP Authentication) states:
A client SHOULD assume that all paths at or deeper than the depth of
the last symbolic element in the path field of the Request-URI also
are within the protection space specified by the Basic realm value of
the current challenge. A client MAY preemptively send the
corresponding Authorization header with requests for resources in
that space without receipt of another challenge from the server.
It is highly inconsistent if the server expects HTTP authentication for one URL, but does not honor it for URLs beneath it, requiring a separate cookie-based authentication for them. If anything should be changed in your server implementation, it should be to harmonize the authentication scheme for all resources.