I am trying to reconstruct/imitate a browser request from my spring boot backend.
I do a GET request to the server and store all the response headers with 'Set-Cookie', which are three in total.
When I delete my cookies and re-fire the request in browser though, ten cookies are added to my browser cookies instead of three. I went through the network traffic recording of Chrome and could not find any responses which set cookies despite the first one with three.
Where do the other seven cookies come from? How can I save those seven missing cookies to my backend in order to make a valid follow up request?
For reference: https://www.dhl.de/de/privatkunden/pakete-empfangen/verfolgen.html
Cookies in browser request
Thank you so much in advance, any resource means a lot to me.
Javascript, document.cookies.
(I need more characters to hit the minumum for an answer)
I'm using playframework and nginx. playframework may add following cookies to http response: PLAY_SESSION, PLAY_FLASH, PLAY_LANG.
I want to make sure that only above cookies (PLAY_*) are allowed in nginx level. If there are other cookies (let's say they're added accidentally) they should be removed by nginx.
How can I allow only predefined cookies in http response in nginx?
PS: If it's not possible to solve this issue in nginx, I need to fix by using playframework.
How cookies work?
First, let's establish what's cookies — they're little pieces of "sticky" hidden information that lets you keep state on your web-site for a given User-Agent. These cookies are often used for tracking users, keeping session and storing minor preference information for the site.
Set-Cookie HTTP response header (from server to client)
Cookies can be set by the server through the Set-Cookie response header (with a separate header for each cookie), or, after the page has already been transferred from the server to the client, through JavaScript.
Note that setting cookies is a pretty complex job — they have expiration dates, http/https settings, path etc — hence the apparent necessity to use a separate Set-Cookie header for each cookie.
This requirement to have a separate header is usually not an issue, since cookies aren't supposed to be modified all that often, as they usually store very minimal information, like a session identifier, with the heavy-duty information being stored in an associated database on the server.
Cookie HTTP request header (from client to server)
Regardless how they were first set, cookies would then included in eligible subsequent requests to the server by the client, using the Cookie request header, with a whole list of eligible cookies in one header.
Note that, as such, these cookies that are sent by the client back to the server is a simple list of name and attribute pairs, without any extra information about the underlying cookies that store these attributes on the client side (e.g., the expiration dates, http/https setting and paths are saved by the client internally, but without being revealed in subsequent requests to the server).
This conciseness of the Cookie request header field is important, because, once set, eligible cookies will be subsequently included in all forthcoming requests for all resources with the eligible scheme / domain / path combination.
Caching issues with cookies.
The normal issue of using cookies, especially in the context of acceleration and nginx, is that:
cookies invalidate the cache by default (e.g., unless you use proxy_ignore_headers Set-Cookie;),
or, if you do sloppy configuration, cookies could possibly spoil your cache
e.g., through the client being able to pass cookies to the upstream in the absence of proxy_set_header Cookie "";,
or, through the server insisting on setting a cookie through the absence of proxy_hide_header Set-Cookie;.
How nginx handles cookies?
Cookie from the client
Note that nginx does support looking through the cookies that the client sends to it (in the Cookie request header) through the $cookie_name scheme.
If you want to limit the client to only be sending certain cookies, you could easily re-construct the Cookie header based on these variables, and send only whichever ones you want to the upstream (using proxy_set_header as above).
Or, you could even make decisions based on the cookie to decide which upstream to send the request to, or to have a per-user/per-session proxy_cache_key, or make access control decisions based on the cookies.
Set-Cookie from the backend
As for the upstream sending back the cookies, you can, of course, decide to block it all to considerably improve the caching characteristics (if applicable to your application, or parts thereof), or fix up the domain and/or path with proxy_cookie_domain and/or proxy_cookie_path, respectively.
Otherwise, it's generally too late to make any other routing decision — the request has already been processed by the selected upstream server, and the response is ready to be served — so, naturally, there doesn't seem to be a way to look into these individual Set-Cookie cookies through normal means in nginx (unless you want to go third-party modules, or lua, or perl), since it'd already be too late to make any important routing decisions for a completed request.
Basically, these Set-Cookie cookies have more to do with the content than with the way it is served or routed, so, it doesn't seem appropriate to have integrated functionality to look into them through nginx.
(If you do need to make routing decisions after the completion of the request, then nginx does support X-Accel-Redirect, as well as some other special headers.)
If your issue is security, then, as I've pointed out above, the upstream developer can already use JavaScript to set ANY extra cookies however they want, so, effectively, trying to use nginx to limit some, but not all, Set-Cookie responses from the server is kind of a pointless endeavour in the real world (as there is hardly any difference between the cookies set through JavaScript compared to Set-Cookie).
In summary:
you can easily examine and reconstruct the Cookie header sent by the client to the server before passing it over to the backend, and only include the sanctioned cookies in the request to upstream backend,
but, unless you want to use lua/perl, or have your own nginx module (as well as possibly quarantine the JavaScript from the pages you serve), then you cannot pass only certain Set-Cookie headers back from the upstream backend to the client with a stock nginx.conf — with the Set-Cookie headers, it's an all-or-nothing situation, and there doesn't seem to be a good-enough use-case for a distinct approach.
For an Nginx solution it might be worth asking over at serverfault. Here is a potential solution via Play Framework.
package filters
import javax.inject._
import play.api.mvc._
import scala.concurrent.ExecutionContext
#Singleton
class ExampleFilter #Inject()(implicit ec: ExecutionContext) extends EssentialFilter {
override def apply(next: EssentialAction) = EssentialAction { request =>
next(request).map { result =>
val cookieWhitelist = List("PLAY_SESSION", "PLAY_FLASH", "PLAY_LANG")
val allCookies = result.newCookies.map(c => DiscardingCookie(c.name))
val onlyWhitelistedCookies = result.newCookies.filter(c => cookieWhitelist.contains(c.name))
result.discardingCookies(allCookies: _*).withCookies(onlyWhitelistedCookies: _*)
}
}
}
This solution utilizes Filters and Result manipulation. Do test for adverse effects on performance.
MDN says, when the credentials like cookies, authorisation header or TLS client certificates has to be exchanged between sites Access-Control-Allow-Crendentials has to be set to true.
Consider two sites A - https://example1.xyz.com and another one is B- https://example2.xyz.com. Now I have to make a http Get request from A to B. When I request B from A I am getting,
"No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://example1.xyz.com' is therefore not allowed
access."
So, I'm adding the following response headers in B
response.setHeader("Access-Control-Allow-Origin", request.getHeader("origin"));
This resolves the same origin error and I'm able to request to B. When and why should I set
response.setHeader("Access-Control-Allow-Credentials", "true");
When I googled to resolve this same-origin error, most of them recommended using both headers. I'm not clear about using the second one Access-Control-Allow-Credentials.
When should I use both?
Why should I set Access-Control-Allow-Origin to origin obtained from request header rather than wildcard *?
Please quote me an example to understand it better.
Allow-Credentials would be needed if you want the request to also be able to send cookies. If you needed to authorize the incoming request, based off a session ID cookie would be a common reason.
Setting a wildcard allows any site to make requests to your endpoint. Setting allow to origin is common if the request matches a whitelist you've defined. Some browsers will cache the allow response, and if you requested the same content from another domain as well, this could cause the request to be denied.
Setting Access-Control-Allow-Credentials: true actually has two effects:
Causes the browser to actually allow your frontend JavaScript code to access the response if credentials are included
Causes any Set-Cookie response header to actually have the effect of setting a cookie (the Set-Cookie response header is otherwise ignored)
Those effects combine with the effect that setting XMLHttpRequest.withCredentials or credentials: 'include' (Fetch API) have of causing credentials (HTTP cookies, TLS client certificates, and authentication entries) to actually be included as part of the request.
https://fetch.spec.whatwg.org/#example-cors-with-credentials has a good example.
Why should I set Access-Control-Allow-Origin to origin obtained from request header rather than wildcard *?
You shouldn’t unless you’re very certain what you’re doing.
It’s actually safe to do if:
The resource for which you’re setting the response headers that way is a public site or API endpoint intended to be accessible by everyone, and
You’re just not setting cookies that could enable an attacker to get access to sensitive information or confidential data.
For example, if your server code is just setting cookies just for the purpose of saving application state or session state as a convenience to your users, then there’s no risk in taking the value of the Origin request header and reflecting/echoing it back in the Access-Control-Allow-Origin value while also sending the Access-Control-Allow-Credentials: true response header.
On the other hand, if the cookies you’re setting expose sensitive information or confidential data, then unless you’re really certain you have things otherwise locked down (somehow…) you really want to avoid reflecting the Origin back in the Access-Control-Allow-Origin value (without checking it on the server side) while also sending Access-Control-Allow-Credentials: true.
If you do that, you’re potentially exposing sensitive information or confidential data in way that could allow malicious attackers to get to it. For an explanation of the risks, read the following:
https://web-in-security.blogspot.jp/2017/07/cors-misconfigurations-on-large-scale.html
http://blog.portswigger.net/2016/10/exploiting-cors-misconfigurations-for.html
And if the resource you’re sending the CORS headers for is not a public site or API endpoint intended to be accessible by everyone but is instead inside an intranet or otherwise behind some IP-address-restricted firewall, then you definitely really want to avoid combining Access-Control-Allow-Origin-reflects-Origin and Access-Control-Allow-Credentials: true. (In the intranet case you almost always want to only be allowing specific hardcoded/whitelisted origins.)
Ive made some configurations to (finally) have my cookies set on HTTP only.
"Seem" to work.
Ive tried them with postman and I have the following:
When I hit the login page:
On the cookies section, my cookie with name JSESSIONID appears to be HTTP only (it has the check)
When I enter to the logged area , the same result...
The headers dont give me more details.
Then,
I check it with google chrome. I open the developers toolbar.
I load the login page.
At the headers on the response headers I get
Set-Cookie: JSESSIONID=434434..... HttpOnly
So, its fine (I guess).
Then I reload the page (or sign in).
Then the problem:
No response headers received.
The Request Headers brings my cookie (with the same ID at then the previous one) without the httponly, host info or any other cookie value I set before.
At the cookies tab I get Request Cookies only and no Response cookie.
And the request cookie is non http-only
At my resources tab, the Cookie is there, as HTTP only and with the previous values I set.
My question now is... Is it a really http-only cookie? Or my configuration is not properly set?
Should I always get the response cookie or the request cookie should be always http-only (In case I am trying to set it as http-only) or is this behavior normal (or at least accepted) ?
When I try to print my cookie with Javascript at both scenarios I get a null as response (what makes me think then it is correct).
Ideas?
Client doesn't send cookie attributes other than name and value back to server.
See also RFC6265 section 4.2.2 (emphasis mine).
4.2.2. Semantics
Each cookie-pair represents a cookie stored by the user agent. The
cookie-pair contains the cookie-name and cookie-value the user agent
received in the Set-Cookie header.
Notice that the cookie attributes are not returned. In particular,
the server cannot determine from the Cookie header alone when a
cookie will expire, for which hosts the cookie is valid, for which
paths the cookie is valid, or whether the cookie was set with the
Secure or HttpOnly attributes.
Everything's behaving as specified.
My goal is that I want to log into a site with some java code and do some work when logged in. (in order to write some java cooking handling I first need to understand how this all actually works)
The problem is that I can't quite figure out how to manage the cookie session.
Here's what I've observed when using chrome's dev. tools:
1) on first request for the site's url I send no cookies and I get some in response:
Set-Cookie:PHPSESSID=la2ek8vq9bbu0rjngl2o67aop6; path=/; domain=.mtel.bg; HttpOnly
Set-Cookie:PHPSESSID=d32cj0v5j4mj4nt43jbhb9hbc5; path=/; domain=.mtel.bg; HttpOnly
2) on moving on log page I now send (on HTTP GET):
Cookie:PHPSESSID=d32cj0v5j4mj4nt43jbhb9hbc5;
__utma=209782857.1075318979.1349352741.1349352741.1349352741.1;
__utmb=209782857.1.10.1349352741;
__utmc=209782857;
__utmz=209782857.1349352741.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
__atuvc=1%7C40
and indeed when I check my stored cookies (after my first get request for the site's main url) in chrome's cookie tab - they exist in the way they're passed on the next get/post request.
Can you explain what's really happening after my 1st cookies are received and why such name/value pairs are stored?
There's a Javascript from Google Analytics on the page which sets some cookies.