My question is similar to the one asked here, but pertains specifically to non-standard request headers.
If my server is expecting non-standard request headers (like say X-Forwarded-For), how will the browser know it's supposed to send it?
X-Forwarded-For is added when the request come into a proxy server or load-balancer that hides the main server address behind that. Also some of those headers are required by some specific servers and the clients must send the data over headers, such as x-api-key or some other names which are defined at the servers.
Related
I'm currently writing a proxy server in nodejs. To proceed, I need to know how to reliably determine the originally intended domain of the client. When a client is configured to use a proxy, is there a universal way that the client sends this information (e.g. one of the two examples below), or is it application specific (e.g. Chrome proxy settings may do it differently to IE proxy settings, which may be different to a configuration for a proxy for an entire Windows machine, etc.)?
An HTTP request to the proxy server could look something like this, which would suffice:
GET /something HTTP/1.1
Host: example.com
...
In this case, the proxy could get the hostname from the 'Host' header, get the path in the first line of the HTTP request, and then have sufficient information.
It could also look something like this, which would suffice:
GET http://example.com/something HTTP/1.1
...
with a FQDN in the URL, in which case the proxy could just retrieve the path of the HTTP request in the first line.
Any information regarding this would be greatly appreciated! Thanks in advance for the help!
I have a web application running on a server (let's say on localhost:8000) behind a reverse proxy on that same server (on myserver.example:80). Because of the way the reverse proxy works, the application sees an incoming request targeted at localhost:8000 and the framework I'm using therefore tries to generate absolute URLs that look like localhost:8000/some/ressource instead of myserver.example/some/ressource.
What would be "the correct way" of generating an absolute URL (namely, determining what hostname to use) from behind a proxy server like that? The specific proxy server, framework and language don't matter, I mean this more in an HTTP sense.
From my initial research:
RFC7230 explicitly says that proxies MUST change the Host header when passing the request along to make it look like the request came from them, so it would look like using Host to determine what hostname to use for the URL, yet in most places where I have looked, the general advice seems to be to configure your reverse proxy to not change the Host header (counter to the spec) when passing the request along.
RFC7230 also says that "request URI reconstruction" should use the following fields in order to find what "authority component" to use, though that seems to also only apply from the point-of-view of the agent that emitted that request, such as the proxy:
Fixed URI authority component from the server or outbound gateway config
The authority component from the request's firsr line if it's a complete URI instead of a path
The Host header if it's present and not empty
The listening address or hostname, alongside with the incoming port number if it's not the default one for the protocol
HTTP 1.0 didn't have a Host header at all, and that header was added for routing purposes, not for URL authority resolution.
There are headers that are made specifically to let proxies to send the old value of Host after routing, such as Via, Forwarded and the unofficial X-Forwarded-Host, which some servers and frameworks will check, but not all, and it's unclear which one should even take priority given how there's 3 of them.
EDIT: I also don't know whether HTTPS would work differently in that regard, given that the headers are part of the encrypted payload and routing has to be performed another way because of this.
In general I find it’s best to set the real host and port explicitly in the application rather than try to guess these from the incoming request.
So for example Jira allows you to set the Base URL through which Jira will be accessed (which may be different to the one that it is actually run as). This means you can have Jira running on port 8080 and have Apache or Nginx in front of it (on the same or even a different server) on port 80 and 443.
In developer.mozilla.org says:
HTTP headers allow the client and the server to pass additional
information with the request or the response
but I don't understand what is the use of that? What is the need to pass additional information with the request or the response?
This is a hard question to answer concisely because of the many different types of HTTP headers and what they do, but here's an attempt at a one-line answer:
HTTP headers allow a client and server to understand each other better, meaning they can communicate more effectively.
So then if you look at individual headers, it becomes clearer why each is needed:
User-Agent header
Sent by the client
Tells the server about the client's setup (browser, OS etc.)
Mostly used to improve client experience, e.g. tailoring responses for mobile devices or dealing with browser compatibility issues
set-cookie header
Sent by the server
Tells the browser to set a cookie
host header
Sent by the client
Specifies the exact domain name of the site the client wants to reach, this is used when a single server hosts multiple websites (a.k.a. virtual hosting)
Is there a good reason to appends ips separated by comma or would the following work?
if (req.restarts == 0) {
if (!req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = client.ip;
}
}
The reason why it's done this way is probably RFC7239
The "for" parameter is used to disclose information about the client
that initiated the request and subsequent proxies in a chain of
proxies.
Client side proxies like squid also may add the originating IP to the X-Forwarded-For header, so if you do it like that and the request already has the header set, varnish would not add what it considers the client.ip as part of the header field.
Update:
the "real IP" would only be the sole value of the header, if the client doesn't already supply one - but proxies like squid do just that. If you always want IP of the client (which might be the IP of a intransparent proxy - hence the creation of XFF header) as the only value, leave out the if (!req.http.x-forwarded-for) condition.
But I really would leave the XFF header alone and use a custom header field for usage in the backend, especially if you plan to use it for resource access control, which is an inherent unsafe thing using the XFF header anyways.
Please also note, that HTTP headers can be set in javascript using setRequestHeader() method.
It would be useful to know what your goal is for recommendations.
Note: Please do read the full question
I'm trying to understand as to why the browsers doesn't show me any X-Forwarded-For header every time a request a page
BTW here are my Request Headers look like
Request URL:http://localhost:3000/users/sign_in
Request Method:GET
Status Code:304 Not Modified
Request Headers:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:max-age=0
Connection:keep-alive
Cookie:undefined=0; poasterapp=s%3A4faaa6b1723e7c6fbd949083532c52598652547b.sNX%2BKOEed2TEQkQN7I7K5lgpoHMRpwerKFvUegMnTVI; _minerva_session=BAh7CUkiD3Nlc3Npb25faWQGOgZFRkkiJWEyM2Q0ZTViMWEyODBiYmFmODEwZTJhZmUwNWU5ODk5BjsAVEkiE3VzZXJfcmV0dXJuX3RvBjsARiIGL0kiCmZsYXNoBjsARm86JUFjdGlvbkRpc3BhdGNoOjpGbGFzaDo6Rmxhc2hIYXNoCToKQHVzZWRvOghTZXQGOgpAaGFzaHsGOgphbGVydFQ6DEBjbG9zZWRGOg1AZmxhc2hlc3sGOwpJIgAGOwBUOglAbm93MEkiEF9jc3JmX3Rva2VuBjsARkkiMUN0Uk56SXU0dUdIdzgwcFZJM3R0L2N4dlovRllTSGRrQ2o1R0VVanhIaVk9BjsARg%3D%3D--6bd89ce9d29e9bdcf56573f9a153dc663a8fe755
Host:localhost:3000
If-None-Match:"785d34e3998360353567fc710af123fb"
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36
Response Headers(Not need but still )
Cache-Control:max-age=0, private, must-revalidate
Connection:close
ETag:"785d34e3998360353567fc710af123fb"
Server:thin 1.5.0 codename Knife
Set-Cookie:_minerva_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJWEyM2Q0ZTViMWEyODBiYmFmODEwZTJhZmUwNWU5ODk5BjsAVEkiE3VzZXJfcmV0dXJuX3RvBjsARiIGL0kiEF9jc3JmX3Rva2VuBjsARkkiMUN0Uk56SXU0dUdIdzgwcFZJM3R0L2N4dlovRllTSGRrQ2o1R0VVanhIaVk9BjsARg%3D%3D--dfb3ce9f5c97463cfcd0229a133654e6cc606d98; path=/; HttpOnly
X-Request-Id:41a6f3062dc8bc36b7b3eae71dc5075d
X-Runtime:89.238257
X-UA-Compatible:IE=Edge
Now as said, I dont see any X-Forwarded-For in request headers
Reading through the wiki pages of X-Forwarded-For make me feel that ,it is something done by caching server(which in my case I believe is my ISP provider) so am I safe to believe that the **X-Forwarded-For** headers is something that is added at the caching server side (ISP provider)
If yes their is this one bugging me about it then that is
why? is the same true (i.e X-Forwarded-For not appearing in the request-headers ) for my server running locally on my machine and I accessing them via browser like http://localhost:3000
X-Forwarded-For is not a standard request header as specified in RFC 2616 Section 5.3 that addresses the protocol standard request headers, which are (as specified in the RFC)
Accept
Accept-Charset
Accept-Encoding
Accept-Language
Authorization
Expect
From
Host
If-Match
If-Modified-Since
If-None-Match
If-Range
If-Unmodified-Since
Max-Forwards
Proxy-Authorization
Range
Referer
TE
User-Agent
In order for your incoming request to have the custom [X-Forwarded-For] header, it must be explicitly added to that request by the calling client. The easiest explanation of why you are not seeing that header is that the client sending the request did not manually add it.
The tricky thing is that the header you are expecting to see is not a header that you should necessarily be expecting to receive unless there is a contract in place between your service and the caller that is apart from the HTTProtocol indicating that you should expect an X-Forwarded-For value to be specified in your request header. As others have already stated, the XFF header is typically set by a proxy server or a load balancer to indicate who the real requester is that is acting through their proxy.
As a service provider, if you demand that an [X-Forwarded-For] header be set in all requests, you must enforce it at a service policy level. If you do not want to service proxy accounts that do not identify who they are shielding with their proxy IP, bounce their request with a 403 Forbidden. If you are in a situation where you must service these requests but depend on this header being set, then you're going to have to come up with a custom process where you could communicate their error back.
Here is what the HTTProtocol has to say about anonymity:
Because the source of a link might be private information or might
reveal an otherwise private information source, it is strongly
recommended that the user be able to select whether or not the
Referer field is sent. For example, a browser client could have a
toggle switch for browsing openly/anonymously, which would
respectively enable/disable the sending of Referer and From
information.
Clients SHOULD NOT include a Referer header field in a (non-secure)
HTTP request if the referring page was transferred with a secure
protocol.
Authors of services which use the HTTP protocol SHOULD NOT use GET
based forms for the submission of sensitive data, because this will
cause this data to be encoded in the Request-URI. Many existing
servers, proxies, and user agents will log the request URI in some
place where it might be visible to third parties. Servers can use
POST-based form submission instead
...
Elaborate user-customized accept header fields sent in every request,
in particular if these include quality values, can be used by servers
as relatively reliable and long-lived user identifiers. Such user
identifiers would allow content providers to do click-trail tracking,
and would allow collaborating content providers to match cross-server
click-trails or form submissions of individual users. Note that for
many users not behind a proxy, the network address of the host
running the user agent will also serve as a long-lived user
identifier. In environments where proxies are used to enhance
privacy, user agents ought to be conservative in offering accept
header configuration options to end users. As an extreme privacy
measure, proxies could filter the accept headers in relayed requests.
General purpose user agents which provide a high degree of header
configurability SHOULD warn users about the loss of privacy which can
be involved.
Personally, I would bounce the request with a 401.2 and route the requester to a challenge screen via the WWW-Authenticate response header that presents them with notification that they will not be allowed anonymous access to your site. It's kind of a bastardized way of using the WWW-Authenticate header, but it seems like you're expecting the X-Forwarded-For header to acknowledge and identify the real requester and allowing public non-anonymous access to your service. To me, that's an authentication concern.
ISP providers does not adds X-Forwarded-For.
X-Forwarded-For is not for end user to identify application behind proxy/balancer.
X-Forwarded-For is for application behind proxy/balancer to identify
user.
For example:
You got web application (php, java, etc.)
also you have http server (Apache, nginx, etc.) then:
User do request to http server.
Http server redirect request to web application with X-Forwarded-For as user ip.
Yours web application know that it is behind http server so it does read X-Forwarded-For as user ip.
Why are you expecting X-Forwarded-For to appear in the first place? You are connecting to a web server running on localhost, so there is no ISP provider involved at all. Even if you were connecting to your web server via an ISP, it is still not likely to add X-Forwarded-For to the requests. X-Forwarded-For is typically added by an HTTP proxy server or load balancer, neither of which you are going through. X-Forwarded-For is never included by a web browser.