Note: Please do read the full question
I'm trying to understand as to why the browsers doesn't show me any X-Forwarded-For header every time a request a page
BTW here are my Request Headers look like
Request URL:http://localhost:3000/users/sign_in
Request Method:GET
Status Code:304 Not Modified
Request Headers:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:max-age=0
Connection:keep-alive
Cookie:undefined=0; poasterapp=s%3A4faaa6b1723e7c6fbd949083532c52598652547b.sNX%2BKOEed2TEQkQN7I7K5lgpoHMRpwerKFvUegMnTVI; _minerva_session=BAh7CUkiD3Nlc3Npb25faWQGOgZFRkkiJWEyM2Q0ZTViMWEyODBiYmFmODEwZTJhZmUwNWU5ODk5BjsAVEkiE3VzZXJfcmV0dXJuX3RvBjsARiIGL0kiCmZsYXNoBjsARm86JUFjdGlvbkRpc3BhdGNoOjpGbGFzaDo6Rmxhc2hIYXNoCToKQHVzZWRvOghTZXQGOgpAaGFzaHsGOgphbGVydFQ6DEBjbG9zZWRGOg1AZmxhc2hlc3sGOwpJIgAGOwBUOglAbm93MEkiEF9jc3JmX3Rva2VuBjsARkkiMUN0Uk56SXU0dUdIdzgwcFZJM3R0L2N4dlovRllTSGRrQ2o1R0VVanhIaVk9BjsARg%3D%3D--6bd89ce9d29e9bdcf56573f9a153dc663a8fe755
Host:localhost:3000
If-None-Match:"785d34e3998360353567fc710af123fb"
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36
Response Headers(Not need but still )
Cache-Control:max-age=0, private, must-revalidate
Connection:close
ETag:"785d34e3998360353567fc710af123fb"
Server:thin 1.5.0 codename Knife
Set-Cookie:_minerva_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJWEyM2Q0ZTViMWEyODBiYmFmODEwZTJhZmUwNWU5ODk5BjsAVEkiE3VzZXJfcmV0dXJuX3RvBjsARiIGL0kiEF9jc3JmX3Rva2VuBjsARkkiMUN0Uk56SXU0dUdIdzgwcFZJM3R0L2N4dlovRllTSGRrQ2o1R0VVanhIaVk9BjsARg%3D%3D--dfb3ce9f5c97463cfcd0229a133654e6cc606d98; path=/; HttpOnly
X-Request-Id:41a6f3062dc8bc36b7b3eae71dc5075d
X-Runtime:89.238257
X-UA-Compatible:IE=Edge
Now as said, I dont see any X-Forwarded-For in request headers
Reading through the wiki pages of X-Forwarded-For make me feel that ,it is something done by caching server(which in my case I believe is my ISP provider) so am I safe to believe that the **X-Forwarded-For** headers is something that is added at the caching server side (ISP provider)
If yes their is this one bugging me about it then that is
why? is the same true (i.e X-Forwarded-For not appearing in the request-headers ) for my server running locally on my machine and I accessing them via browser like http://localhost:3000
X-Forwarded-For is not a standard request header as specified in RFC 2616 Section 5.3 that addresses the protocol standard request headers, which are (as specified in the RFC)
Accept
Accept-Charset
Accept-Encoding
Accept-Language
Authorization
Expect
From
Host
If-Match
If-Modified-Since
If-None-Match
If-Range
If-Unmodified-Since
Max-Forwards
Proxy-Authorization
Range
Referer
TE
User-Agent
In order for your incoming request to have the custom [X-Forwarded-For] header, it must be explicitly added to that request by the calling client. The easiest explanation of why you are not seeing that header is that the client sending the request did not manually add it.
The tricky thing is that the header you are expecting to see is not a header that you should necessarily be expecting to receive unless there is a contract in place between your service and the caller that is apart from the HTTProtocol indicating that you should expect an X-Forwarded-For value to be specified in your request header. As others have already stated, the XFF header is typically set by a proxy server or a load balancer to indicate who the real requester is that is acting through their proxy.
As a service provider, if you demand that an [X-Forwarded-For] header be set in all requests, you must enforce it at a service policy level. If you do not want to service proxy accounts that do not identify who they are shielding with their proxy IP, bounce their request with a 403 Forbidden. If you are in a situation where you must service these requests but depend on this header being set, then you're going to have to come up with a custom process where you could communicate their error back.
Here is what the HTTProtocol has to say about anonymity:
Because the source of a link might be private information or might
reveal an otherwise private information source, it is strongly
recommended that the user be able to select whether or not the
Referer field is sent. For example, a browser client could have a
toggle switch for browsing openly/anonymously, which would
respectively enable/disable the sending of Referer and From
information.
Clients SHOULD NOT include a Referer header field in a (non-secure)
HTTP request if the referring page was transferred with a secure
protocol.
Authors of services which use the HTTP protocol SHOULD NOT use GET
based forms for the submission of sensitive data, because this will
cause this data to be encoded in the Request-URI. Many existing
servers, proxies, and user agents will log the request URI in some
place where it might be visible to third parties. Servers can use
POST-based form submission instead
...
Elaborate user-customized accept header fields sent in every request,
in particular if these include quality values, can be used by servers
as relatively reliable and long-lived user identifiers. Such user
identifiers would allow content providers to do click-trail tracking,
and would allow collaborating content providers to match cross-server
click-trails or form submissions of individual users. Note that for
many users not behind a proxy, the network address of the host
running the user agent will also serve as a long-lived user
identifier. In environments where proxies are used to enhance
privacy, user agents ought to be conservative in offering accept
header configuration options to end users. As an extreme privacy
measure, proxies could filter the accept headers in relayed requests.
General purpose user agents which provide a high degree of header
configurability SHOULD warn users about the loss of privacy which can
be involved.
Personally, I would bounce the request with a 401.2 and route the requester to a challenge screen via the WWW-Authenticate response header that presents them with notification that they will not be allowed anonymous access to your site. It's kind of a bastardized way of using the WWW-Authenticate header, but it seems like you're expecting the X-Forwarded-For header to acknowledge and identify the real requester and allowing public non-anonymous access to your service. To me, that's an authentication concern.
ISP providers does not adds X-Forwarded-For.
X-Forwarded-For is not for end user to identify application behind proxy/balancer.
X-Forwarded-For is for application behind proxy/balancer to identify
user.
For example:
You got web application (php, java, etc.)
also you have http server (Apache, nginx, etc.) then:
User do request to http server.
Http server redirect request to web application with X-Forwarded-For as user ip.
Yours web application know that it is behind http server so it does read X-Forwarded-For as user ip.
Why are you expecting X-Forwarded-For to appear in the first place? You are connecting to a web server running on localhost, so there is no ISP provider involved at all. Even if you were connecting to your web server via an ISP, it is still not likely to add X-Forwarded-For to the requests. X-Forwarded-For is typically added by an HTTP proxy server or load balancer, neither of which you are going through. X-Forwarded-For is never included by a web browser.
Related
My question is similar to the one asked here, but pertains specifically to non-standard request headers.
If my server is expecting non-standard request headers (like say X-Forwarded-For), how will the browser know it's supposed to send it?
X-Forwarded-For is added when the request come into a proxy server or load-balancer that hides the main server address behind that. Also some of those headers are required by some specific servers and the clients must send the data over headers, such as x-api-key or some other names which are defined at the servers.
I am looking at the http-requests in BurpSuite. I see a field named as 'Host'. What is the importance of this field?
What happens if I change this field and then send the request? If I change the host header field to some other IP then would the server respond back to this new modified IP?
A single web server can host multiple websites with different domains and subdomains.
The Host header allows it to distinguish between them.
Given the limited availability of IPv4 addresses, this is important as there are more websites than available IP addresses.
What happens if I change this field and then send the request?
If the server pays attention to it and recognises the hostname, it will respond with that website (otherwise it may fall back to its default website or throw an error).
For an example, see Name-based Virtual Host Support in the Apache HTTPD manual.
If I change the host header field to some other IP then would the server respond back to this new modified IP?
No. The Host header is the host the client is asking for. It has nothing to do with where the response should be sent.
To quote from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Host :
The Host request header specifies the host and port number of the server to which the request is being sent.
If no port is included, the default port for the service requested (e.g., 443 for an HTTPS URL, and 80 for an HTTP URL) is implied.
A Host header field must be sent in all HTTP/1.1 request messages. A 400 (Bad Request) status code may be sent to any HTTP/1.1 request message that lacks a Host header field or that contains more than one.
I have a web server cluster behind a proxy/load balancer. That proxy contains my SSL certs and hands the web servers the decrypted traffic, and along the way adds an "x-forwarded-for" header into the HTTP header the web application receives. This application has seen millions of IP addresses over the past decade, but something weird happened today.
For the first time, I saw an x-forwarded-for that contained a second address reach the application [addressed altered]:
x-forwarded-for: 62.211.19.218, 177.168.159.85
This indicates that the traffic came through a proxy, and I understand this is normal for x-f-f. I would have thought this was impossible (or at least unlikely) with https as the protocol.
Can someone explain how this is legit?
As per RFC 7239, this HTTP header is specified as
X-Forwarded-For: client, proxy1, proxy2, ...
Where client is the IP of the original client and then each proxy adds the IP it received the request from, at the end of the list. In the above example, you would see IP of proxy3 in your webserver and proxy2 is the IP which connected to the proxy3.
As anyone can put anything inside this header, you should accept it only from known sources like your own reverse proxy or whitelist of known legit proxies. For example Apache has mod_rpaf, which transparently changes client IP address to the one provided in this header, but only if the request is received from the IP of known proxy server.
On corporate networks you can easily do transparent proxying for HTTPS traffic without any notice from normal users. Just create your own certification authority, use for example Windows Group Policy to install & trust this CA on all corporate workstations. Then redirect all HTTPS connections to your proxy which will generate certificate for all visited domains on the fly. This is something which is happening and you can even buy enterprise hardware proxies using this method.
So to summarize the reasons why you could see multiple IPs in the X-Forwarded-For header:
Transparent HTTPS proxy as mentioned above
The header was added by the requestor itself (browser, wget, script) for whatever reason, for example to hide its own IP
Some CDN like Cloudflare could add that header if used
Multiple reverse proxies defined either intentionally or by mistake
Conclusion: You should only trust this header if it originates from your own proxy (in case of multiple IPs, trust only the last one).
MAYBE it's using the Proxy protocol for HTTPS. Granted you may not be using httproxy, but this seems to be a decent description:
http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
I'm not sure about the SSL cert, but there's no guarantee someone is doing something pathalogical (maybe unintentionally) like running all their HTTPS traffic through a proxy and then accepting all the invalid certificates. But I suspect the proxy protocol might make this work; it does expose the HTTP headers to the proxy in some sense.
I've found some interesting reading on the X-Forwarded-* headers, including the Reverse Proxy Request Headers section in the Apache documentation, as well as the Wikipedia article on X-Forwarded-For.
I understand that:
X-Forwarded-For gives the address of the client which connected to the proxy
X-Forwarded-Port gives the port the client connected to on the proxy (e.g. 80 or 443)
X-Forwarded-Proto gives the protocol the client used to connect to the proxy (http or https)
X-Forwarded-Host gives the content of the Host header the client sent to the proxy.
These all make sense.
However, I still can't figure out a real life use case of X-Forwarded-Host. I understand the need to repeat the connection on a different port or using a different scheme, but why would a proxy server ever change the Host header when repeating the request to the target server?
If you use a front-end service like Apigee as the front-end to your APIs, you will need something like X-FORWARDED-HOST to understand what hostname was used to connect to the API, because Apigee gets configured with whatever your backend DNS is, nginx and your app stack only see the Host header as your backend DNS name, not the hostname that was called in the first place.
This is the scenario I worked on today:
Users access certain application server using "https://neaturl.company.com" URL which is pointing to Reverse Proxy. Proxy then terminates SSL and redirects users' requests to the actual application server which has URL of "http://192.168.1.1:5555". The problem is - when application server needed to redirect user to other page on the same server using absolute path, it was using latter URL and users don't have access to this. Using X-Forwarded-Host (+ X-Forwarded-Proto and X-Forwarded-Port) allowed our proxy to tell application server which URL user used originally and thus server started to generate correct absolute path in its responses.
In this case there was no option to stop application server to generate absolute URLs nor configure it for "public url" manually.
I can tell you a real life issue, I had an issue using an IBM portal.
In my case the problem was that the IBM portal has a rest service which retrieves an url for a resource, something like:
{"url":"http://internal.host.name/path"}
What happened?
Simple, when you enter from intranet everything works fine because internalHostName exists but... when the user enter from internet then the proxy is not able to resolve the host name and the portal crashes.
The fix for the IBM portal was to read the X-FORWARDED-HOST header and then change the response to something like:
{"url":"http://internet.host.name/path"}
See that I put internet and not internal in the second response.
For the need for 'x-forwarded-host', I can think of a virtual hosting scenario where there are several internal hosts (internal network) and a reverse proxy sitting in between those hosts and the internet. If the requested host is part of the internal network, the requested host resolves to the reverse proxy IP and the web browser sends the request to the reverse proxy. This reverse proxy finds the appropriate internal host and forwards the request sent by the client to this host. In doing so, the reverse proxy changes the host field to match the internal host and sets the x-forward-host to the actual host requested by the client. More details on reverse proxy can be found in this wikipedia page http://en.wikipedia.org/wiki/Reverse_proxy.
Check this post for details on x-forwarded-for header and a simple demo python script that shows how a web-server can detect the use of a proxy server: x-forwarded-for explained
One example could be a proxy that blocks certain hosts and redirects them to an external block page. In fact, I’m almost certain my school filter does this…
(And the reason they might not just pass on the original Host as Host is because some servers [Nginx?] reject any traffic to the wrong Host.)
X-Forwarded-Host just saved my life. CDNs (or reverse proxy if you'd like to go down to "trees") determine which origin to use by Host header a user comes to them with. Thus, a CDN can't use the same Host header to contact the origin - otherwise, the CDN would go to itself in a loop rather than going to the origin. Thus, the CDN uses either IP address or some dummy FQDN as the Host header fetching content from the origin. Now, the origin may wish to know what was the Host header (aka website name) the content is asked for. In my case, one origin served 2 websites.
Another scenario, you license your app to a host URL then you want to load balance across n > 1 servers.
Is any request header added or modified to the HTTP request before forwarding to the server by a proxy server?
If so, are the changes done to the same packets, or are the contents used to create new request packets with the modifications?
There are a few different types of proxy servers. Because you've mentioned request headers, I'm going to assume that you're talking about HTTP proxy servers, which forward HTTP requests, not packets.
NOTE: In the special case of HTTPS requests (TLS/SSL via CONNECT), proxy servers will just forward the content of the TCP packets (and are unable to inspect the packets unless acting as a man-in-the-middle proxy).
Of course it depends on the proxy software and its configuration, but HTTP proxies are expected to follow the W3C Guidelines for Web Content Transformation Proxies, which states many things, but most relevantly:
Other than to convert between HEAD and GET proxies must not alter request methods.
If the request contains a Cache-Control: no-transform directive, proxies must not alter the request other than to comply with transparent HTTP behavior defined in RFC 2616 HTTP sections section 14.9.5 and section 13.5.2 and to add header fields as described in 4.1.6 Additional HTTP Header Fields.
Other than the modifications required by RFC 2616 HTTP proxies should not modify the values of header fields other than the User-Agent, Accept, Accept-Charset, Accept-Encoding, and Accept-Language header fields and must not delete header fields.
Proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header field.
Proxies must (in accordance with RFC 2616) include a Via HTTP header field.
In summary, you can generally expect these HTTP headers to be changed/added by a standards-compliant proxy:
User-Agent
Accept
Accept-Charset
Accept-Encoding
Accept-Language
X-Forwarded-For
Via