I'm debugging some DNS issues on a WiFi access point I'm setting up. I am connecting an ESP32 to my network so it can output what it's up to. I want to get my Public IP address as a test and I want to use http://ifconfig.me/ip to go and get it. This is superb when DNS is working, but as per standard HTTPS stuff, I can use the Host: name header in a call to the IP address to test connectivity to determine if it's just a DNS failure I'm seeing. If I set up the following in Postman, everything is super...
When I do the following on the ESP32, which should mimic the above, I get a 404 - which happens when the Host header is not sent or is incorrect.
...
String new_url = "http://34.117.59.81/ip";
String host = "ifconfig.me";
HTTPClient http;
http.begin(new_url.c_str());
http.addHeader("Host", host.c_str());
int httpResponseCode = http.GET();
...
What am I doing wrong? I've tried the addHeader() call either side of the begin()
I've tried collectHeaders() call, but that only keeps the response headers so I can't see what I'm actually sending.
On the positive note, the fact I'm getting a 404 does mean connectivity is fine, and the issue is my DNS as expected, but why can't I (allegedly) send the Host header?
The HTTPClient library adds the Host header for you automatically.
The library is written to silently ignore attempts to add your own Host field (along with Connection, UserAgent and Authorization):
// not allow set of Header handled by code
if(!name.equalsIgnoreCase(F("Connection")) &&
!name.equalsIgnoreCase(F("User-Agent")) &&
!name.equalsIgnoreCase(F("Host")) &&
!(name.equalsIgnoreCase(F("Authorization")) && _base64Authorization.length())){
String headerLine = name;
Unfortunately it doesn't currently offer a method to tell it to use a particular Host with an IP address that's not resolved from it.
You could copy the library into your project and just delete the check that stops it from using your Host field. Or (I hate suggesting this because I usually push people to use the HTTPClient library rather than roll their own) you could do your own simple HTTP implementation using WiFiClient.
It sounds like you understand this already but for others who may read this, HTTP allows one IP address to serve multiple domains (back in the old days it didn't but as we ran low on IP addresses this became critical). The server uses the Host field to decide which domain to direct traffic to. It's likely that the server you're connecting to is accepting the connection but without the Host field being set properly the request isn't making it to the correct server and is failing with a 404.
Related
I have a web application running on a server (let's say on localhost:8000) behind a reverse proxy on that same server (on myserver.example:80). Because of the way the reverse proxy works, the application sees an incoming request targeted at localhost:8000 and the framework I'm using therefore tries to generate absolute URLs that look like localhost:8000/some/ressource instead of myserver.example/some/ressource.
What would be "the correct way" of generating an absolute URL (namely, determining what hostname to use) from behind a proxy server like that? The specific proxy server, framework and language don't matter, I mean this more in an HTTP sense.
From my initial research:
RFC7230 explicitly says that proxies MUST change the Host header when passing the request along to make it look like the request came from them, so it would look like using Host to determine what hostname to use for the URL, yet in most places where I have looked, the general advice seems to be to configure your reverse proxy to not change the Host header (counter to the spec) when passing the request along.
RFC7230 also says that "request URI reconstruction" should use the following fields in order to find what "authority component" to use, though that seems to also only apply from the point-of-view of the agent that emitted that request, such as the proxy:
Fixed URI authority component from the server or outbound gateway config
The authority component from the request's firsr line if it's a complete URI instead of a path
The Host header if it's present and not empty
The listening address or hostname, alongside with the incoming port number if it's not the default one for the protocol
HTTP 1.0 didn't have a Host header at all, and that header was added for routing purposes, not for URL authority resolution.
There are headers that are made specifically to let proxies to send the old value of Host after routing, such as Via, Forwarded and the unofficial X-Forwarded-Host, which some servers and frameworks will check, but not all, and it's unclear which one should even take priority given how there's 3 of them.
EDIT: I also don't know whether HTTPS would work differently in that regard, given that the headers are part of the encrypted payload and routing has to be performed another way because of this.
In general I find it’s best to set the real host and port explicitly in the application rather than try to guess these from the incoming request.
So for example Jira allows you to set the Base URL through which Jira will be accessed (which may be different to the one that it is actually run as). This means you can have Jira running on port 8080 and have Apache or Nginx in front of it (on the same or even a different server) on port 80 and 443.
Is it possible to have the log output for icecast read a variable from a header sent to the server? Currently I am setting up an icecast server in Kuberenetes and I'm trying to get source IP preservation onto the stream for analyzing log data. However even with the necessary steps on kubernetes side I am not seeing the logs for icecast have source IP. How ever I was able to sniff incoming request and I am seeing:
X-Real-IP: 142.x.x.x
X-Forwarded-For: 142.x.x.x
As headers coming into the server.
Is it possible to get these into the logs somehow?
Not at the moment.
We plan to support reverse proxying in release 2.5.
Our general recommendation at the moment is to not reverse proxy Icecast due to many possible problems beyond just losing the originating IP address.
Just expose the Icecast ports directly to the Internet, e.g. through port forwarding.
edit: You can just declare protocol: TCP ports for your service. also "Proxy-mode: iptables"
If you don't mind a little tinkering you can do it.
Clone the source
Find these lines in the source files, which define what is logged as listener ip:
./src/logging.c:159: client->con->ip,
./src/admin.c:700: xmlNewTextChild(node, NULL, XMLSTR(mode == OMODE_LEGACY ? "IP" : "ip"), XMLSTR(client->con->ip));
Add this snippet to define a new variable, this will be set to value of X-Real-IP header if present or client IP if not:
const char *realip;
realip = httpp_getvar (client->parser, "x-real-ip");
if (realip == NULL)
realip = client->con->ip;
Change the previously mentioned lines to reference the new variable:
./src/logging.c:163: realip,
./src/admin.c:700: xmlNewTextChild(node, NULL, XMLSTR(mode == OMODE_LEGACY ? "IP" : "ip"), XMLSTR(realip));
Build Icecast from source as per the instructions
Let's say I have this DNS entry: mysite.sample. I am developing, and have a copy of my website running locally in http://localhost:8080. I want this website to be reachable using the (fake) DNS: http://mysite.sample, without being forced to remember in what port this site is running. I can setup /etc/hosts and nginx to do proxing for that, but ... Is there an easier way?
Can I somehow setup a simple DNS entry using /etc/hosts and/or dnsmasq where also a non-standard port (something different than :80/:443) is specified? Without the need to provide extra configuration for nginx?
Or phrased in a simpler way: Is it possible to provide port mappings for dns entries in /etc/hosts or dnsmasq?
DNS has nothing to do with the TCP port. DNS is there to resolv names (e.g. mysite.sample) into IP addresses - kind of like a phone book.
So it's a clear "NO". However, there's another solution and I try to explain it.
When you enter http://mysite.sample:8080 in your browser URL bar, your client (e.g. browser) will first try to resolve mysite.sample (via OS calls) to an IP address. This is where DNS kicks in, as DNS is your name resolver. If that happened, the job of DNS is finished and the browser continues.
This is where the "magic" in HTTP happens. The browser is connecting to the resolved IP address and the desired port (by default 80 for http and 443 for https), is waiting for the connection to be accepted and is then sending the following headers:
GET <resource> HTTP/1.1
Host: mysite.sample:8080
Now the server reads those headers and acts accordingly. Most modern web servers have something called "virtual hosts" (i.e. Apache) or "sites" (i.e. nginx). You can configure multiple vhosts/sites - one for each domain. The web server will then provide the site matching the requested host (which is retreived by the browser from the URL bar and passed to the server via Host HTTP header). This is pure HTTP and has nothing to do with TCP.
If you can't change the port of your origin service (in your case 8080), you might want to setup a new web server in front of your service. This is also called reverse proxy. I recommend reading the NGINX Reverse Proxy docs, but you can also use Apache or any other modern web server.
For nginx, just setup a new site and redirect it to your service:
location mysite.example {
proxy_pass http://127.0.0.1:8080;
}
There is a mechanism in DNS for discovering the ports that a service uses, it is called the Service Record (SRV) which has the form
_service._proto.name. TTL class SRV priority weight port target.
However, to make use of this record you would need to have an application that referenced that record prior to making the call. As Dominique has said, this is not the way HTTP works.
I have written a previous answer that explains some of the background to this, and why HTTP isn't in the standard. (the article discusses WS, but the underlying discussion suggested adding this to the HTTP protocol directly)
Edited to add -
There was actually a draft IETF document exploring an official way to do this, but it never made it past draft stage.
This document specifies a new URI scheme called http+srv which uses a DNS SRV lookup to locate a HTTP server.
There is an specific SO answer here which points to an interesting post here
I have a general networking question but it's related with security aspect.
Here is my case: I have a host which is infected by a malware. The malware creates an http packet to communicate with it's command and control server. While constructing the packet, the IP layer contains the correct IP address of the command and control server. The tcp layer contains the correct port number 80.
Before sending the packet out, the malware modifies the http header to replace the host header with “google.com" instead of it's server address. It then attaches the stolen data with the packet and sends it out.
My understanding is that the packet will get delivered to the correct server because the routing will happen based on the IP.
But can I host a webserver on this IP that would receive all packets with header host google.com and parse it correctly?
Based on my reading on the internet, it is possible but if it is that easy then why have malware authors not adopted this technique to spoof the http headers and bypass traditional domain whitelisting engines.
When you make a request to let's say Apache2 server, what actually Apache does is match your "Host" header with any VirtualHost within server's configuration. Only if it cannot be found / is invalid, Apache will route the request to default virtualhost if it's defined. Basically nothing stops you from changing these headers.
You can simply test it by editing your hosts file and pointing google.com to any other IP - you will be able to handle the google.com domain on your server, but only you will be to use it this way - no one else.
Anything you send inside HTTP headers shouldn't be trusted - it just a guide for your server on how to actually handle the traffic.
The fake host header is just there to trick some deep-inspection firewalls ("it's for Google? you may pass..."). The server on that IP either doesn't care about the host header (default vhost) or is explicitly configured to accept it.
Passing the loot on by using fake headers or just as plain data behind the headers is another trick to fool data loss prevention.
These methods can mislead shallow application-layer inspection but won't pass a decent firewall.
I'm using qnetworkaccessmanager for making HTTP requests. The hostname (FQDN) of the server I connect to, has two IP addresses in DNS and I need to control which one to use. The obvious solution (change the URL's hostname to IP address) does not work, because the server sends back a 302 redirect with the original hostname in the location field. If I follow the redirect, QT seems to randomly choose which IP it connects to.
Is there a way to tell qnetworkaccessmanager to use a given IP address on the TCP connect() level and use the Host header from the URL ? If not, any workaround suggestions are appreciated.
Edit: using QT 4.7.4
This seems to work and is a simple workaround: set the QNetworkRequest's URL to contain the desired IP address to connect to in the host part, but also use setCustomHeader("Host", "<server hostname>") to avoid the redirection. In my tests, QT will always use the IP set in the URL.
You should be able to build a custom QNetworkRequest and specify the QNetworkRequest::LocationHeader to force a specific destination URL in case of a redirect. If you look at QNetworkAccessManager::sendCustomRequest (QNetworkAccessManager::sendCustomRequest doc), and QNetworkRequest::Attribute::RedirectionTarget and QNetworkRequest::KnownHeaders it should give you some hints about it.
(footnote: I'm using the harmattan documentation as the proper Qt documentation is down as of time of answer)