I'm using qnetworkaccessmanager for making HTTP requests. The hostname (FQDN) of the server I connect to, has two IP addresses in DNS and I need to control which one to use. The obvious solution (change the URL's hostname to IP address) does not work, because the server sends back a 302 redirect with the original hostname in the location field. If I follow the redirect, QT seems to randomly choose which IP it connects to.
Is there a way to tell qnetworkaccessmanager to use a given IP address on the TCP connect() level and use the Host header from the URL ? If not, any workaround suggestions are appreciated.
Edit: using QT 4.7.4
This seems to work and is a simple workaround: set the QNetworkRequest's URL to contain the desired IP address to connect to in the host part, but also use setCustomHeader("Host", "<server hostname>") to avoid the redirection. In my tests, QT will always use the IP set in the URL.
You should be able to build a custom QNetworkRequest and specify the QNetworkRequest::LocationHeader to force a specific destination URL in case of a redirect. If you look at QNetworkAccessManager::sendCustomRequest (QNetworkAccessManager::sendCustomRequest doc), and QNetworkRequest::Attribute::RedirectionTarget and QNetworkRequest::KnownHeaders it should give you some hints about it.
(footnote: I'm using the harmattan documentation as the proper Qt documentation is down as of time of answer)
Related
I'm debugging some DNS issues on a WiFi access point I'm setting up. I am connecting an ESP32 to my network so it can output what it's up to. I want to get my Public IP address as a test and I want to use http://ifconfig.me/ip to go and get it. This is superb when DNS is working, but as per standard HTTPS stuff, I can use the Host: name header in a call to the IP address to test connectivity to determine if it's just a DNS failure I'm seeing. If I set up the following in Postman, everything is super...
When I do the following on the ESP32, which should mimic the above, I get a 404 - which happens when the Host header is not sent or is incorrect.
...
String new_url = "http://34.117.59.81/ip";
String host = "ifconfig.me";
HTTPClient http;
http.begin(new_url.c_str());
http.addHeader("Host", host.c_str());
int httpResponseCode = http.GET();
...
What am I doing wrong? I've tried the addHeader() call either side of the begin()
I've tried collectHeaders() call, but that only keeps the response headers so I can't see what I'm actually sending.
On the positive note, the fact I'm getting a 404 does mean connectivity is fine, and the issue is my DNS as expected, but why can't I (allegedly) send the Host header?
The HTTPClient library adds the Host header for you automatically.
The library is written to silently ignore attempts to add your own Host field (along with Connection, UserAgent and Authorization):
// not allow set of Header handled by code
if(!name.equalsIgnoreCase(F("Connection")) &&
!name.equalsIgnoreCase(F("User-Agent")) &&
!name.equalsIgnoreCase(F("Host")) &&
!(name.equalsIgnoreCase(F("Authorization")) && _base64Authorization.length())){
String headerLine = name;
Unfortunately it doesn't currently offer a method to tell it to use a particular Host with an IP address that's not resolved from it.
You could copy the library into your project and just delete the check that stops it from using your Host field. Or (I hate suggesting this because I usually push people to use the HTTPClient library rather than roll their own) you could do your own simple HTTP implementation using WiFiClient.
It sounds like you understand this already but for others who may read this, HTTP allows one IP address to serve multiple domains (back in the old days it didn't but as we ran low on IP addresses this became critical). The server uses the Host field to decide which domain to direct traffic to. It's likely that the server you're connecting to is accepting the connection but without the Host field being set properly the request isn't making it to the correct server and is failing with a 404.
Let's say I have this DNS entry: mysite.sample. I am developing, and have a copy of my website running locally in http://localhost:8080. I want this website to be reachable using the (fake) DNS: http://mysite.sample, without being forced to remember in what port this site is running. I can setup /etc/hosts and nginx to do proxing for that, but ... Is there an easier way?
Can I somehow setup a simple DNS entry using /etc/hosts and/or dnsmasq where also a non-standard port (something different than :80/:443) is specified? Without the need to provide extra configuration for nginx?
Or phrased in a simpler way: Is it possible to provide port mappings for dns entries in /etc/hosts or dnsmasq?
DNS has nothing to do with the TCP port. DNS is there to resolv names (e.g. mysite.sample) into IP addresses - kind of like a phone book.
So it's a clear "NO". However, there's another solution and I try to explain it.
When you enter http://mysite.sample:8080 in your browser URL bar, your client (e.g. browser) will first try to resolve mysite.sample (via OS calls) to an IP address. This is where DNS kicks in, as DNS is your name resolver. If that happened, the job of DNS is finished and the browser continues.
This is where the "magic" in HTTP happens. The browser is connecting to the resolved IP address and the desired port (by default 80 for http and 443 for https), is waiting for the connection to be accepted and is then sending the following headers:
GET <resource> HTTP/1.1
Host: mysite.sample:8080
Now the server reads those headers and acts accordingly. Most modern web servers have something called "virtual hosts" (i.e. Apache) or "sites" (i.e. nginx). You can configure multiple vhosts/sites - one for each domain. The web server will then provide the site matching the requested host (which is retreived by the browser from the URL bar and passed to the server via Host HTTP header). This is pure HTTP and has nothing to do with TCP.
If you can't change the port of your origin service (in your case 8080), you might want to setup a new web server in front of your service. This is also called reverse proxy. I recommend reading the NGINX Reverse Proxy docs, but you can also use Apache or any other modern web server.
For nginx, just setup a new site and redirect it to your service:
location mysite.example {
proxy_pass http://127.0.0.1:8080;
}
There is a mechanism in DNS for discovering the ports that a service uses, it is called the Service Record (SRV) which has the form
_service._proto.name. TTL class SRV priority weight port target.
However, to make use of this record you would need to have an application that referenced that record prior to making the call. As Dominique has said, this is not the way HTTP works.
I have written a previous answer that explains some of the background to this, and why HTTP isn't in the standard. (the article discusses WS, but the underlying discussion suggested adding this to the HTTP protocol directly)
Edited to add -
There was actually a draft IETF document exploring an official way to do this, but it never made it past draft stage.
This document specifies a new URI scheme called http+srv which uses a DNS SRV lookup to locate a HTTP server.
There is an specific SO answer here which points to an interesting post here
In general on any non-HTTP server. Would there be a way to detect what domain was used to reach the IP?
I know HTTP servers get the domain passed within the request header, but would this be possible with any other server that does not require this information to be received from the client?
I'm especially looking for a way to do this with the minecraft server (Bukkit) so my preferred language (if needed for you to answer) would be Java. But I'd like to not have the theories about this language specific.
In general, no, which is why the HTTP protocol includes it in the headers.
In order to reach your server, first a DNS lookup is performed to resolve your IP, which is then followed by the connection itself. These two steps are separate, and hard to link together.
Logging what domain was last requested by a client is tricky, too, as DNS information is often cached, so the DNS request may not even reach your DNS server before being answered.
If it isn't cached, it also often isn't directly looked up by the end client, but rather by a caching DNS server operated, for instance, by the ISP.
No. The only way to get the DNS name used to connect to a server is to have the client provide it.
No, if there are no means for this in the protocol itself like the Host header in HTTP you cannot find out which hostname was used on the client to resolve your IP address.
I need to ask a question about HTTP protocol. I am trying to develop a sandbox (web browser) where any one can surf the website with different identities. Different identity means that on each request to a page will be from different IP address.
Now I don't know how scripts on web servers check the IP address of the one who generated the request. This is possible and I am aware of this. But I need to know whether this is HTTP request header that has the IP address or something else.
Simply speaking, I want to fool the websites. :)
Umair
Uh, the IP address is provided EVERY time you connect to ANYTHING. It has nothing to do with http headers.
See IPv4 -> packet structure -> header
You need to read up on the layers that build up a network from the wires[1] to the application. I think you'll find the the IP address is known long before HTTP gets involved.
See http://en.wikipedia.org/wiki/OSI_model
[1] or photons, or radio waves, or smoke signals...
I am trying to call a web service from asp.net 3.5 application. I have a URL that has the DNS in it and when it is used I get the following Error.
(the xxxxxx is there for privacy concerns)
The request failed with the error message: -- 301 Moved Permanently Moved Permanently The document has moved here.
When I use the URL with the physical IP it works just fine. Are there any setting that I am missing. I currently have the URL behavior set to dynamic so that it uses the url from the webconfig.
Hm - maybe I do not understand your question correctly, but it sounds like the web service URL simply has changed from the one you use to the one returned by the 301 response (the xxxxxxxx one).
Are you sure you call the web service with exactly the xxxxxxx URL?
PS:
I have a URL that has the DNS in it
This is probably not what you wanted to say - DNS stands for Domain Name System, which would be the system that translates URLs to IP Addresses. I assume you wanted to say FQDN, meaning Fully Qualified Domain Name.
It's possible for request rewriting to be happening on the server side, based on the incoming request, including what you pass for the hostname in the URL. A request rewrite may result in a 301 response.
In other words, requests with a hostname of www.domain.com may be rewritten, while requests using a particular ip address, even if the IP address is the address that www.domain.com resolves to, may not be rewritten.
The solution is to either use the IP address, or use the new location that you get from the 301 response.
If you are using a Web Reference, then you can set the AllowAutoRedirect property of the proxy instance to true. In this case, the redirection will happen behind the scenes.