Is it possible to force a k6 HTTP request to a custom IP?
i.e. DNS has host name pointing to IP address A, while a test website, same hostname, is on IP address B; need to send k6 load test requests to website B.
It's HTTPS, so there's certificate issues calling https://[ip address] and headers Host: [hostname] (shows "x509: cannot validate certificate for [ip address] because it doesn't contain any IP SANs" messages)
Working from localhost I'm currently editing /etc/hosts as a work around.
Setting IP address is possible in the loadimpact.com API (https://loadimpact.com/load-script-api#http), I'm hoping could be done via k6 API.
No, not yet without running into the SNI issue you describe with TLS. We have an issue to track the DNS override feature here: https://github.com/loadimpact/k6/issues/474
As you mention, modifying the local /etc/hosts or C:\Windows\System32\drivers\etc\hosts file is the workaround until this has been implemented in k6.
Related
So I did a DNS lookup of a website hosted through cloudflare.
I pasted the IP address in my address bar and got a page saying:
Error 1003 Ray ID: 729ca4f4aff82e38 • 2022-07-12 20:49:14 UTC
Direct IP access not allowed
If my browser is doing the same thing i.e. fetching the ip using the url then sending a HTTPS req to the same IP, but when I do it manually I am getting this error - how can cloudflare detect that its a direct IP access attempt?
how can cloudflare detect that its a direct IP access attempt?
Just based on what you (your browser) are sending!
An URL is of the form http://hostname/path considering that hostname can be an IP address.
When you put that in your browser, the browser will split the parts and do an HTTP query.
The HTTP protocol defines an HTTP message to be headers plus an optional body. Among headers, one is called host and the value is exactly what was in URL.
Said differently, between http://www.example.com/ and http://192.0.2.42/ (if www.example.com was resolving to that IP address):
at the TCP/IP level nothing changes: in both cases, through OS, the browser connects at IP address 192.0.2.42 (because the www.example.com from first URL will be resolved to its IP address)
when it starts the HTTP exchange, the message sent by the client will then have as header either host: www.example.com in the first case or host: 192.0.2.42 in the second case
the webserver sees obviously all headers sent by client, including this host one and hence can do whatever it wants with it, and most importantly select which website was requested if multiple websites resolves to the same IP address (if you understand the text above, you now see why the host header is necessary). If URLs are https:// and not just http:// there is a subtetly because there is another layer between the TCP/IP connection and the HTTP application protocol, which is TLS, and the equivalent of the host header is sent also at the TLS level, through what is called the SNI extension, so that the server can also decide which server certificate it needs to send back to the client, before even the first byte of the HTTP exchange is done.
I have a firebase project with 2 web applications:
name: url
PC pc-my-app.web.app
Mobile m-my-app.web.app
firebase gives me the IP X.X.X.X for both applications.
I have a domain, example.com with the following records:
A example.com X.X.X.X
A m.example.com X.X.X.X
I thought that having these records pointing to the same IP would mean I see the same webpage, but that is clearly not the case.
Since ports are (apparently) never specified, how can the server on X.X.X.X tell which application I am trying to connect to?
My guess is that the server on X.X.X.X also recieves the original URL as a parameter somehow...
Firebase Hosting (and most other hosting providers) check the Host header in the incoming request to determine what content to serve. And this Host header value allows a single server to map different requests to the correct web site content (often referred to as a "virtual host").
Also see:
The MDN documentation for Host header
What is HTTP "Host" header?
Let's say I have this DNS entry: mysite.sample. I am developing, and have a copy of my website running locally in http://localhost:8080. I want this website to be reachable using the (fake) DNS: http://mysite.sample, without being forced to remember in what port this site is running. I can setup /etc/hosts and nginx to do proxing for that, but ... Is there an easier way?
Can I somehow setup a simple DNS entry using /etc/hosts and/or dnsmasq where also a non-standard port (something different than :80/:443) is specified? Without the need to provide extra configuration for nginx?
Or phrased in a simpler way: Is it possible to provide port mappings for dns entries in /etc/hosts or dnsmasq?
DNS has nothing to do with the TCP port. DNS is there to resolv names (e.g. mysite.sample) into IP addresses - kind of like a phone book.
So it's a clear "NO". However, there's another solution and I try to explain it.
When you enter http://mysite.sample:8080 in your browser URL bar, your client (e.g. browser) will first try to resolve mysite.sample (via OS calls) to an IP address. This is where DNS kicks in, as DNS is your name resolver. If that happened, the job of DNS is finished and the browser continues.
This is where the "magic" in HTTP happens. The browser is connecting to the resolved IP address and the desired port (by default 80 for http and 443 for https), is waiting for the connection to be accepted and is then sending the following headers:
GET <resource> HTTP/1.1
Host: mysite.sample:8080
Now the server reads those headers and acts accordingly. Most modern web servers have something called "virtual hosts" (i.e. Apache) or "sites" (i.e. nginx). You can configure multiple vhosts/sites - one for each domain. The web server will then provide the site matching the requested host (which is retreived by the browser from the URL bar and passed to the server via Host HTTP header). This is pure HTTP and has nothing to do with TCP.
If you can't change the port of your origin service (in your case 8080), you might want to setup a new web server in front of your service. This is also called reverse proxy. I recommend reading the NGINX Reverse Proxy docs, but you can also use Apache or any other modern web server.
For nginx, just setup a new site and redirect it to your service:
location mysite.example {
proxy_pass http://127.0.0.1:8080;
}
There is a mechanism in DNS for discovering the ports that a service uses, it is called the Service Record (SRV) which has the form
_service._proto.name. TTL class SRV priority weight port target.
However, to make use of this record you would need to have an application that referenced that record prior to making the call. As Dominique has said, this is not the way HTTP works.
I have written a previous answer that explains some of the background to this, and why HTTP isn't in the standard. (the article discusses WS, but the underlying discussion suggested adding this to the HTTP protocol directly)
Edited to add -
There was actually a draft IETF document exploring an official way to do this, but it never made it past draft stage.
This document specifies a new URI scheme called http+srv which uses a DNS SRV lookup to locate a HTTP server.
There is an specific SO answer here which points to an interesting post here
I have a general networking question but it's related with security aspect.
Here is my case: I have a host which is infected by a malware. The malware creates an http packet to communicate with it's command and control server. While constructing the packet, the IP layer contains the correct IP address of the command and control server. The tcp layer contains the correct port number 80.
Before sending the packet out, the malware modifies the http header to replace the host header with “google.com" instead of it's server address. It then attaches the stolen data with the packet and sends it out.
My understanding is that the packet will get delivered to the correct server because the routing will happen based on the IP.
But can I host a webserver on this IP that would receive all packets with header host google.com and parse it correctly?
Based on my reading on the internet, it is possible but if it is that easy then why have malware authors not adopted this technique to spoof the http headers and bypass traditional domain whitelisting engines.
When you make a request to let's say Apache2 server, what actually Apache does is match your "Host" header with any VirtualHost within server's configuration. Only if it cannot be found / is invalid, Apache will route the request to default virtualhost if it's defined. Basically nothing stops you from changing these headers.
You can simply test it by editing your hosts file and pointing google.com to any other IP - you will be able to handle the google.com domain on your server, but only you will be to use it this way - no one else.
Anything you send inside HTTP headers shouldn't be trusted - it just a guide for your server on how to actually handle the traffic.
The fake host header is just there to trick some deep-inspection firewalls ("it's for Google? you may pass..."). The server on that IP either doesn't care about the host header (default vhost) or is explicitly configured to accept it.
Passing the loot on by using fake headers or just as plain data behind the headers is another trick to fool data loss prevention.
These methods can mislead shallow application-layer inspection but won't pass a decent firewall.
I have multiple sites running on my IIS, now for one of the websites (SiteB) we need to support ssl requests. I have enabled it editing bindings for the website, but the problem is when I selected protocol SSL editing bindings HostName field is disabled, being unable to set hostname to respond to https request, this causes that all sites of my IIS if are requested with https:// loads web site of siteB.
For example my bidings are the next
Site A
IP Port HostName
* 80 www.sitea.com
Site B
IP Port Hostname
* 443 www.siteb.com
* 80 www.siteb.com
If I type https://www.siteb.com in my browser it works correctly, but if I type https://www.sitea.com in the browser, siteb webpage is loaded with the hostname of sitea.
How Can I make that only https://www.siteb.com responds to https requests on my IIS?
I have tried with command appcmd too but It't doesnt work.
appcmd set site /site.name:{sitB} /bindings.[protocol='https',bindingInformation='*:443:*'].bindingInformation:*:443:siteB.com
Thanks for your help.
The Root Problem
This unexpected behavior isn't because of IIS so much as it is because of the web encryption protocols.
The two major web encryption protocols are SSL and TLS. Both of these protocols negotiate a secure connection before passing any request information to the server. This means that, on secure requests, servers don't actually learn the hostname until after the secure connection is made.
An extension to TLS and SSL has been created to address this limitation. It's called SNI (Server Name Identification). The problem is that this extension needs to be supported on both the server and client machines. Currently the client browser support is somewhat spotty. See the SNI article for a browser list.
IIS's Handling Of The Problem
It is because of the above mentioned hostname limitation that IIS doesn't allow you to bind hostnames to HTTPS bindings. There is no way for IIS to route HTTPS requests to a particular hostname since it doesn't know the requested hostname when it first begins to negotiate the connection.
Once IIS has negotiated a secure connection with a client and learns that their requested hostname is for a site other than the one with the HTTPS binding (e.g. a request for https://sitea.com) IIS can either return a failure code or try to fail gracefully. IIS chooses the latter and tries to fail gracefully by serving up the site with the HTTPS binding even though the user is requesting a different site.
Solutions/Workarounds
Create a rewrite rule to redirect all HTTPS requests for nonsecure websites to HTTP.
Upgrade to IIS 8 to use the SNI extension. Then ask visitors to upgrade to browsers that suport SNI.
Have your secure site return an error message when it receives a request for a different domain.
Bind by IP address instead of hostname since IIS can route HTTPS requests by IP address
References
Most of my information came from the Wikipedia article on SNI
We run webservers with multiple sites requiring SSL with no problem.
If I understand your problem correctly - you'll need to set up a binding instead of a host name - which won't work. So, for each SSL-enabled site we host, we require a distinct external IP address. Then, enter that IP address as the binding when setting up the site in IIS.