On my Ubuntu deployment server Nginx is dropping a custom request header (a token), only if the request is coming from Microsoft Edge or Internet Explorer. Requests coming from Firefox, Chrome or Safari just work fine.
I've done a tcpdump to check the difference between the incoming requests, and the requests look exactly the same (only the User-Agent is different, which seems normal). All the browsers are sending the token to nginx
Because my header contains an underscore, I have in nginx.conf the line
underscores_in_headers on;
I am logging the header in access log of nginx,and it shows up for all browsers but IE.
Nginx is proxying to a Python Flask application, using gunicorn. In the Flask application I immediately log the incoming requests and the token is disappeared if the browser is IE. So apparently nginx drops the header before sending it to gunicorn.
Any advice what can cause this.
TLDR: Do you use a WAF? Maybe a WAF as a service?
I'd suggest you investigate your full infrastructure/routing topology. There may be load balancers/things in the path that you're not taking into account.
We literally just ran into this exact same issue at my work and your post was the only thing on the internet that sounded like our problem. We ended up figuring out the root cause.
Here's a simplified version of our topology from a DNS routing perspective:
newwebsite.company.com --> Web Application Firewall as a Service (if this fails it fails open) --> Nginx+ (with WAF plugin) --> Kubernetes Nginx Ingress Controller --> Custom Angular Javascript frontend hosted on Nginx Pod
legacywebsite.company.com --> F5 load balancer --> Windows IIS Web Server.
(There was a section of the new site that used the same backend server of the legacy website, and we'd see hidden 500 errors if we used Chrome Developer Tools.
We checked IIS logs and found out headers with underscores were getting stripped from the clients HTTP request b4 they'd get to the backend IIS server/we found out that we had to add underscores_in_headers on; to every Nginx Load Balancer in the path and that fixed it! ... or so we thought. It turned out the problem was fixed for every browser except for Internet Explorer / Microsoft Edge. (Your exact scenario)
The crazy thing is if you were on the one url path of the new site that would forward your traffic to the old site's load balancers, then you were going through a crazy amount of load balancers. (The nginx pod that hosted the Angular Javascript frontend would redirect you to the F5 load balancer). We discovered the root cause by process of elimination to get rid of that crazy amount of load balancers in the routing in a way that involved minimal testing. I edited my hostfile for newwebsite.company.com to bypass the WAF as a Service and point straight to the Nginx+ LB acting as a WAF, and it started working/no more 500 errors for IE/Edge.
Our theory is that our WAF as a Service was stripping out a HTTP header that has an underscore (which Win IIS web servers use), and they were only stripping out this HTTP header for Edge/IE. So we've got a ticket with them explaining the situation and directions for reproduceability.
Related
I have a few websites that use to be on a server using iis7 as the host. I have moved these websites to a new server that is IIS8 and the database has been upgraded from SQL 2005 to SQL 2014.
Another part of this change is it now runs through a DMZ reverse proxy that redirects to an internal server.
This works fine in Chrome or Edge. But Firefox and IE I get a 500 URL Rewrite Module Error. Not much more information in the error other than that.
I have other sites on the reverse proxy that work with no issue. But all of the ones that work are .net 4.0 or higher. The sites I am having issue with are both 3.5 framework.
I have tried setting the app pool framework on the dmz to match the internal server.
There are currently 2 inbound rules one converts http to https and the other is the proxy rule. There is 1 Outbound rule which is also part of the revers proxy. The reverse proxy currently takes the https traffic and uses http internally and then the outbound sends it back as https. This is that same on all of the site on this server that currently work without any issues.
Some more information. I turned of error tracing and the fuller error I received is
Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip").
This is because the responses that are coming from the back end server are using HTTP Compression, and URL rewrite cannot modify a response that is already compressed. This causes a processing error for the outbound rule resulting in the 500.52 status code.
There are two ways to work around this: either you turn off compression on the backend server that is delivering the HTTP responses (which may or may not be possible, depending on your configuration), or we attempt to indicate to the backend server the client does not accept compressed responses by removing the header when the request comes into the IIS reverse proxy and by placing it back when the response leaves the IIS server
There are a number of step needed to complete this fix you can find them and all the information you need at https://blogs.msdn.microsoft.com/friis/2016/08/25/iis-with-url-rewrite-as-a-reverse-proxy-part-2-dealing-with-500-52-status-codes/
It is a 3 part post and the second post in the series was the solution.
We have two Windows 2008 R2 serveres set up with a Cisco Ace-30 load balancer, and a few MVC and WebForms apps (.net 4.5 and 4.5.1) with federated authentication (thinktecture). The load balancer is configured with ssl, but the traffic from load balancer to web servers is on http. The problem I'm trying to solve is that the part of the url is case sensitive. Take this urls:
https://my-server.temp.net/MyApp/ (working)
https://my-SERVER.temp.net/MyApp/ (working)
https://my-SERVER.temp.net/MyApp/DEfault.AsPX (working)
https://my-server.temp.net/myapp/ (not-working, that is no response)
In the load balancer, I've disabled the default case sensitivity, and from what I've read there's no such setting in IIS. I've tried creating a static website (no auth) and this works as expected:
https://my-server.temp.net/StaticSITE/ (working)
https://my-server.temp.net/staticsite/ (working)
I've also tried setting up a url rewrite rule, and once that rule is applied the the timeout appear. My guess is that it is IIS that rewrites the incomming http request to a new http request, and the load balancer does not recognize its the https request that has been modified. But I'm guessing and beyond my comfort zone here.
So now I'm a bit confused and unsure what's causing the url to be case sensitive. Is it:
The load balancer
IIS
The authentication
Something else
I would be really happy for any ideas or debugging tips
Thanks
Larsi
ok, I figured out the problem (or atleast a solution). The IIS was sending a redirect and that redirect was to a http address.
So when we configured the load balancer to redirect http to https everything works as expected. My guess is that the federation was causing the IIS to do an extra redirect when the casing was not correct, but that's just a thought.
Anyway - http to https redirect on load balancer solved it.
Larsi
I am having a problem with an old school asp.net site that has been moved behind a SSL terminated loadbalancer. All traffic to IIS is coming from the load balancer as port 80 http traffic. IIS is rendering all the script/image/link urls into http://....
The problem for us is the client browers are seeing pages with absolute URLs containing http protocol links within a https:// page. The browsers are warning about mixing secure and insecure content.
How can this be fixed so that the links are generated with https:// instead of http://
Thanks in Advance
bille
We solved our issue. The problem turned out to be the load balancer configuration. The client is very compartmentalized and denied us access to the LB swearing that the config was correct. After proving to them that telnetting into 443 was being blocked for some stuff we were allowed access to the LB. In addition to just being set up wrong (cut and paste) the LB url rewrite rules were messed up. Fixing those all is good.
I've got an internal web application that's designed to work in concert with a server running locally on the client machine. (For the curious: the local server is used to decrypt data retrieved from the server using the client machine's GPG key.)
The internal web app is served over HTTPS while the local app is accessible via localhost. It used to be that I could make unencrypted AJAX requests from the page to localhost without any issues; but it seems that recently Chrome was updated to disallow HTTP requests to any destination from pages served over HTTPS.
I understand that in the vast majority of cases, HTTP requests from a page served via HTTPS constitute a security hole. However, since I have complete control over the endpoint in this case (i.e., localhost), it seems to me that it should still be perfectly safe to make HTTP requests to that one destination even when the host page has been served via HTTPS.
Is this possible? To whitelist localhost somehow?
Since you are in control of both the client and the server, it sounds like a good candidate for Cross-Origin Resource Sharing (CORS). The server will have to set a few response headers to give access to the client. You can learn more here: http://www.html5rocks.com/en/tutorials/cors/
I have an IIS6 ASP.NET 2/3.5 site that ALWAYS seems to be caching anything that goes through a https request. HTTP requests always work the same, but i see old/invalid data on https requests.
I see this a lot with pages where i'm doing http rewriting--but in general, it almost seems like the site has one set of values it shows to http requests and another to https requests-- without having any backend code that renders differently based upon the scheme.
Any ideas? Does IIS have a weird ssl caching option i'm not aware of?
The ports were set up wrong in IIS-- although the domain was correct because i was using port 443 it got forwarded to another site (because of host headers setup).