I have a page call it 'Service.asmx' that I'm trying to research via IIS (W3C) logs. However all traffic to that page does not appear in the logs.
Here are a few things to note:
All other traffic to the site does seem to appear in the log
other .asmx services in the same directory and application pool appear successfully in the log
All requests to the service result in 200 OK
Other traffic to the site with 200 OK shows successfully in the log
even though other hits within the application pool are successful I attempted to add the user associated with the pool to the local admins group. This did not result in a change.
I've wondered many times if I was looking in the correct log instance, however, by adding a custom response header and calling via a browser I'm sure I'm looking at the correct log as well.
I've seen several articles online referencing <httpLogging> configuration that might affect a single page, but I've full-text searched all *.config instances on the server and have found no references to this tag set at all.
Related
We have a .NET WPF container app in which we host several web apps using CEFSharp.WinForms control. At times, we see that for some users, some JavaScript resource requests fail with the ERR_CONTENT_DECODING_FAILED error message. This issue gets resolved if we reload the app after either clearing the CEF cache or after disabling the cache from the network tab in the developer toolbar window. Please note that this issue isn't confined to a specific subset of resource files: instead, we have seen it happening sporadically for a variety of JavaScript resource files (some hosted on Apache while the others hosted on IIS servers).
While a possible cause for usual ERR_CONTENT_DECODING_FAILED error is a server-side content-encoding issue, in this specific case, we believe this could potentially be related to the CEF browser caching. Please see the analysis section below for the reasons we believe so.
Application Setup
When we initialize CEF settings, we set MultiThreadedMessageLoop setting to true and set CachePath property to a location under %localappdata% on windows 10 machine. When the container app starts, it creates three CEF web browser controls and launches web apps in them. All three apps load concurrently. After that, more CEF web browsers are created as the user visits more apps. The user also reloads some of these apps over time. All the web apps are internal apps sharing the same domain but physically hosted on different web servers. The JavaScript resource files in question usually have caching policy set to allow them to be cached for a week.
CEFSharp version - 79.1.360.0
CEF version - r79.1.36+g90301bd+chromium-79.0.3945.130
Chromium version - 79.0.3945.130
Our Analysis so far
We checked the web-server logs for the failing JavaScript resources. We observed that in most cases, the server requests for those resource files (by the impacted user) were made a few days ago. The users are usually able to use the application well for some days before they sporadically start getting this error.
We checked the network logs (*.HAR file). We see that for the failing JavaScript resource, _transferSize is 0 (which seems to indicate that response was served from the cache)
When the error occurs, it gets resolved when we reload the app after either clearing the cache or disabling the cache from the network tab.
We tried artificially simulating this error. We used Fiddler's autoresponder feature to deliberately respond with a bad server response (the content was 'gzip' encoded however Content-Encoding header indicated 'br'). We could simulate the ERR_CONTENT_DECODING_FAILED error, however, we could see in network logs that _tranferSize was a non-zero value. We also observed that chrome did not cache the bad response. This test indicates that when the original JavaScript response was cached by the browser, it must have been a correctly encoded response, or else the browser would not have cached it.
All of the above points lead us to believe that, JavaScript resource files were downloaded (with correct encoding) and cached in CEF cache. The user was also able to use the apps for some time. After that however, in certain scenarios, some of these files potentially got corrupted in CEF cache, leading to the content decoding error.
We tried using CEF response filter mechanism as explained here to capture the bad response when content decoding error occurs. Unfortunately, we observed that dataIn stream which gets passed to filter function is null when the response fails with this error.
Summary and Questions
This is a sporadic issue which our users are facing. We haven't found a way to deterministically recreate this problem. However based on our analysis so far, we believe some JavaScript files may be getting corrupted in CEF cache over time. We are not sure if the fact that we host several CEF web browsers and load them concurrently could be playing some role in causing this issue.
Has anyone else observed/reported a similar issue? Do you have any idea if we are missing or overlooking something here or going in the wrong direction? Any pointers will be greatly appreciated.
We have an ASP.NET deployed application running on IIS 7. Lately we started having problems with the website, which usually starts at high traffic times, and the issue is that the page stops loading without showing an error. It essentially continues spinning and does not load. IIS reset would usually fix the issue but we have tried everything to resolve it with no success. Below are additional information about what we have already tried.I can intentionally put the website into this state by running 25 concurrent users to the landing page after which I would have to reset iis because it would stop responding. I am thinking this might have to do with a settings in IIS. Maximum Concurrent Connection is set to the default which is 4294967295. Kind of at a lost here.
We turned on Failed log tracing on IIS. The data in the error log did not provide anything conclusive.This might be partly related to the request not failing completely hence no log was created. Most errors were collected based on time the page took to respond.
I have also looked at the app pool and host log files and found nothing out of place
25 concurrent users is nothing. What's the back end stack? There's not too many details listed here but I'd start with looking at each stage the request is in and in addition enable failed request tracing. Mike (formerly from the IIS team) has a great write up on this, in a nutshell though:
Troubleshoot hanging requests on IIS in 3 steps
View requests
%windir%system32inetsrvappcmd list requests /elapsed:30000
Enabled Failed Request Tracing (modify if not using Default Web Site of course)
%windir%system32inetsrvappcmd configure trace "Default Web Site" /enablesite
%windir%system32inetsrvappcmd configure trace "Default Web Site" /enable /path:test.aspx /timeTaken:00:00:30
Then hopefully you can find some details via
appcmd list traces | findstr "yourpage.aspx"
When a unavailable url is accessed, we internally raise an exception and email to support team. We do this to identify is there is hidden error in our web application. Couple of days back, suddenly there was a huge number of access to unavailable url which added load to server and casued SMTP to queue large exception emails. This attempt has brought IIS completely down and none of the applications are accessible.
How to prevent this? Is there any other option like firewall etc to disallow continues request from same ip. I have seen this behavior in google. How can we achieve that?
I'd suggest caching already sent notifications. Before your application sends email, it can check if this error is already reported.
And you can set the cache validity to, say 1 minute, so you get maximum 1 same email/minute.
It is quite easy to implement in ASP.NET.
I am using a script manager for ASP.NET MVC to combine and compact CSS files and JavaScript files for pages on a website. For the most part this works as expected, however there are times (couple of times per week) when the HTTP handler responsible for returning the content returns an empty response and so pages load without any CSS - the HTML returns and the images load as well. When this happens, refreshing the page does not resolve the problem, while resetting IIS always resolves the problem. Also, without resetting IIS, after some time the problem stops.
Normally the script handler logs errors, however there are no errors logged during the issue. It seems as if the handler is never invoked. There are no failed request logs generated by IIS.
I monitored resource usage when this was happening and did not notice anything out of the ordinary. The web server is running IIS 7 and has low CPU usage. I increased some parameters in IIS settings regarding the number of allowable requests to process, the problem still exists though perhaps less frequent. The website receives about 1.5 million pageviews monthly.
This was found to be an issue with HTTP cache headers.
Ran into this issue yesterday on one of our sites. First of all the site is hosted in a web farm environment and for the time being I have added a static machineKey to the web.config on both nodes (2 node environment). This has solved the issue and everything is running fine now.
This raised the following question:
Why is it that all our other sites that run on this environment does not require this (machineKey in the web.config).
I checked event logs to make sure that we are not having the same issue on other sites and everything looks fine. I also confirmed that the app pool is not recycling too often and everything was normal with regards to app pool settings.
The only explanation I can come up with is that the site is rendered by one node and subsequent post backs go to another node - which would leave me to believe that the problem lies with the load balancer. Our infrastructure guys tells me that everything is as it should be with regards to the load balancer and the scenario that I am proposing will not happen.
Am I missing the obvious here or are there anything else that I can consider?
Thanks in advance
Basically, yes, you're right - you generally see this in a web farm environment when "Sticky Sessions" aren't properly configured in the load balancer, and the users postback is sent to a different server.
To be fair to your network guys, it's possible that most requests are being sent to one server, but that this application is tipping the usage such that requests are often sent to another server - but you should be seeing that across all sites, unless the traffic patterns are completely different.
The other possible cause is that either your page is taking too long to load, and the users are posting back before the page has completely finished loading - I'd managed to get one of my sites doing that with a couple of remote advert calls buried halfway through the page load, or the users are waiting too long between page render and postback and the session on the loadbalancer is timing out so it thinks it's a new request.
If you are working with a web-farm environment, machine key values, if specified in the web.config need to be synced. In addition, you will want to make sure that the machine key values in the machine.config file are also synced between the two.