Using the HttpClient/WebRequest API in C# to build a REST API client. I've changed the underlying RequestCachePolicy to allow it to cache responses that have the appropriate HTTP cache control headers. This is working great.
However, I often want to clear the cache during testing. Everything I've read says that
The underlying implementation is the WinInet cache.
You can clear it via Internet Explorer.
On Win7 the cache is stored in C:\Users\namehere\AppData\Local\Microsoft\Windows\Temporary Internet Files
When my code is deployed to IIS in a middle-tier scenario, and the HTTP client is executing from there, I've found that
The underlying implementation is still the WinInet cache.
Internet Explorer doesn't clear it.
It's stored in C:\Windows\Temp\Temporary Internet Files\Content.IE5\
I was able to use System.Net tracing to monitor the client side cache behavior.
My question is how to clear this Content.IE5 cache in a non-programmatic way. Certainly I can go in and delete the files by hand, but its a hassle with all the alphanumeric folders under Content.IE5, etc.
Related
We have a .NET WPF container app in which we host several web apps using CEFSharp.WinForms control. At times, we see that for some users, some JavaScript resource requests fail with the ERR_CONTENT_DECODING_FAILED error message. This issue gets resolved if we reload the app after either clearing the CEF cache or after disabling the cache from the network tab in the developer toolbar window. Please note that this issue isn't confined to a specific subset of resource files: instead, we have seen it happening sporadically for a variety of JavaScript resource files (some hosted on Apache while the others hosted on IIS servers).
While a possible cause for usual ERR_CONTENT_DECODING_FAILED error is a server-side content-encoding issue, in this specific case, we believe this could potentially be related to the CEF browser caching. Please see the analysis section below for the reasons we believe so.
Application Setup
When we initialize CEF settings, we set MultiThreadedMessageLoop setting to true and set CachePath property to a location under %localappdata% on windows 10 machine. When the container app starts, it creates three CEF web browser controls and launches web apps in them. All three apps load concurrently. After that, more CEF web browsers are created as the user visits more apps. The user also reloads some of these apps over time. All the web apps are internal apps sharing the same domain but physically hosted on different web servers. The JavaScript resource files in question usually have caching policy set to allow them to be cached for a week.
CEFSharp version - 79.1.360.0
CEF version - r79.1.36+g90301bd+chromium-79.0.3945.130
Chromium version - 79.0.3945.130
Our Analysis so far
We checked the web-server logs for the failing JavaScript resources. We observed that in most cases, the server requests for those resource files (by the impacted user) were made a few days ago. The users are usually able to use the application well for some days before they sporadically start getting this error.
We checked the network logs (*.HAR file). We see that for the failing JavaScript resource, _transferSize is 0 (which seems to indicate that response was served from the cache)
When the error occurs, it gets resolved when we reload the app after either clearing the cache or disabling the cache from the network tab.
We tried artificially simulating this error. We used Fiddler's autoresponder feature to deliberately respond with a bad server response (the content was 'gzip' encoded however Content-Encoding header indicated 'br'). We could simulate the ERR_CONTENT_DECODING_FAILED error, however, we could see in network logs that _tranferSize was a non-zero value. We also observed that chrome did not cache the bad response. This test indicates that when the original JavaScript response was cached by the browser, it must have been a correctly encoded response, or else the browser would not have cached it.
All of the above points lead us to believe that, JavaScript resource files were downloaded (with correct encoding) and cached in CEF cache. The user was also able to use the apps for some time. After that however, in certain scenarios, some of these files potentially got corrupted in CEF cache, leading to the content decoding error.
We tried using CEF response filter mechanism as explained here to capture the bad response when content decoding error occurs. Unfortunately, we observed that dataIn stream which gets passed to filter function is null when the response fails with this error.
Summary and Questions
This is a sporadic issue which our users are facing. We haven't found a way to deterministically recreate this problem. However based on our analysis so far, we believe some JavaScript files may be getting corrupted in CEF cache over time. We are not sure if the fact that we host several CEF web browsers and load them concurrently could be playing some role in causing this issue.
Has anyone else observed/reported a similar issue? Do you have any idea if we are missing or overlooking something here or going in the wrong direction? Any pointers will be greatly appreciated.
I tried out different WebDAV clients and all of them behave a little different. In this special case I use WinSCP to connect to a SabreDAV based, self written (in php), WebDAV server which emulates a filesystem.
The problem I have is that WinSCP creates - unlike several other WebDAV clients (eg BitKinex, CarotDAV) - a new php session each time I do a command (refresh, upload file, ...).
So is there a possibility to ensure reusing session (programmatically on server side) or do I need to set a specific option in WinSCP? If the latter one, I could not figure out so far.
WinSCP indeed does not process HTTP cookies.
I've never seen a WebDAV server that would require it. The cookies are definitely not mentioned in WebDAV specifications. WebDAV servers that need to authentication or track its users use HTTP authentication. Not cookies.
Anyway, I've added a request for processing cookies to WinSCP tracker:
https://winscp.net/tracker/1419
Well, I implemented a workaround that allows me to mostly reuse same session every time.
Hopefully WinSCP will process cookies one time though to be sure.
What is the difference between kernel mode caching and user mode caching and how to track both ?
Kernal Mode caching is essentially going to handle caching requests at the OS-level, so contents that are stored in it can be accessed without ever going down the rest of the usual pipeline (i.e. it will not have to go down to the ASP.NET or IIS-level caches to check for the contents) :
So the request hits the initial cache (http.sys), finds what it needs and sends it back, all without ever having to proceed further down the pipeline.
As a the result of this, it's usually quite fast. A limitation of it however is that it does not support many user-level features such as authentication and authorization, so it may not fit all scenarios.
User-mode on the other hand is going to fill in the gaps where Kernal-mode cannot be used, which primarily surrounds authorized/authenticated content (as it requires a check to see if the user can actually access the contents), but there are many other scenarios that could cause the http.sys cache to not be used.
With regards to actually checking to see if content is or is not being cached (and possibly why), you can use FREB (Failed Request Event Buffering). The following command can be used to find out which content is cached in kernel mode:
netsh http show cachestate
Using the browser cache offers the big advantage of saving a lot of traffic and making your site faster. However, the big disadvantage is that cached resources cannot be "uncached", until they expire and the browser requests the resources again. Or can they?
Is there a way to explicitly tell the browser (in a seperate request or JavaScript, etc.) to uncache a certain resource?
I know of appending version strings like image.jpg?12342, but I'm looking for a more elegant alternative.
E-Tags are a cool thing, but they don't really cache. We may save sending the actual resource payload, but the browser still does a request.
You might want to check out a cache manifest especially if you don't have access to the server.
Yes it is primarily used to "cache" files for using web apps offline however, you can also expliticly declare certain URI's to be non-cached and even versioning your cache manifest will trigger a reload of noted URI's.
CACHE MANIFEST
# Version x
NETWORK:
uri-path.here
http://html5doctor.com/go-offline-with-application-cache/
For a current project, I was thinking of implementing WebDAV to present a virtual file store that clients can access. I have only done Google research so far but it looks like I can get away with only implementing two methods:
GET, PROPFIND
I think that this is great. I was just curious though. If I wanted to implement file uploading via:
PUT
I haven't implemented it, but it seems simple enough. My only concern is whether a progress meter will be displayed for the user if they are using standard Vista Explorer or OSX Finder.
I guess I'm looking for some stories from people experienced with WebDAV.
For many WebDAV clients and even for read only access, you will also need to support OPTIONS. If you want to support upload, PUT obviously is required, and some clients (MacOS X?) will require locking support.
(btw, RFC 4918 is the authorative source of information).
I implemented most of the WebDAV protocol in about a day's work: http://github.com/nfarina/simpledav
I wrote it in Python to run on Google App Engine, and I expect any other language would be a similar effort. All in all, it's about two pages of code.
I implemented following methods: OPTIONS, PROPFIND, MKCOL, DELETE, MOVE, PUT, GET. So far I've tested Transmit and Cyberduck and both work great with it.
Hopefully this can provide some guidance for the next person out there interested in implementing a WebDAV server. It's not a difficult protocol, it's just very dense with abstracted language like 'depth' and 'collections' and blah.
Here's the spec: http://www.webdav.org/specs/rfc4918.html
But the best way to understand the protocol is to watch a client interacting with a working server. I used Transmit to connect to Box.net's WebDAV server and monitored traffic with Charles Proxy.
Bit late to the party, but I've implemented most of the webdav protocol and I can tell with confidence you'll need to implement most of the protocol.
For OS/X you'll need class-2 WebDAV support, which includes LOCK and UNLOCK (I found it particularly difficult to fully implement the http If: header, but for Finder you'll only need a bit of that.)
These are some of my personal findings:
http://sabre.io/dav/clients/windows/
http://sabre.io/dav/clients/finder/
Hope this helps
If you run Apache Jackrabbit under, say, Tomcat, it can be configured to offer WebDAV and store uploaded files. Perhaps that will be a useful model, or even a good enough replacement for the planned implementation.
Apache Jackrabbit Support for WebDAV
Also, you may want to be aware of the BitKinex client (free 30 day trial), which I have found to be a useful tool for testing a WebDAV server.
BitKinex Home Page
We use WebDAV internally to provide a folder-based view of some file shares to clients outside of our firewall. We're using IIS6 for this.
Basically, it boils down to creating a Virtual Directory in IIS that maps to each network file system that you want to make available via WebDAV. Set it up with the content coming from "A share located on another computer" -- use the UNC path to the share for the Network Directory value. We turn on all options except Index this resource. Disable all default content pages. Turn on Windows Integrated Authentication (ours is set up using SSL as well). I have the root set up to deny access to anonymous and allow access to any authenticated user. We also have a wildcard MIME mapping (.* to application/octet-stream). Enable the WebDAV web service extension in IIS. You also need to set up the web server to delegate permissions to all the file servers you may be accessing so it can pass on the user's credentials.
If you have Macintosh clients you may also need an ISAPI filter that maps 401 to 403 errors for Darwin clients. Microsoft and Apple disagree on how to handle the situation when you don't have permission to write to a directory. Apple keeps resending the credentials on a 401 (Access Denied) error, translating it to a 403 (Forbidden) error keeps this from happening. By default Apple likes to write a "dot" file to every directory it accesses. Navigating through directories where you don't have write access will end up crashing the Finder if you don't have the filter. I have source code for this if needed.
This is all off the top of my head. It's possible (probable?) that I may have missed something. Feel free to contact me via the contact information on my web site if you have problems.
We have a webDAV servlet on our web based product.
i've found Apache Jackrabbit a good help for implementing it. however webDav is a serious P.I.T.A on the client side support.
many client implementation differ widely in their behavior and you most likely will have to support several different kinds of bugged implementations.
some examples:
MS vista only supports authentication over SSL
most windows based webDAV client assume your webdav-server/let is a sharepoint server and will act accordingly (thus not according to the webDAV protocol)
one example of this is that you NEED to allow and Unauthenticated LOCK request on the root of your server (ie yourdomain.com/ not yourdomain.com/where/webdav/should/live) else you wont be able to get write acces in MS windows.
(this is a serious P.I.T.A on a tomcat machine where your stuff usualy lives in server.com/servlets/paths/thelocation)
most(all?) versions of MS office respond different to webdav links.
i guess my point is integrating webdav support into an existing product can be a LOT harder then you would expect. and if possible i would advice to use a (semi)-standalone webDAV server such as jackrabbit webdavServer, or apache mod_webdav
I've found OS X's Finder WebDAV support to be really finicky. In order to get read-write support, you have to implement LOCK, in addition to other bits.
I wrote a WebDAV interface to a Postres database, where python modules were stored in the database in a hierarchical folder-like structure. Accessing it with cadaver worked fine, and IIRC a GUI windows browser worked too, but Finder refused to mount the share as anything other than read-only.
So, I don't know if if would give a progress bar. The files I was dealing with were small enough that a read/copy from them was virtually instantaneous. I think a copy of a large file using the Finder would probably give a progress bar - it does for any other type of mounted share.
Here is another open source project for WSGI WebDAV
http://code.google.com/p/wsgidav/
where I picked up the PyFileServer project.