I have an ASP.NET web application where a Microsoft Media Player object on the page (in IE) is issuing a request for an .aspx web page. In the page I use TransmitFile to send back the audio file. This works fine most of the time.
But, in some cases (a combination of IE version and a specific client, at least from what I can see) there is a second request issued, with the exact same URL. The only difference I can see between the first and second request is the user-agent value. The first request will have User-Agent: Windows-Media-Player/9.00.00.4508 and the second one will have User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
This second request is causing the audio file to be sent a second time over the net, which is wasteful. This is what I'm trying to avoid.
I had a related question here, but in this case there is no Range request. It is just the same exact request again (same headers, except for the user-agent).
I was trying to suppress the second response (based on the user-agent in the header) with all kind of HTTP status responses (304, 404, 500 etc.) This works for some clients, some of the time, but breaks occasionally (the Media Player will just not play the audio, even though Fiddler will show it was transfered on the first request).
I would like to "convince" the browser to avoid the second request, if possible. As a second option I would like to find a response to the second request that will not break the playback, but avoid sending the whole audio buffer.
The only thing I can think of so far is that maybe they have a plugin/toolbar installed that is trying to do something when it detects media.
I think VLC does something like that for Firefox, and I know there are many 'video downloader' kind of addons for Firefox, maybe there are equivalents made for IE.
You could try asking the client that is having the problems to give you their list of addons (Should be Tools -> Manage Add-ons in IE8).
Hope this helps : )
EDIT: One more thing you could check is to ask them to try enabling compatibility mode on your site and see if it changes anything.
Please provide full headers dump, are you sure both requests are GET ones?
First request might be just to check if the cached version is good enough.
Second thought: try to return 'expires' header to supress cache check.
Related
While it is important that people should use approved browsers downloaded from their legitimate sites to use them, is there any way for the server to detect if someone is spoofing the browser (user agent)?
My question is in particular reference to security. What if someone creates a browser (user agent) and does not respect some contracts (for example, Same origin policy of cookies) to exploit vulnerabilities there? This illegitimate browser can claim that it is a genuine user agent by populating the User-Agent header with standard values used in Firefox or Chrome.
Is there any way at the server side to detect if the user is using spoofed user agent so that the server can take counter measures if needed? Or is this the absolute responsibility of the individual using the browser to use approved browsers only (servers have no way to detect it)?
Browsers are just high level user interfaces to HTTP. Prior to the introduction of various security methods there was not much in place to prevent such attacks. Nowadays browsers (Chrome, Firefox, Edge) have restrictions and abide by certain rules/contracts (in order to work properly).
One can spoof(send) anything with a HTTP client (a very lean
"browser") such as CURL.
curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" http://blah.nonexistent.tld
(The default curl header is something like User-Agent: curl/7.16.3)
A server could restrict or detect uncommon User-Agents to prevent scraping or scanning, but the user agent is nothing but a "name" and could just be changed to a common one as done above.
The security methods (contracts) that have been added such as Same Origin Resource Policy / Cross Origin Resource Sharing / HTTP Only are there to protect the client (browser) and server. They must be implemented by both in order to function properly(securely) to protect against an attack as mentioned. If the client and the server don't properly use the contracts agreed upon, then cookies could be exfiltrated (A modern browser is designed to fail fast and would still prevent this).
If you meant that you were to create your own browser set its User-Agent as Chrome, ignore contracts in place by properly configured servers then they might ignore you. What user cookies would you steal from a "custom" browser, that few people may use?
Recently I put some hidden links in a web site in order to trap web crawlers. (Used CSS visibility hidden style in order to avoid human users accessing it).
Any way, I found that there were plenty of HTTP requests with a reference of browsers which have accessed the hidden links.
E.g : "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31"
So now my problems are:
(1) Are these web crawlers? or else what can be?
(2) Are they malicious?
(3) Is there a way to profile their behaviour?
I searched on the web but couldn't find any valuable information. Can you please provide me some resources, or any help would be appreciated.
This is a HTTP user agent. They are not malicious at all. It's following the pattern, for example Mozilla/<version> and so on. A browser is a user-agent for example. However, they can be used by attackers and this can be identified by looking at anomalies. You can read this paper.
The Hypertext Transfer Protocol (HTTP) identifies the client software
originating the request, using a "User-Agent" header, even when the
client is not operated by a user.
The answer to your questions are, in order:
They are not web crawlers. They are user agents. Common term for a web developer.
Generally they aren't malicious but they can be, as I suggest, look at the paper.
I don't understand what you mean by profiling behaviour, they aren't malware!
Here is the scenario:
We have multiple clients,dealers and staff in our application. this problem occurs only at dealer's area. and it works fine on some of the dealer's machine. but two of our dealers are facing this problem. these dealers are very big banks inside UK. When we tried our dealer's login on our machines everything worked fine. this is just happening to two specific dealers. what could be the reason for this? and how to avoid this? i have searched alot on internet but nothing worked for me. kindly give a solid reason and explanation. problem occurs only at IE.
when they click on a specific button , they see this exception. on that button we just open an aspx page in iframe. details of the error are given below.
User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.2; .NET4.0C; .NET4.0E)
Timestamp: Thu, 17 Nov 2011 14:19:14 UTC
Message: Sys.WebForms.PageRequestManagerParserErrorException: The message received from the server could not be parsed. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.
Details: Error parsing near 'uamFuX1R5cGU9amF2YV9zY3JpcHQmRmluamFuX0x'.
Line: 5
Char: 62099
Code: 0
URI: http://xyz.com/ScriptResource.axd?d=_FIiVFNdF1PHkbuLKG5hopSmLy4o3JvRIyD6vVyYwvpDZR7-f336pr-a6hLEOPIccb7DRK78POXYTQfl9EZSx4SxizvUioc19B1P43shEyWowLvhIGL3AeK1wy_YyeW1GriC7BqWtcuIU_bsb1M41M4Otm81&t=ffffffffe783cd7f&sfgdata=+sfgRmluamFuX1R5cGU9amF2YV9zY3JpcHQmRmluamFuX0xhbmc9dGV4dC9qYXZhc2NyaXB0+a
I was looking around on the net and found a similar question asked on StackOverflow - ASP.NET Ajax Error: Sys.WebForms.PageRequestManagerParserErrorException
There seem to be a number of things that could be causing this error, and this article might be a good helper to point you in the right direction. The article lists things to avoid when you are getting this error:
Calls to Response.Write():
Place an or similar control on your page and set its Text property. The added benefit is that your pages will be valid HTML. When using Response.Write() you typically end up with pages that contain invalid markup.
Response filters:
The fix might just be to not use the filter. They're not used very often anyway. If possible, filter things at the control level and not at the response level.
HttpModules:
Same as response filters.
Server trace is enabled:
Use some other form of tracing, such as writing to a log file, the Windows event log, or a custom mechanism.
Calls to Server.Transfer():
I'm wondering if there are any utilities out there that will display the request/response headers sent/received by my web browser during a browsing session. Does anyone know of anything useful?
I'm familiar with the Modify Headers add-on for Firefox 4 and the HTTP Client utility for MacOSX but neither of these do quite what i'm looking for.
I suspect Fiddler might help here - it captures all of the traffic, including headers, content, etc. It works on startup with IE or Chrome; Firefox needs to be configured to use it as a web proxy.
Update:
Looks like the header request information is the culprit. How would I change the max-age property of the request header? TIA.
Hi, I'm using #font-face on a website and i'm experiencing delayed loading of the text (presumably due to the loading of the font every page). I understand the client has to download the font once to display properly, but every page?
Is there a way I can force the browser to cache that file? Or is there another alternative to speed up the font's loading time? (Is this a question more appropriate to post on Server Fault?)
Thanks in advance. Worst case, I'll live with the delay, so I don't need any "take off #font-face" answers... ;)
Additional Information:
I've tested this in both Safari (4) and Firefox (3.5RC1) on both Mac and Windows (XP and 7)
All the browsers I've tested on are currently set up to allow caching (it's on by default)
The URL is not dynamic, it's simply "/fonts/font.otf"
The font URL is correct, as the page loads the font and displays it correctly, albeit slower then normal
Request Header :
Cache-Control:max-age=0
If-Modified-Since:Wed, 24 Jun 2009 03:46:28 GMT
If-None-Match:W/"484d9f2-a5ac-46d10ff2ebcc0"
Referer:http://testurl.com/
User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/530.13 (KHTML, like Gecko) Version/4.0 Safari/530.15
Response headers:
Connection:Keep-Alive
Date:Thu, 25 Jun 2009 02:21:31 GMT
Etag:"484d9f2-a5ac-46d10ff2ebcc0"
Keep-Alive:timeout=10, max=29
Server:Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
You can never force a browser to cache something, only encourage it. I can think of no reason why a font file with the correct expires headers wouldn't be cached which brings us to:
It's a browser bug (you don't say which browser)
Your cache control headers are missing or wrong
Your browser is configured to not cache anything (do images cache?)
Your font URL is dynamic so the browser thinks each request is for a different resource
The font face file is actually missing or or the URL misspelt.
The delay is NOT caused by the font download (you did say you presume this is the issue)
I think more information is in order.
EDIT: To set cache control is a server and language specific thing. Look at mod_expires for information on caching in Apache.
Are you sure your font files are cachable? Just like other static content, they should have far-future expires dates, and their headers should be configured to allow them to be cached. If you are hosting your fonts on a server farm, you will want to make sure your etag header is normalized across all the servers in the farm...otherwise subsequent requests for the font may force it to be re-downloaded from an alternative server even though the same data was already downloaded from another server.