asp.net webservice handling gzip compressed request - asp.net

I have an asp.net .asmx webservice written to handle requests from a third party tool. The third party tool makes an http POST request to the webservice to get user information. I'm using IIS7
Running Fiddler with "Remove All Encodings" checked, I can see the webservice call and and everything functions properly. If I uncheck "Remove All Encodings", the webservice call fails with a 400 Bad Request. The difference I see is that the header "Content-Encoding: gzip" is being removed by Fiddler and the content is being decompressed.
So, when the Content-Encoding header is removed and the content is decompressed, my webservice functions perfectly. When the header is present and the content is compressed, the webservice fails.
How can I either:
Configure my webservice to tell the client that it won't accept compressed requests (and hope that the third party tool respects that)
Decompress the content early in the asp.net handling
Modify my webservice to work with compressed data
Update: To be clear, I don't need to configure gzip encoding in the Response, I need to deal with a Request TO my webservice that is gzip encoded.
Update 2: The third-party tool is the Salesforce.com Outlook plugin. So, I don't have access to modify it and it is used by many other companies without trouble. It's got to be something I'm doing (or not doing)
Update 3: I found one post here that says that IIS does not support incoming POST requests with compressed data, it only supports compressed Responses. Can this still be true?

The simplest technique is to create an HttpModule that replaces the request filter. It is more reusable and avoids having a Global.asax. There is also no need to create a new decompress stream class as the GZipStream is ready for that. Here is the full code, that also removes the Content-Encoding: gzip that is not needed any more:
public class GZipRequestDecompressingModule : IHttpModule
{
public void Init(HttpApplication context)
{
context.BeginRequest += (sender, e) =>
{
var request = (sender as HttpApplication).Request;
string contentEncoding = request.Headers["Content-Encoding"];
if (string.Equals(contentEncoding, "gzip",
StringComparison.OrdinalIgnoreCase))
{
request.Filter = new GZipStream(request.Filter,
CompressionMode.Decompress);
request.Headers.Remove("Content-Encoding");
}
};
}
public void Dispose()
{
}
}
To activate this module, add the following section into your web.config:
<system.webServer>
<modules runAllManagedModulesForAllRequests="true">
<add name="AnyUniqueName"
type="YourNamespace.GZipRequestDecompressingModule, YourAssembly"
preCondition="integratedMode" />
</modules>
</system.webServer>

Since the 3rd party service is just sending you a POST, I do not think that it is possible to tell them not to send in compressed.
You could try to override GetWebRequest and decompress it on the way in
public partial class MyWebService : System.Web.Services.Protocols.SoapHttpClientProtocol
{
protected override WebRequest GetWebRequest(Uri uri)
{
base.GetWebRequest(uri);request.AutomaticDecompression = System.Net.DecompressionMethods.GZip;
return request;
}
}

GZIP compression is a function of the server.
If you're using IIS6, consult this link.
If you're using IIS7, you could use ISAPI_Rewrite to disable gzip. See this link.
That said, because gzip is a function of IIS, you really shouldn't need to do anything "special" to get it to work with a web service (IIS should handle decompressing and compressing requests). Hopefully this info will get you further down the road to troubleshooting and resolving the issue.

I am not sure that IIS supports decompressing incoming requests, so this might have to be done further down the pipe.
Shiraz's answer has the potential of working and it would be the first thing I would try.
If that doesn't work you might consider switching your server .asmx service to WCF, which while a bit more difficult to setup it also gives more flexibility.
On the WCF side there are two things I can suggest. The first is quite easy to implement and is based on setting the WebRequest object used by WCF to automatically accept compression. You can find the details here. This one is the WCF equivalent to the solution proposed by Shiraz.
The second is more complicated, since it involves creating Custom Message Encoders, but if none of the above methods work, this should solve the problem. Creating a message compression encoder is described here. You might also want to check the answer in here which presents a sample config for the message encoder.
Please let me know if this helped or if you need more help.

I've found a partial answer here.
class DecompressStream : Stream
{
...
public override int Read(byte[] buffer, int offset, int count)
{
GZipStream test = new GZipStream(_sink, CompressionMode.Decompress);
int c = test.Read(buffer, offset, count);
return c;
}
...
}
I can then specify the filter on the request object like this:
void Application_BeginRequest(object sender, EventArgs e)
{
string contentEncoding = Request.Headers["Content-Encoding"];
Stream prevCompressedStream = Request.Filter;
if(contentEncoding == null || contentEncoding.Length == 0)
return;
contentEncoding = contentEncoding.ToLower();
if(contentEncoding.Contains("gzip"))
{
Request.Filter = new DecompressStream(Request.Filter);
}
}
I say partial answer because even though I can now process the incoming request, the response is getting a "Content-Encoding: gzip" header even though the response is not encoded. I can verify in Fiddler that the content is not encoded.
If I do encode the response, the client for the webservice fails. It seems that even though it is sending "Accept-Encoding: gzip", it does not in fact accept gzip compressed response. I can verify in Fiddler that the response is compressed and Fiddler will decompress it successfully.
So, now I'm stuck trying to get a stray "Content-Encoding: gzip" header removed from the response. I've removed all references I can find to compression from the application, the web.config, and IIS.

Related

ASP.Net Core 2.0 - ResponseCaching Middleware - Not Caching on Server

I want to use server-side response caching (output cache) with asp.net core 2.0 and found out about Response Caching Middleware and wanted to give it a try with a brand new asp.core mvc project.
Here is the description from the link above which makes me think this could be used like output cache.
The middleware determines when responses are cacheable, stores responses, and serves responses from cache.
Here is how my startup.cs looks like.
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddResponseCaching();
services.AddMvc();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseResponseCaching();
if (env.IsDevelopment())
{
app.UseBrowserLink();
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();
app.UseMvc(routes =>
{
routes.MapRoute(
name: "default",
template: "{controller=Home}/{action=Index}/{id?}");
});
}
}
and here is the HomeController.cs
[ResponseCache(Duration = 60)]
public class HomeController : Controller
{
public IActionResult Index()
{
return View();
}
public IActionResult About()
{
ViewData["Message"] = "Your application description page.";
return View();
}
public IActionResult Contact()
{
ViewData["Message"] = "Your contact page.";
return View();
}
public IActionResult Error()
{
return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
}
}
there is also a timestamp at the bottom of _Layout.cshtml file so i can tell when the page is rendered, like below.
<p>© 2018 - ResponseCachingMiddleware - #DateTime.UtcNow</p>
Cache-Control headers seem to be fine, this is what I get in headers when I load the page but time stamp keeps getting updated on every refresh every second.
Cache-Control:public,max-age=60
What I'm understanding from MS documentations is Response Caching Middleware is the server-side caching mechanism that takes care of caching the response while Response Caching seems to be just a filter to manipulate response headers for caching.
Can't tell if there is something wrong with my understanding or code and I wanna complain that I'm feeling this way too often since I started prototyping with ASP.Net Core. Maybe you could also suggest better resources as a side topic.
I've checked out this post before
ASP.NET Core 2.0 - Http Response Caching Middleware - Nothing cached
Also checked this out but it seems like the only difference is I'm using mvc.
https://github.com/aspnet/ResponseCaching/blob/dev/samples/ResponseCachingSample/Startup.cs
Thanks
Edit: I'm seeing the message below in the output window, cannot find anything about it on google except the few places I already checked for response caching middleware.
Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware:Information:
The response could not be cached for this request.
Note: I wish I could create #response-caching-middleware tag. Not sure #responsecache is relevant.
I had the same issue, I was about to pull my hairs over it, I'd set app.UseResponseCaching(); as well as services.AddResponseCaching(); and add ResponseCache on top of my action exactly like what was told in Microsoft official Docs, despite the the cache-controll header was set correctly on response returning from server but still nothing cached at server-side.
After couple of hours of sweating on this issue I figured out where the problem arises and why nothing cached at server.
Browsers by default set cache-controll value to max-age=0 for the request (if the request is not caused by back or forward) even though you set cache-controller correctly in your response by adding ResponseCache attribute on top of you action (or controller) since the cache-controller sent by request is set to max-age=0, the server is unable to cache response, I think this must be added to list of Response Caching limitation as well
Anyway you can override browser default behavior by adding few line of code right before calling app.UseResponseCaching(); on the other hand you need to add a custom middle-ware to modify request cache-control header value before calling app.UseResponseCaching();.
See code below, worked for me hope work for you too
app.Use(async (ctx, next) =>
{
ctx.Request.GetTypedHeaders().CacheControl = new Microsoft.Net.Http.Headers.CacheControlHeaderValue()
{
Public = true,
MaxAge = TimeSpan.FromSeconds(60)
};
await next();
}
);
app.UseResponseCaching();
for ensuring that ResponseCaching works as expected you can also use postman but you must set 'Send no-cache Header' to off in the setting, see image below
I had this same confusion recently.
ASP.Net Core's ResponseCaching does provide both client-side caching (through HTTP response headers) & server-side (through a memory cache'd middleware that short-circuits other middlewares if the response is in the cache). The server-side portion reads the HTTP response cache headers to determine if it should do server-side caching (similar to what an ISP or CDN might do).
Unfortunately, debugging the server-side ResponseCaching is tricky because it has weird rules & there's not adequate logging. In my case I pulled down Microsoft's source code to step through it & find the issue with my code.
The note you found in the output window "The response could not be cached for this request" is a clue.
There's 2 parts to the server-side caching of a request. The server has to prime the cache the first time the url is requested. It will serve the cached version the 2nd time it's requested. Pay attention to when the error message shows up, if it's on the 1st or 2nd request. That'll tell you if it couldn't be stored in the cache or if it couldn't be retrieved from the cache.
The rules for both storage & retrieval are in this source code file:
https://github.com/aspnet/ResponseCaching/blob/3bf5f6a1ce69b65c998d6f5c739822a9bed4a67e/src/Microsoft.AspNetCore.ResponseCaching/Internal/ResponseCachingPolicyProvider.cs
Your "Cache-Control:public,max-age=60" header should match these rules just fine.
My guess is you actually had it working, but didn't know how to test it correctly.
There is a counter-intuitive portion of ResponseCaching noted in this issue: https://github.com/aspnet/Home/issues/2607
Essentially, if the browser sends a no-cache or no-store header (when you hit CTRL+F5 or have your debugger tools open), ASP.Net Core's ResponseCaching will honor the browser's request & re-generate the response.
So, to test if your code was working you probably loaded the page, which primed the cache, then you hit CTRL+F5 to force-refresh your browser & you expected the server-side to respond with a cached entry rather than running your WebAPI code. However, it honored the no-cache request header & bypassed the cache (& wrote that message in your output log).
The way to test this would be to clear your browser cache in-between requests (or switch to incognito), rather than using CTRL+F5.
On a side note, honoring the no-cache/no-store request headers was probably a poor design choice since ASP.Net Core's ResponseCache will most likely be used by a server who owns the response, rather than an intermediary cache like a CDN/ISP. I've extended the base ResponseCache with an option to disable honoring these headers (as well as serialize the cache to disk, rather than in-memory only). It's an easy drop-in replacement for the default cache.
You can find my extension here:
https://github.com/speige/AspNetCore.ResponseCaching.Extensions
https://www.nuget.org/packages/AspNetCore.ResponseCaching.Extensions
There are also a few other other gotchas with ResponseCaching to watch out for which you may have already read about in the blog urls you posted. Authenticated requests & responses with set-cookie won't be cached. Only requests using GET or HEAD method will be cached. If the QueryString is different, it'll make a new cache entry. Also, usually you'll want a "Vary" header to prevent caching if certain conditions of a request differ from the previously-cached request (example: user-agent, accept-encoding, etc). Finally, if a Middleware handles a request it'll short-circuit later Middlewares. Make sure your app.UseResponseCaching() is registered before app.UseMVC()
If the Cache-Control header is coming through, then it's working. That's all the server can do from that perspective. The client ultimately makes the decision whether or not to actually cache the resource. Sending the header doesn't force the client to do anything; in fact, the server, in general, cannot force the client to do anything.

Securing http headers

I Have website that is in production server and it supposed to be very secure so i want to secure http header so that no unwanted information is leaked.
I have searched on net about securing http headers and so far found that we can remove un anted information like removing
'Server Microsoft-IIS/7.5
X-AspNet-Version 4.0.303319
X-Powered-By ASP.NET -'
I have found solution for X-Aspnet and X powered by :
1. For X-AspNet i have added below code in system.web section
<httpRuntime enableVersionHeader="false"/>
For X-Powered i have added below code in system.webserver section
But for Server header removal code is not working :(
Code i am using for is :
I have added a class with name CustomHeaderModule and inside that class code is as below
///
/// Summary description for CustomHeaderModule
///
public class CustomHeaderModule : IHttpModule
{
public void Dispose()
{
throw new NotImplementedException();
}
public void Init(HttpApplication context)
{
context.PostReleaseRequestState += PostReleaseRequestState;
}
void PostReleaseRequestState(object sender, EventArgs e)
{
//HttpContext.Current.Response.Headers.Remove("Server");
// Or you can set something funny
HttpContext.Current.Response.Headers.Set("Server", "CERN httpd");
}
}
and then registered this in web.config under system.webserver section
<modules runAllManagedModulesForAllRequests="true">
<add name="CustomHeaderModule" type="CustomHeaderModule" />
</modules>
Now this code is not working ..i am still seeing server in header in chrome browser..
how can i fix this and apart from these 3 setting is there any other to secure more ?
Considering your problem what I would suggest you is to use ASafaWeb to test your Website!
Second is to read these articles from Troy Hunt and Paul Bouwer:
Shhh… don’t let your response headers talk too loudly
Clickjack attack – the hidden threat right in front of you
ASafaWeb, Excessive Headers and Windows Azure
Following this articles you will finally have a look at NWebSec!
Sorry if this doesn’t answer your question directly but I wouldn’t really bother removing those headers. Someone can easily find out what server are you using by looking at the html code on the browser side.
If I look at source code and I see things like __VIEWSTATE I’ll immediately know this is ASP.NET and if I dig a little deeper I’ll probably be able to figure out the version too.
What I’d suggest is that you focus on standard security and risk procedures such as making sure you are not open to SQL injections, validating everything on the server side, making sure you have all backups in place and ready to be up in several mins, adding additional layer of authentication if needed, making sure you have all security updates on the server and such…
I have found one solution which works on IIS but not on local but i am okay with that...Removing/Hiding/Disabling excessive HTTP response headers in Azure/IIS7 without UrlScan
anyways apart from these 3 settings ..is there any other way i can more secure http headers..

StaticFileHandler that supports caching of VirtualPathProvider served content

I have a library of reusable partial views, scripts and images that are embedded in assembly and shared between projects.
Everything works fine, I've modified web.config to make all necessary file types to be served by System.Web.StaticFileHandler, but unfortunately, it serves all resources with Cache-Control: private.
I can write my own StaticFileHandler that would serve VPP content with Cache-Control: public and expiration date.
How do I implement caching support using VirtualPathProvider.GetCacheDependency?
I figured out why this happens. I looked at the source for the StaticFileHandler. For embedded files, it doesn't set any of the caching headers. It only does for files in the file system. Meaning this will never work right.
You have two options.
1.Find another http handler. I have never used this but it has come up in my searching: https://code.google.com/p/talifun-web/wiki/StaticFileHandler
2.Create an http module that checks to see if the static file handler was used, if so set the caching details.
Good luck.
Based on my researches, I found a hacky way to achieve caching.
In your VPP implementation, you should be implementing your own VirtualFile class, extending System.Web.Hosting.VirtualFile base class. It just expects a stream to read file if it is needed from VPP. At that phase, you can inject headers and even change cachability of resource. Because priorly, if I request a static file from VPP, it was coming with a header Cache-Control:private. Actually, server was saying that: I do not care your local caches, etags and so on. I decide whether you should cache it or not. The code below changes it to public and add required e-tag header so that it should stay at cache unless that assembly is changed:
class EmbeddedResourceVirtualFile : VirtualFile
{
readonly EmbeddedResource embedded;
public EmbeddedResourceVirtualFile(string virtualPath, EmbeddedResource embedded)
: base(virtualPath)
{
this.embedded = embedded;
}
public override Stream Open()
{
var assemblyLastModified = embedded.AssemblyLastModified;
var etag = assemblyLastModified.Ticks;
var response = HttpContext.Current.Response;
var cache = response.Cache;
cache.SetCacheability(HttpCacheability.Public);
cache.SetETag(etag.ToString());
cache.SetLastModified(assemblyLastModified);
cache.SetExpires(assemblyLastModified.AddYears(2));
return embedded.GetStream();
}
}
Special thanks for the commentor :)

HttpModule not receiving cookies on IIS 6

I have an HttpModule that I created and am running on IIS 6 using Windows Server 2003. I can send cookies to the browser, but I can not read them on the next request, cookie is always null.
If I run this module on IIS 7 though, it works just fine. IIS 7 not an option at the moment as we have not switched over and this needs to get done before that will happen.
Also, I've already tried using the PostAcquireRequestState hook.
public void Init(HttpApplication httpApp)
{
httpApp.BeginRequest += OnBeginRequest;
}
public void OnBeginRequest(Object sender, EventArgs e)
{
var httpApp = (HttpApplication)sender;
var context = httpApp.Context;
const string cookieName = "sId";
if (!string.IsNullOrEmpty(context.Request.QueryString["cookie"]))
{
var ck = new HttpCookie(cookieName)
{
Value = httpApp.Context.Request.QueryString["cookie"],
Expires = DateTime.Now.AddDays(1)
};
httpApp.Response.Cookies.Add(ck);
}
else
{
var cookie = httpApp.Request.Cookies[cookieName]
}
}
I ran into a similar problem, but had a different solution, so I thought I'd share, in case it helps someone. I took zengchun's suggestion as well to use some tools to inspect request & response headers. Since I'm using IE, the F12 Dev Tools works great for this. As soon as I saw the response header for the cookie, I noticed the secure flag was set. Sure enough, I had copied code from a production SSL-hosted site to a test site that did not use SSL, so the secure flag on the cookie prevented the code from being able to read it. I updated the web.config to remove requireSSL from the httpcookies node, and my site started working. :)
your code look worked.the problem may be occur in the client-side how to request the next page.you can use the firebug with firefox or the fidder tools that can log your client-side request and see the request whether send cookd value in the request header to the server.
for example
the request headers:
get /1.aspx
.....
Cookie: sId=123 [if the client has a cookie then it will appear in here.]
the response headers:
Set-Cookie: sId=123; expires=Fri, 30-Mar-2012 07:20:23 GMT;
path=/
if the server add cookie to the response,then response it look like the above.
now,i guess the problem in your cook domain or you cookie path is different.
the best method to set cookie is like the follow code:
var ck = new HttpCookie(cookieName)
{
Value = httpApp.Context.Request.QueryString["cookie"],
Expires = DateTime.Now.AddDays(1),
Path="/",
Domain="your domain"
};
good luck.
Thanks to zhengchun I was able to get to the root of the problem. It turns out I was unable to set the cookie using requests to static files. I created .aspx files for my initial requests that redirected to the static files after setting the cookie. My HttpModule could then read the cookie after being set in the .aspx file. Not sure why I need a .aspx file to set the cookie instead of the HttpModule, but this fixed it.

How to drop/discard a Request in ASP.NET?

I'm trying to figure out how to drop/discard a request, I'm basically trying to implement a blocked list of IPs in my app to block spammers and I don't want to return any response (just ignore the request), is this possible in ASP.NET?
Edit: Some of the answers suggest that I could add them in the firewall, while this will certainly be better it's not suitable in my case. To make a long story short, I'm adding a moderation section to my website where moderators will check the posts awaiting moderation for spam (filtered by a spam fitler), I want the IP of the sender of some post to be added to the list of blocked IPs once a post is marked as spam by the moderator, this is why I wan to do it in the application.
Edit: Calling Response.End() returns a response to the user (even though it's empty), the whole purpose of my question was how not to return any response. Is this possible (even out of curiosity)? There's also Response.Close() which closes the socket but it sends notification (in TCP/IP) when it does this, I just wan to ignore as it if was never received (i.e. send nothing to the user)
Thanks
For this type of thing, I would go with an HTTP Module as opposed to doing this with a handler or a page.
Why use a module instead of a handler? An HTTP Module is called earlier in the request pipeline. For these type of blacklisted requests, you want to drop the request as early and fast as possible. This means the earlier in the request you can drop the request, the better.
You can read a little more about Modules vs Handlers here.
Maybe you could use
Response.Clear();
Response.End();
My first thought was what John suggested. The most ideal solution would be to add those IPs to the list of blocked IPs in your firewall if you have access to it, that way your application isn't having to do any processing at all.
EDIT: This might be better implemented as an HTTPModule instead of a handler, but the basic idea holds true. See http://www.kowitz.net/archive/2006/03/08/ihttpmodule-vs-ihttphandler.aspx for more details
You could probably do this using an httphandler, that way you won't have to worry about checking for this in your application code - its handled before your application is even executed.
Psudo code - not tested
class IgnoreHandler : IHttpHandler
{
#region IHttpHandler Members
public bool IsReusable
{
get { return true; }
}
public void ProcessRequest(HttpContext context)
{
context.Response.Clear();
context.Response.StatusCode = 401;
context.Response.Status = "Unauthorized";
context.Response.End();
}
#endregion
}
Obviously, if you want it to return HTTP 200 (OK) and a blank page, just remove the two lines referring to the StatusCode and Status.
And then register it in web.config
<httpHandlers>
<add verb="*"
path="*"
validate="false"
type="MyNamespace.IgnoreHandler, MyAssembly" />
</httpHandlers>
Here you go:
public class RequestDropper : IHttpModule
{
public void Dispose()
{
throw new NotImplementedException();
}
public void Init(HttpApplication context)
{
context.BeginRequest += Context_BeginRequest;
}
public void Context_BeginRequest(object sender, EventArgs e)
{
var request = ((HttpApplication)sender).Context.Request;
if (todayIAmNotInAMoodToProcessRequests)
{
request.Abort();
}
}
}
Hope this saves you some time
I would suggest the best solution is not to give a specific "no response" but to give a more discouraging standard http error message. Whenever I am attempting to discourage unauthorised users I issue a page/url not found message as follows:
throw new HttpException(404, "File not found - " + Request.AppRelativeCurrentExecutionFilePath);
I think that this is very effective at convincing the person at the other end to give up and go away.
You may want to have a look at this http://www.angrypets.com/tools/rdos/, which does a similar thing to what you are talking about, for similar reasons, and is implemented as a module.
An other annoying but easy way to do it would be to Sleep the request for a discouraging amount of time before closing it.
AFAIK there is no way to outright drop a request without a response. You can keep the request spinning in ASP.NET with whatever thread sleeping technique you like but you're not making life better for your server that way.
The best way that I know of to achieve what you want is having a traffic manager like ZXTM close the request before it gets to the webserver at all.

Resources