We have a secured & authenticated WCF service which cannot use service references. Thus, we provide the interface for the contracts and open client channel manually.
We have found out that as long we open it once, everything works fine. We can call several methods several times. However, if the channel is closed or just set to a new instance, the Login() (which happens to be required for first step prior to using the service), times out.
To make the matters even more mysterious, this only happens on our production server. If I run the same project locally, I am able to login many times as I want. Consuming the methods inside a web browser (even on a code-behind ASPX page) do not have this problem even with the production server. ONLY when it's a .NET client trying to open a client channel against the production server, do we have this problem.
We are not even sure where to start looking. Any advices would be greatly appreciated.
UPDATE:
As per #Rene's suggestion, we turned on logging on both sides. From client's log, there is a record of error which is basically the same timeout error we already got via the exception. Nothing meaningful. On the server's logs, there are records of service methods being invoked successfully even after 2nd login() and from server's POV, the request is served.
Additionally, I discovered that I could not even reproduce this issue on my machine using same test project to reproduce this problem. This reproduces on my developer's machine. I verified that we were at same version of .NET framework and Visual Studio. It has to be surely a client-side problem. What could be it?
In case anyone else is looking for answer, we finally found it -- the issue is due to the need to set on client's side System.Net.ServicePointManager.DefaultConnectionLimit to some higher value. The default value is 2 but in reality this allows only one proxy to be created and be usable. Setting it to 3 would allow 2 proxies to be created & be used.
Related
I have an awkward issue with IIS 10.0 on Windows Server 2016 and ASP.Net 4.5.2 and MVC 5.2.7.
At times, certain requests do not receive a response and run for minutes, maybe 10 or so, before ending in a lost connection (PR_CONNECT_RESET_ERROR in Firefox on Windows, NSURLDomainError in Firefox on iOS). These are mostly POST requests. When this issue occurs, other GET requests will receive a swift response and a correct result. Normally, POST-request do no take long to be processed, typically less than 3 seconds.
Recycling the associated worker process will make the issue go away, for hours or days.
When today inspected the web server when the issue was going on, I saw little CPU usage, less than 10%, memory 56%, the worker process a modest 615 MB. I saw neither logging in the W3C log of these requests, nor in my custom application logs.
I added the Web-Request-Monitor conform How do I see currently executing web request on IIS 8, but in doing so, the the worker process probably got recycled, as the issue is not currently occurring.
There are a reverse proxy and an access manager between the internet and my web server. I suppose they can have something to do with this issue, but it certainly is related to IIS, as recycling helps.
All of this is happening on a acceptation web server running a newer version of my application. I am not aware of any big changes to the application's architecture that could be involved. Also, there will be very little traffic from other clients, if none at all.
What could be next steps to investigate this issue further?
Update
This issue was definitely caused by log4net. However, it was not related to the log4net.Internal.Debug setting. It was caused by two application domains accessing the same log file. This occasionally resulted in concurrency issues with accessing the log file. It appeared that log4net could not properly handle this and got stuck while writing to the log file.
This log file was configured with the RollingFileAppender option. Since we also used AdoNetAppender, we decided to remove file logging all together.
Original
I have found a probable cause. I'll report the steps I took to investigate the issue.
I activated the Worker Processes feature in IIS.
When, after a couple of days of waiting, the issue started again, I found long running requests. They all had State ExecuteRequestHandler and Module Name ManagedPipelineHandler. They had Time Elapsed of hundreds of seconds.
I also activated the Failed Requests Tracing with a rule for long running requests with a Time Taken of 1 minute.
After a couple of days, I started to receive failed request reports. The failed request all have a GENERAL_SET_RESPONSE_HEADER event as their last event.
I added additional debug logging events for each requests. When debugging in my development environment, at one point, I started to see the hanging behaviour there, on one of the new logging statements(!). The application uses log4net.
I captured a stack trace:
log4net.dll!log4net.Appender.AppenderSkeleton.DoAppend(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Util.AppenderAttachedImpl.AppendLoopOnAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.CallAppenders(log4net.Core.LoggingEvent loggingEvent) log4net.dll!log4net.Repository.Hierarchy.Logger.Log(System.Type callerStackBoundaryDeclaringType, log4net.Core.Level level, object message, System.Exception exception) log4net.dll!log4net.Core.LogImpl.DebugFormat(string format, object arg0)
The DoAppend method uses lock(this), which may very well cause hangs.
I also found out that the config setting log4net.Internal.Debug was set to true, which I do not want under normal circumstances and this may be related. I did not attempt to understand the log4net code, but I remember that logging initially did not work, in the acceptance environment, so the setting may very well have been set to true then, causing the issue to start.
Another indication that this is happening with log4net is that when the issue last occurred, I realized that logging of level standard, only occurs in some POST requests. I found a POST-request that does not log and requests to it where handled normally, while the other POST-requests still hung.
For now, I have set log4net.Internal.Debug to false and will wait to see what happens.
IIS recycle fix this issue doesn't mean that this is an IIS issue because all asp.net application run in .net runtime unless it is proved that the request is hang in IIS module.
So you may need to wait this issue happen again, then create a Failed request tracing rule for time-taken. Then it will tell us this issue is happening on IIS pipeline module or .net runtime.
If all request hang in .net runtime. Then you may have to capture a hang dump and do a deep analysis via WINDGB and mex extension. It will tell us what's happening there.
I'm using web api self host inside a windows service and I've encountered a problem and after googling for couple of hours haven't found a reasonable answer.
One of the api controllers serves large stream of data (not really that large, couple of tens of MB). It takes some time to produce data so I've decided to use TransferMode.StreamedResponse to minimize the time client has to wait for response. I've also added a CompressHandler and custom CompressedContent (derived from HttpContent) mostly based on a following answer.
The controller returns an instance of IDataReader, which is then serialized by custom formatter which is lastly compressed inside CompressedContent that I've mentioned. The whole data passing is streamed so while the client receives data, a data reader on server side may still be reading rows from database. Everything works fine when client is acting nicely.
The problem occurs when a client drops connection while the data is still being serialized to the underlying network stream. I've tried to watch for IsFaulted task inside of ContinueWith delegate (in CompressedContent from the link) and dispose underlying network Stream. Unfortunately the CommunicationException (The specified network name is no longer available) is still being thrown when the control leaves my code. From the stacktrace it looks like the exception is thrown when the Web Api tries to close (end) the underlying network stream (http channel?). As it happens with unobserved exceptions it brings entire windows service down.
I've mitigated the problem by setting windows service recovery options but I would like to know if this failure can be handled in code.
Is there a way to setup a custom error handler (IErrorHandler presumably) inside web api self hosting service mode to prevent this kind of error?
I'm using Beta version, I will try to reproduce this error on RC but I somehow doubt that setting up this kind of error handler would change in any way
We had the same issue. I was able to submit a fix to MS and they have in turn released a nightly build that fixes this. They are looking at back porting the fix to the RTM. You can see the pull release here: http://aspnetwebstack.codeplex.com/SourceControl/network/forks/rdean79/issue284/contribution/3329
Hai Guys,
My application deals scheduled mail concept (i.e) every morning 6.00 am my users gets a remainder mail about their activities for the day... I dont know how to do this.... Many told use windows service but i will host my website on a shared server i may not get rights to do windows service... Is there any dll for sending mails at a schduled time through asp.net application ..please help me out guys......
You cant do much in a shared hosting. Try upgrading your hosting or else write a windows service, to run on your machine, which will call an asp.net which can send out emails. Of course your machine has to be switched on all the time or at least during 6:00 AM :). You will have to take proper steps to avoid unauthorized request for that aspx page.
you can check this article too: http://www.codeproject.com/KB/aspnet/ASPNETService.aspx
You can't really do this with ASP.Net. ASP.Net is for web pages - which are reactive to HTTP requests.
You need a scheduled task or a service. All a website can do is respond to requests. I guess you could program the functionality into a web page and have a remote process request the page every morning - but what happens if someone else requests the page?
You can either have a program that runs constantly and has a timer or a loop that checks the time of day and then sleeps for a really long time and when the timer goes off or it's the right time of day it sends an email, or you can launch a program as a scheduled task. The first method can also be implemented as a service if you would like. Keep in mind you dont need ASP.Net to send emails, all you need is a console application that uses System.Net.Mail. Check out the mailer sample on MSDN for a very simple idea.
One other thing you can consider: IIS has an smtp service that you can install and it uses a pickup directory to send mail. You write an email to the pickup directory as an .eml file and IIS grabs it and sends it almost immediately. If you do that, you'll still have to write the emails (System.net.Mail will write the .eml files from a MailMessage, just set SmtpClient.DeliveryMethod to SpecifiedPickupDirectory or PickupDirectoryFromIIS and call SmtpClient.Send) but it will then send them for you. You'll still need to schedule something somehow so this might not be all that more useful but I thought I'd at least let you know that it exists.
One thing to be aware of: when the IIS SMTP service reads the send envelope of the .eml file, the order of the Sender and From headers is significant; if the From header appears before the Sender header then the MAIL FROM command will use the From header, which is incorrect (and MS won't be fixing this one). This appears to be an issue ONLY with the IIS SMTP service as it hasn't been reported anywhere else that I'm aware of. Reversing the order of the headers is the work-around. By default SmtpClient always writes the From header first. I'm aware of the issue and IIS isn't fixing it but I may be able to get a fix into SmtpClient for the .NET 4.0 RC build that re-orders the headers for you but no promises.
If you happen to have it handy (and I assume you do), you can use a SQL Server Agent job to make a request to an ASP.NET page that sends the email.
Here's some example code:
http://nicholasclarke.co.uk/blog/2008/01/16/web-request-from-sql-server-via-c/
Of course, since you're using SQL Server to call CLR code anyway, you could just have that code send out the emails (via System.Net.Mail) rather than requesting a page on IIS to do so. To do this, SQL Server would need:
Access to all of the data needed to send the emails
Outbound firewall access to send an email
CLR code that encapsulates all of the logic needed to know where/what to send.
Okay this is interesting, and what I did fits silky's definite of 'cheating', but no it was pretty cool for me.
What I did was spawn a new thread from ASP.Net code (it was possible on that host), and that thread did the scheduled job.
I checked whether the thread was alive (which is pretty easy) on every visit to the website (not so reliable I know, but it worked cause that website has plenty of visitor).
If at all you do this
Treat this as a stop-gap while you arrange to get a dedicated host or VPS.
Rest assured that the hosting company will kill your thread and withdraw permissions when they discover you're doing this.
I have a web service that is on an internal server. It can be called from any website on our network.
More and more developers are starting to use it. Current probably 20+ pages use this service, and the number is growing fast. I can see a year from now, someone asking what pages are using this service and what methods.
I would like to log the url of the pages that use my web service as the request come in.
It would also be nice to know the method they are calling.I need to do something in such a way, that it does not affect the client web sites.My first thought was that I could write some code in the global.asax.
I have added some code to the Application_BeginRequest to log the request object details, but there does not appear to be anything about the requesting url.
What am I missing? Should I be looking at a different object?
Thanks.
Without disrupting existing users this is going to be difficult. The httpContect.Current.RequestUrl will just return the URL used to call your web service, not which web page called it.
The closest you can do without disrupting existing apps and forcing developers to change them is to grab the HttpContext.Current.Request.UserHostAddress, so you can at least get the IP of the machine calling your service.
Beyond this, what you might want to consider is adding a parameter to your functions for "CallingApp" and then log that in your code. That's pretty much what we did once re realized that we needed to know which apps are calling our service. We actually have an application monitoring service that uses a GUID for every new app we develop, and we pass that GUID to any web service. It[s extra work but to us it was critical because it allows us to know which apps will be affected when we need to perform updates or take the app server down for maintenance.
Edit - added
As a side note, at the point we realized we needed to track this, we had already been using web services for about a year. When faced with the same problem, we created a new set of web services, and included the extra field for the calling app in all of the new services, and then slowly went back and changed the older programs to point to the new services.
IN retrospect, we wish we had known we would need to do this up front because it created a lot of extra work. I'm guessing you'll be facing something similar if you really want to know exactly who is calling your services.
The only thing you can probably retrieve from the consumer is the IP address without changing your interface.
If you can change this you could do this e.g. by adding authentication and logging who is calling what, or by having some simple "token" principle.
However both methods require you to change the interface and therefore break backwards compatibility - which you should never do.
By always ensuring both back and forward compatibility you should not need to know exactly who is calling your service, but only that it is actually used.
#David Stratton
Thanks for your help. I think your suggestions were great. I accually did something very different, after your answer gave me some new ideas.
I should have mentioned that I was generating the web proxy that most of my users were using to make calls against my web service. My client in general do NOT use the proxy that Visual Studio creates.
Here is what did:
I generated my web proxy client again, and added calls to log the httpcontext of the client before every call. Because the proxy is running on the client, he had access to everything I needed. That allowed me to record everything about the client and the specific call they were making. I realize this would not work for most cases. But all of my clients are internal web sites.
It also had the advantage in that the clients did not have to modify their code at all. I just gave them all a new DLL. Problem solved. I get all the tracking data I want, and they did not have to modify their code.
I was stuck trying to solve the problem from the web service's point of view.
I realize that there is still a whole in this implementation, because someone does not have to use my client proxy to call my service. I guess I could force that at some point in the future. For now, they could let Visual Studio genereate a web proxy for my service. However, if they do that I guess I don't care. That is not the recommened way to call my service. I think the only one doing that is an ASP.NET 1.1 web site. When they upgrade, they will probably switch to my generated proxy.
Without implementing some sort of authentication, there isn't a guraenteeted way of knowing exactly who is calling your service - web metrics are the only way you can gauge what volume of traffic is hitting your service.
I'm sure you already know this but the whole point of a web service isn't to know or care who is calling it.
I have successfully used ...
Dim strReferrer As String = HttpContext.Current.Request.UrlReferrer.AbsoluteUri
to get the calling page that called my WEB API 2 Web Service.
I have a web application which fetches information from a web service. It works fine in our development environment.
Now I'm trying to get it to work in a customer's environment instead, where the web service is from a third party. The problem is that the first time the application tries to fetch information it cannot connect to the web service. When it tries again just seconds later it works fine. If I wait a couple of hours and try again, the problem occurs again.
I'm having a hard time believing this is a programming error, as our customer and the maker of the web service thinks. I think it has to do with one of the IIS or some security in the network. But I don't have much to go on and can't reproduce the error in our development environment.
Is it failing with timeOutException when you try to connect first time?. If yes, this could be the result on start up time of the service
I have a rule: "Always assume its your fault until you can demonstrate otherwise". After over 20 years, I still stick to it.
So there are therefore two cases:
The code is broken
There is a specific issue with the live environment
Since you want to demonstrate that the problem is (2) you need to test calls to the service, from the live environment, using something other than your application. Exactly what will depend on the nature of the web service but we've found SoapUI to be helpful.
The other thing that's not clear is whether you are making calls to the live service from your development environment - if, in testing, you're not communicating with the same instance of the service then that's an additional variable that will need to be considered (and I appreciate that you're not always given the option).
Lastly, #Krishna is right - there may be a spin up issue with the remote service (hence my question about whether you're talking to the same service from your dev environment) and -horrible as it is - the solution in the first instance may simply be to find a way to allow for this!
The error was the web service from the third party. The test stub we got to develop against was made in C# and returned only dummy answers. The web service in the customer environment actually connected to a COM object. The first communication with the COM object after a longer wait took almost a minute.
Good for me that the third party developers left the source code on the customer servers...