I am taking my first steps with visual studio enterprise load tests.
I have created a webtest following this tutorial: https://www.visualstudio.com/en-us/docs/test/performance-testing/run-performance-tests-app-before-release
The webtest works OK, hits my page properly and succeeds.
However when I have created a load test with this webtest, each request fails (regardless of the number of users, even with 2 users)
The error is
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond 10.27.8.6:443
and the stack trace
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at
System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,
Socket s4, Socket s6, Socket& socket, IPAddress& address,
ConnectSocketState state, IAsyncResult asyncResult, Exception&
exception)
The thing that potentially might be an issue is the fact that I have a self signed SSL certificate that obviously shows the certificate error warning. However, this is something I cannot change for now, so the tests will have to ignore this.
Also, I used https://www.webperformance.com/download/ tool to do these tests and it all worked OK with a relatively high intensity, so this cannot be a matter of poor performance of the site.
Any idea what could it be?
Not sure what more details can I give than below (as I said, for sure its not a matter of performance, because similar load test with different tool runs OK. It is also not a matter of webtest being wrong because it still passes when run as single test).
Webtest setup
Settings:
Loadtest setup
Loadtest settings
We have a web application that occasionally makes HTTPS requests to third party web services. Most of the time these work without issue. However occasionally the application will begin throwing exceptions on each request:
System.Net.WebException: The underlying connection was closed: Could not establish
trust relationship for the SSL/TLS secure channel. --->
System.Security.Authentication.AuthenticationException: The remote certificate is
invalid according to the validation procedure.
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
...etc...
Now the odd part is that when we recycle the web application, the exception stops being thrown. For a while anyway.
We determined that a library was occasionally setting System.Net.ServicePointManager.ServerCertificateValidationCallback to a method which did not correctly validate certificates. The reason it seemed random is that certificate validation worked until the library was used and that callback was set.
The solution is to fix the library. And as general rule never globally set ServerCertificateValidationCallback in production code.
I set up Amazon SES and it initially worked for a few hours, then all of a sudden stopped. All of the e-mails I'm sending as, and our domain, have been verified. We are no sending bulk e-mails - only a few hundred per day. Whenever I make changes to the web.config it seems to allow it work again for another 2-3 hours. For example, It stopped working so I switched port 587 to 25, and it began working for 2-3 hours, then stopped. Then I switched back to 587 and the same thing happened. Once it stops working it doesn't ever seem to start again on its own. It is running on two load balanced servers, asp.net framework v2.0, IIS 7.5. Here is the code I'm using:
web.config:
<system.net>
<mailSettings>
<smtp deliveryMethod="Network" from="no-reply#ourdomain.com">
<network defaultCredentials="false" host="email-smtp.us-east-1.amazonaws.com" userName="***" password="***" port="587" />
</smtp>
</mailSettings>
</system.net>
c# code:
var smtpClient = new SmtpClient() { EnableSsl = true };
var mailMessage =
new MailMessage(fromAddress, toAddress, subject, body)
{
IsBodyHtml = true
};
smtpClient.Send(mailMessage);
Here are the two errors I've been receiving:
The following exception was thrown by the web event provider '(null)' in the application '/' (in an application lifetime a maximum of one exception will be logged per provider instance):
System.Web.HttpException: Unable to send out an e-mail to the SMTP server. Please ensure that the server specified in the <smtpMail> section is valid. ---> System.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Mail.SmtpConnection.Flush()
at System.Net.Mail.ReadLinesCommand.Send(SmtpConnection conn)
at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)
at System.Net.Mail.SmtpClient.Send(MailMessage message)
--- End of inner exception stack trace ---
at System.Net.Mail.SmtpClient.Send(MailMessage message)
at System.Web.Management.MailWebEventProvider.SendMail(MailMessage msg)
--- End of inner exception stack trace ---
at System.Web.Management.MailWebEventProvider.SendMail(MailMessage msg)
at System.Web.Management.SimpleMailWebEventProvider.SendMessageInternal(WebBaseEventCollection events, Int32 notificationSequence, Int32 begin, DateTime lastFlush, Int32 discardedSinceLastFlush, Int32 eventsInBuffer, Int32 messageSequence, Int32 messagesInNotification, Int32 eventsInNotification, Int32 eventsLostDueToMessageLimit)
at System.Web.Management.SimpleMailWebEventProvider.SendMessage(WebBaseEvent eventRaised)
at *****.Global.SimpleMailWithSslWebEventProvider.ProcessEvent(WebBaseEvent raisedEvent)
at System.Web.Management.WebBaseEvent.RaiseInternal(WebBaseEvent eventRaised, ArrayList firingRuleInfos, Int32 index0, Int32 index1)
The following exception was thrown by the web event provider '(null)' in the application '/' (in an application lifetime a maximum of one exception will be logged per provider instance):
System.Web.HttpException: Unable to send out an e-mail to the SMTP server. Please ensure that the server specified in the <smtpMail> section is valid. ---> System.Net.Mail.SmtpException: Service not available, closing transmission channel. The server response was: Timeout waiting for data from client.
at System.Net.Mail.MailCommand.CheckResponse(SmtpStatusCode statusCode, String response)
at System.Net.Mail.SmtpTransport.SendMail(MailAddress sender, MailAddressCollection recipients, String deliveryNotify, SmtpFailedRecipientException& exception)
at System.Net.Mail.SmtpClient.Send(MailMessage message)
at System.Web.Management.MailWebEventProvider.SendMail(MailMessage msg)
--- End of inner exception stack trace ---
at System.Web.Management.MailWebEventProvider.SendMail(MailMessage msg)
at System.Web.Management.SimpleMailWebEventProvider.SendMessageInternal(WebBaseEventCollection events, Int32 notificationSequence, Int32 begin, DateTime lastFlush, Int32 discardedSinceLastFlush, Int32 eventsInBuffer, Int32 messageSequence, Int32 messagesInNotification, Int32 eventsInNotification, Int32 eventsLostDueToMessageLimit)
at System.Web.Management.SimpleMailWebEventProvider.SendMessage(WebBaseEvent eventRaised)
at *****.Global.SimpleMailWithSslWebEventProvider.ProcessEvent(WebBaseEvent raisedEvent)
at System.Web.Management.WebBaseEvent.RaiseInternal(WebBaseEvent eventRaised, ArrayList firingRuleInfos, Int32 index0, Int32 index1)
I tried to get help on the Amazon forums but haven't had any luck. It seems like something is interfering with the connection but I don't know what. Any ideas? Thanks.
This sounds like a tricky issue indeed - I'm not 100% positive on this, but you seem to miss disposing the SMTP client, which is mentioned in Sending email is often very slow after changing to VPC instance to cause the very error you are seeing:
When I do not dispose of the SmtpClient, I get the error...
'Service not available, closing transmission channel. The server
response was: Timeout waiting for data from client.'
This would actually be a good explanation for intermittent problems, i.e. depending on the executed code path this issue could trigger the SMTP connection to Amazon SES to hang, which is obviously remedied by switching to another port, implying the connection being reset - that's what the SmtpClient.Dispose Method ensures as well:
Sends a QUIT message to the SMTP server, gracefully ends the TCP
connection, and releases all resources used by the current instance of
the SmtpClient class.
Accordingly, an appropriate pattern would be to facilitate a using Statement, as demonstrated in Getting Started with Amazon SES and .NET:
String username = "SMTP-USERNAME"; // Replace with your SMTP username.
String password = "SMTP-PASSWORD"; // Replace with your SMTP password.
String host = "email-smtp.us-east-1.amazonaws.com";
int port = 25;
using (var client = new System.Net.Mail.SmtpClient(host, port))
{
client.Credentials = new System.Net.NetworkCredential(username, password);
client.EnableSsl = true;
client.Send
(
"FROM#EXAMPLE.COM", // Replace with the sender address.
"TO#EXAMPLE.COM", // Replace with the recipient address.
"Testing Amazon SES through SMTP",
"This email was delivered through Amazon SES via the SMTP end point."
);
}
Please note that the sample is using port 25, which I highly recommend to avoid due to the usually implied sending restrictions, see Addendum below for details regarding the respective Amazon EC2 throttling.
Good luck!
Addendum
Amazon EC2 throttling on port 25
You are probably aware of this (and I actually don't think it's an issue here), but Amazon EC2 imposes default sending limits on email sent via port 25 and throttles outbound connections if you attempt to exceed those limits which still applies when using Amazon SES, see Amazon SES SMTP Issues:
You are sending to Amazon SES from an Amazon EC2 instance via port 25
and you cannot reach your Amazon SES sending limits or you are
receiving time outs — Amazon SES EC2 imposes default sending limits on
email sent via port 25 and throttles outbound connections if you
attempt to exceed those limits. To remove these limits, submit a
Request to Remove Email Sending Limitations. You can also connect to
Amazon SES via port 465 or port 587, neither of which is throttled.
Consequently I'd remove port 25 out of your testing/switching scenario and use ports 465/587 only instead in order to avoid false leads (as quoted you could also request to get this limitation removed though, but it will take a few hours and port 25 seems to be best avoided in the first place) - it's a bit unfortunate that several official Amazon SES samples are using port 25 without even mentioning this easily triggered issue indeed.
Although this question is specific to .NET v2.0, I recently came across this issue on .NET v4.5, as well. Even after adding using around SMTP client (as suggested in earlier posts), I was still intermittently receiving this error. After many hours of research I came accross AWS's support for STARTTLS port (2587; http://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html) and, in the very fine print, Microsoft states when "EnableSsl" is true on SMTP client the client expects a STARTTLS session (https://msdn.microsoft.com/en-us/library/system.net.mail.smtpclient.enablessl(v=vs.110).aspx). Thus, after updating my port to 2587 and surround the SmtpClient in a "using {}" I was able to successfully send ~20k emails without reproducing the error.
Thus, the working configuration would be:
<system.net>
<mailSettings>
<smtp deliveryMethod="Network">
<network host="email-smtp.us-west-2.amazonaws.com" port="2587" enableSsl="true" password="password" />
</smtp>
</mailSettings>
</system.net>
One disclaimer is that I still don't fully understand the true configuration of either AWS or .NET SMTP client, so this is still a bit circumstantial.
i am also using AWS ses service
smtp deliveryMethod="Network" from="XXXXXXX#ZXXX.com"
network enableSsl="true" host="email-smtp.us-east-1.amazonaws.com" userName="******" password="*********" port="25"
smtp
Hope this works for you.
We've recently moved our websites to some new servers with Windows 2008 R2 / IIS7.5 and have started getting the following error:
Exception information:
Exception type: HttpException
Exception message: An error occurred while communicating with the remote host. The error code is 0x80070001.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ReadEntityCoreSync(Byte[] buffer, Int32 offset, Int32 size)
at System.Web.HttpRequest.GetEntireRawContent()
at System.Web.HttpRequest.FillInFormCollection()
at System.Web.HttpRequest.get_Form()
at System.Web.HttpRequest.get_HasForm()
at System.Web.UI.Page.GetCollectionBasedOnMethod(Boolean dontReturnNull)
at System.Web.UI.Page.DeterminePostBackMode()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Specified method is not supported.
at System.Web.HttpResponseStream.get_Position()
at System.Drawing.UnsafeNativeMethods.ComStreamFromDataStream.Seek(Int64 offset, Int32 origin)
The error seems to occur pretty randomly on a number of different pages and we've not be able to reproduce it ourself.
I've look on this site and others for similar examples, but have yet to find a solutions. The common response seems to be that it's something to do with TCP Offloading and/or VMware. We're not using any virtualization, so is the only solution to turn off TCP Offloading? What would the impact of turning it off be?
The application is written in ASP .NET 4.0 hosted in IIS 7.5 on machine running Windows Server 2008 R2. The application pool is using Integrated Managed Pipline Mode.
The exception below is thrown when on several different pages which don’t have anything obvious in common. I can’t reproduce the exception myself, but it happened 10 times last week in the production environment.
From the Stack Trace I see that the exception thrown by the method "HttpRequest.GetEntireRawContent" so I was wondering if there is something wrong with request, such as it being too long or containing an invalid character.
Just to be clear, this exception is not thrown from code that I have written, it is from within the System.Web namespace. So I cannot add a try...catch around it.
I'd be greatful for any ideas on cause of the exception or troubleshooting steps to find out more information.
Exception: System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> System.Web.HttpException (0x80004005): An error occurred while communicating with the remote host. The error code is 0x80070001. ---> System.Runtime.InteropServices.COMException (0x80070001): Incorrect function. (Exception from HRESULT: 0x80070001)
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ReadEntityCoreSync(Byte[] buffer, Int32 offset, Int32 size)
at System.Web.HttpRequest.GetEntireRawContent()
at System.Web.HttpRequest.FillInFormCollection()
at System.Web.HttpRequest.get_Form()
at System.Web.HttpRequest.get_HasForm()
at System.Web.UI.Page.GetCollectionBasedOnMethod(Boolean dontReturnNull)
at System.Web.UI.Page.DeterminePostBackMode()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at ASP.report_openinghours_aspx.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
EDIT:
As pointed out by lukiffer there is a similiar question here: stackoverflow.com/questions/7825127/ The suggested fix is to turn off TCP offloading, the way I understand it is that TCP offloading shields the webserver from a lot of the complexity of TCP. Therefore turning it off would consume a lot of CPU cycles so slow down the web servers, which is obviously not desirable.
I also don't understand why TCP offloading would cause IIS to throw an exception.
You couldn't possibly still have this issue, but if others run into this issue here is a link that might help.
TechNet forum
TCP Task Offloading options of the NIC driver should be set at the
physicla NIC level (the nic of the host that the External virtual
network is using). Be sure that you are not disabling chimney and
thinking that you are disabling Task Offloading. In the settings of
the driver are options with 'offloading' in the name - these are the
options to turn off.
Disable Power managment on the phsycial NIC (some drivers turn this on).
Try a physical Intel NIC (the least problematic of all with Hyper-V)
Check that the ICs in the VM are equal to the host.
Check for other issues with the VM such as resource spiking with CPU or disk IO or anything else that might cause a hang.