We have a web farm of IIS 6 servers that runs our application.
Our session is stored on Sql Server 2005 on a diffrent server.
Every couple of months we are getting this error in one of the web server logs:
"Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred beacuse all the pooled connections were is use and max pool size was reached"
Stack trace:
System.Data.ProviderBase.DbConnectionInternal
GetConnection(System.Data.Common.DbConnection)
at
System.Data.ProviderBase.DbConnectionFactory.GetConnnection(DbConnection
owningConnection) at
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection
outerConnection,DbConnectionFactory
connectionFactory) at
System.Data.SqlClient.SqlConnection.Open()
at
Systme.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo
sqlPartitionInfo)
When this exception is thrown the server starts to behave strange - some users can access the app and some do not.
The only solution we found so far is to reset IIS on that server.
I should also mantion that the server is not appered to be overloading and the preformence is pretty normal before this happens..
Any ideas?
This is classic of bad resource management.
If you are using a custom Session Manager (module) with SQL then you are not disposing of the connections properly, and the application pool is running out of connections. Then all subsequent connection are waiting for the connection to automatically be disposed; and this is where the timeout occurs.
However, this is probably not your problem, so what you need to do is limit the timeout time of the connections as such:
Timeouts under heavy load
If your web
servers are under heavy load it may be
useful to increase the time out for
session state access. You can add the
stateNetworkTimeout attribute to the
sessionState settings in web.config
and machine.config
If a Web server or a state server
is under stress and cannot complete
session accesses on time, event ID
1072 and event ID 1076 may be logged
in the event log.
http://idunno.org/articles/277.aspx
Furthermore
You should only use SessionState for basic data types like string, int, bool
If you are storing alot of information or complex data types, maybe you need to reasses what and why it is stored there.
You should look into using Cache or Viewstate. There are many such articles on the internet, for example:
http://www.codeproject.com/KB/session/exploresessionandcache.aspx
Since your session state is SQL based, and this is the slowest mode, you should really try to use it as least as possible. Maybe you could store values in the cache with a unique key, and store the unique key in the session variable. Many workarounds exist.
Another more useful link:
http://devshop.wordpress.com/2008/04/10/how-to-choose-from-viewstate-sessionstate-cookies-and-cache/
As your comments became more specific, I also have the following to add. If you create a class like the following:
public class PartitionResolver : System.Web.IPartitionResolver
{
private String[] partitions;
public void Initialize()
{
// create the partition connection string table
// web1, web2
partitions = new String[] { "192.168.1.1", "192.168.1.2" }; // keep adding servers
}
public String ResolvePartition(Object key)
{
String oHost = System.Web.HttpContext.Current.Request.Url.Host.ToLower().Trim();
if (oHost.StartsWith("10.0.0") || oHost.Equals("localhost"))
return "tcpip=127.0.0.1:42424";
String sid = (String)key;
// hash the incoming session ID into
// one of the available partitions
Int32 partitionID = Math.Abs(sid.GetHashCode()) % partitions.Length;
return ("tcpip=" + partitions[partitionID] + ":42424");
}
}
... and then in your web.config you put something like the following:
<sessionState mode="StateServer"
partitionResolverType="NameSpaceName.PartitionResolver"
cookieless="false"
timeout="60" />
... and then follow the instructions:
http://www.c-sharpcorner.com/UploadFile/gopenath/Page107182007032219AM/Page1.aspx
... and create an identical machine key across all your web servers; then you will not require SQL session state, and will have a common session state, which you can load balance across any number of state servers that you may require.
All you would ever have to do is update this line:
partitions = new String[] { "192.168.1.1", "192.168.1.2" }; // keep adding servers
... and you could have multiple web servers using the same state servers, so even if for whatever reason you switch web servers, you will still maintain your session. And also, as you see the session slowing down, just keep adding state servers.
Related
I am having a web application deployed to IIS, my app uses static Dictionary which is filled in from an external api frequently.
Sometimes I observe that the Dictionary is being cleared once in a while & I suspect it is because of IIS Automatic Recycle.
Can anyone please confirm that this could be a reason?
So basically my question would be will IIS Recycle cleans up the static memory that a webapp is using? (Although I understand that this will only happens when there are no active connections to the server)
Yes, the IIS by default recycles your app pool by calling a garbage collector to clear the memory on every 20 minutes.
You can see Idle-timeout setting in your app pool -> Advanced settings, but better do not change it.
All static things are "Bad" do not use them, your option is caching. You can make a generic cache service that is using the default MVC cache and make it thread safe.
You can also use the [OutputCache] attribute on child actions controller and set minutes. Between this interval the data will be cached
Or you can implement your own caching logic.
From all the three things I will suggest you the first one with using the default MVC cache. I will provide you a sample implementation thanks to #TelerikAcademy and #NikolayKostov
namespace Eshop.Services.Common
{
using System;
using System.Web;
using System.Web.Caching;
using Contracts;
public class HttpCacheService : IHttpCacheService
{
private static readonly object LockObject = new object();
public T Get<T>(string itemName, Func<T> getDataFunc, int durationInSeconds)
{
if (HttpRuntime.Cache[itemName] == null)
{
lock (LockObject)
{
if (HttpRuntime.Cache[itemName] == null)
{
var data = getDataFunc();
HttpRuntime.Cache.Insert(
itemName,
data,
null,
DateTime.Now.AddSeconds(durationInSeconds),
Cache.NoSlidingExpiration);
}
}
}
return (T)HttpRuntime.Cache[itemName];
}
public void Remove(string itemName)
{
HttpRuntime.Cache.Remove(itemName);
}
}
}
The usage of it is super simple with anonymous function and time interval
You can set it as a protected property of a Base Controller and to Inherit BaseController in every controller you use. Than you will have the cache service in every controller and you can simply use it that way
var newestPosts = this.Cache.Get(
"newestPosts",
() => this.articlesService.GetNewestPosts(16).To<ArticleViewModel().ToList(),
GlobalConstants.DefaultCacheTime);
Let's assume that GlobalConstants.DefaultCacheTime = 10
Hope that this answer will be useful to you. :)
If you look at this MS article: https://technet.microsoft.com/pl-pl/library/cc753179(v=ws.10).aspx
In addition to recycling an application pool on demand when problems occur, you can configure an application pool to recycle a worker process for the following reasons:
At a scheduled time
After an elapsed time
After reaching a number of requests
After reaching a virtual memory threshold
After reaching a used memory threshold
So if IIS recycle would not clean up memory recycling it on memory threshold would not make sense. Additionally, IIS recycle cause application restart so it's obviously clears it memory too.
We have a webfarm with 2 servers, both has the DFS replication implemented. We have a section on homepage which displays 4 articles known as "TopStories" and are cached so that it does not fetches from database everytimeNow when editors updates the Topstories, that change should get live immediately to the end users across the web-farm, now say editor made the changes to the Topstories on server 1 and the end-user who accessed the website was taken to server 2, so user won't be able to see the latest updates on topstories, as on server 2 it will still fetch from cacheSo for this we recycles the app-pool by updating the web.config file.
We are using below code to make an update in appsettings element of web.config.
public void RefreshWebConfig()
{
//XmlDocument doc = new XmlDocument();
//doc.Load(HttpContext.Current.Server.MapPath("web.config"));
//doc.Save(HttpContext.Current.Server.MapPath("web.config"));
System.Configuration.Configuration config = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");
System.Configuration.KeyValueConfigurationElement setting = config.AppSettings.Settings["TopStorySessionKey"];
int SessionKey = 0;
if (setting != null)
{
SessionKey = Convert.ToInt32(CommonUtil.GetConfigurationValue<int>("TopStorySessionKey"));
if (SessionKey == 100)
SessionKey = 0;
config.AppSettings.Settings["TopStorySessionKey"].Value = Convert.ToString(SessionKey+1);
}
else
{
config.AppSettings.Settings.Add("TopStorySessionKey", SessionKey.ToString());
}
config.Save();
}
I have 2 questions related to this:
In any instance updating the web.config's app-setting key could delete the whole web.config file using the above code?
Is there any better way to replicate the change on both the servers.
Thanks in advance.
In any instance updating the web.config's app-setting key could delete the whole web.config file using the above code?
I can't see how this code as written would delete the web.config, but it's a rather indirect way to force the app pool to recycle. You can do it directly:
Restarting (Recycling) an Application Pool
Is there any better way to replicate the change on both the servers.
It is rather expensive to recycle the entire app pool just to invalidate a small cache. Instead, consider using a MemoryCache. If you can place the latest articles e.g. a file (whether a single file accessed as a network share, or as a copy of the file on each web server), you can use a file dependency to expire the cache. If the articles are in a SQL database, you can use a SqlChangeMonitor to expire the database. If you can allow the latest articles to be e.g. 2 minutes old, you can use sliding expiration to reload the cache every so often.
I have a web server and its session state is stored in a Sql Server. I need to change the timeout to use an external time source. How would I do this?
Ok, it's not hard to do it. I suppose you're able to read that data from the table - related, as you said, to some User Preference - and to put this data in an int:
int myTimeout = [... read from DB];
It's enough to add this line after the login procedure has been completed (or even during the procedure):
Session.Timeout = myTimeout;
Note: the Timeout is in minutes.
We recently launched a new web site... there are roughly ~150 users active during peak hours. During peak hours, we are experiencing an issue every few minutes, the exception text is listed below.
System.Web.HttpUnhandledException:
Exception of type 'System.Web.HttpUnhandledException' was thrown.
---> System.Data.SqlClient.SqlException: The client was unable to establish a connection because of an error during connection initialization process before login.
Possible causes include the following:
the client tried to connect to an unsupported version of SQL Server;
the server was too busy to accept new connections;
or there was a resource limitation (insufficient memory or maximum allowed connections) on the server. (provider: Named Pipes Provider, error: 0 - No process is on the other end of the pipe.)
Our data access layer calls various DataTableAdapters using the following syntax.
EDIT
Yes, da is the name assigned to the DataTableAdapter. There is no connection.Open() because the DataTableAdapter takes care of all that, right?
using(TheDataLayer.some.strongly.typedNameTableAdapters.suchAndSuchTableAdapter da = new TheDataLayer.some.strongly.typedNameTableAdapters.suchAndSuchTableAdapter())
{
StronglyTyped.DataTable dt = new StronglyTyped.DataTable();
da.FillByVariousArguments(dt, ..., ...);
//da.Dispose();
return something;
}
The connection string looks something like:
<add name="MyConnectionString"
connectionString="Data Source=myDBServerName;Initial Catalog=MyDB;User ID=MyUserName;Password=MyPassword"
providerName="System.Data.SqlClient" />
I'm trying to rule the problem being in Code. Is there anything "simple" that can be done to minimize this issue?
Thanks.
Have you tried "Connection Pooling" directly in connection string settings?
Example:
connectionString="....;Pooling=true;Min Pool Size=1;Max Pool Size=10;..."
You can read more info here: http://msdn.microsoft.com/en-us/library/8xx3tyca%28v=vs.71%29.aspx
Without seeing the code that actually opens and uses the connection, it's hard to say where the problem is.
Please update your question with what happens when you create that DataAdapter (I'm guessing that's what da means).
Also, if you're using the using statement, you shouldn't be disposing of the thing you created the using statement for.
We had similar issue which only happenes in our production environment and it was particularly associated with load. During busy time of day we would recieve several of the above mentioned exception.
We gone through a massive investigation around why this exception occurs and did a lot of changes to fix the issue. The defacto change we did which aleviated the problem was connection pool setting by setting min pool size to 1 and max pool size to 10. (It can vary based on your situation)
This issue will be more prevalent when you have several i.e. 1000's of Customer DB and use default connection string (i.e. database=DBName;server=ServerName). We were not explicitly setting min/max pool size hence it took default settings which set Min pool size to 0 and max pool size to 100.
Again, I dont have concrete proof but the theory is that during busy time of the day based on load it made several connection to DB server and DB server was bombarded with a lot of connection request at single point to several databases. Either Application server or DB server did have bandwidth to handle that many connection in a short period of time. Also, it was happening with server with most databases. Though we did not see a lot of connection at a time but Application server was not able to make connection to databases for a short duration when it had surge of requests going in.
After we set min pool size we aliveated this problem as there is atleast one connection to each database which is available all the time and if there is blast of request which required to make connection to several databases we already had atleast one connection to the database available before we request a new one.
Maybe unrelated to the actual problem you were facing, but this error is also thrown if you are trying to connect without specifying the correct port along with the database server name.
I am having some trouble with one of my ASP.NET 2.0 application's connection string. There are instances that I would get a ConnectionString Property Has not Been Initialized problem which occurs randomly and intermittently and without any reason.
My Connection string is actually coming from a webservice, because different kinds of users are provided with different sets of connection string depending on their user level.
What I have done so far goes like this:
I have a master page (mstr.page) and the corresponding code behind (mstr.page.vb).
In my master page, I retrieve initially the connection string and store the same into a session variable, i.e.
Session("ConnString") = "RetrievedConnectionString"
Now in one of my pages, let us say page1.aspx.vb, I use public shared functions from a class (named MyClass.vb) and use it in my page1.aspx.vb.
Some codes for reference:
[MyClass.vb]
Imports System.Web
NameSpace ClassNameSpace
Public Class Admin
Protected Shared da as New DataAccess()
Public Shared Function MYFunction() as String
'Execute some sql statements here, using the DataAccess
stringToReturn = Ctype(da.ExecuteScalar("SQLCommand"), String)
Return stringToReturn
End Function
End Class
End NameSpace
[DataAccessClass.vb]
Public Class DataAccess()
Private m_ConStr As String = ""
Public Sub New()
m_ConStr = HttpContext.Current.Session("ConnString")
End Sub
'Some methods for SQL statement execution (ExecuteQuery, ExecuteScalar)
End Class
[Page1.aspx.vb]
Imports ClassNameSpace
Protected Sub Page_Load(....) Handles Me.Load
Dim strValue as String = Admin.MyFunction()
End Sub
I have placed the code above to show you some rough idea of how things are going.
Basically, the function Admin.MyFunction() at times fails, because in the data access class, the connection string seems to have lost it's value (either blank or Nothing).
This has troubled me for quite some time already.
I hope someone can point me in the right direction to resolve this. Basically, I want my connection string which is retrieved by each user visiting the web application be maintained across all the time and be used anywhere. Session variable does not seem to be the best fit since when the ASP.NET recycles its process, the session is lost.
By the way, I am retrieving the connectionstring initially via the master page from a web service. I tried to place the same retrieve function in the Data Access class when conditions is that the session variable is lost, but I think my application cannot connect to the Web Service during the recycle process.
Any advice would be greatly appreciated on this.
Update on this:
I tried to use Session Variable and set the mode to State Server, but apparently some DLLs which I am using cannot be serialized, thus I am back to square one.
Is there a better way to do this?
One thing to check is if your Session is getting clobbered. If your using the (default) in-memory Session, then sessions die anytime an ASP.NET worker process is recycled. If this is causing your issue, you might have to look into using the ASP.NET SessionServer in IIS, or SQL Server as your Session storage.
I'd try to limit your use of the Session for this type of thing as much as possible. If you utilize the Web.Config for storing your connection strings, you can access it at anytime and it will not expire on you like the session.
You may also consider having a static data access class if it is the same for all users for the applications instance...
This is the exact same problem I was experiencing, and it turned out to be the session variables dying so the connection string couldn't be initialized properly.
Why cant your app use just one connection string? and why does the connection string need to come in through a webservice? that adds a huge amount of latency to the entire process. I assume it probably has to do something with data security in your database.
If you have to do this, I'd say, can't you have a fall back / default connectionstring? Wrap your code that attempts to pull it out of the session with some error handeling and if it fails revert to your default?
I would place your web service and web app in different app pools. Then increase the timeout length of your web apps pool.