Is anyone aware of an application I might be able to use to record or data-log network delays? Or failing that, would it be feasible to write such a program?
I work for a big corporation who have recently deployed a remote file management platform which is causing severe productivity issues for staff in our branch. We work direct to a server, and every time a file is saved now, there is a significant delay (generally between 5-15 seconds, but sometimes timing out all together). Everything is just extremely unresponsive & slow, and it makes people avoid saving files so often, so quite frequently, when crashes occur, quite a bit of work is lost.
And these delays don't only occur on save operations. They also occur on navigating the network file structure. 2-3 seconds pause / outage between each folder-hop is incredibly frustrating, and adds up to a lot of time when you add it all up.
So when these delays occurs, it freezes out the rest of the system. Clicking anywhere on screen or on another application does nothing until the delay has passed its course.
What I am trying to do is to have some sort of data-logger running which records the cumulative duration of these outages. The idea is to use it for a bit, and then take the issue higher with evidence of the percentage of lost time due to this problem.
I suspect this percentage to be a surprising one to managers. They appear to be holding their heads in the sand and pretending like it only takes away a couple of minutes a day. Going by my rough estimates, we should be talking hours lost per day (per employee), not minutes. :/
Any advice would be greatly appreciated.
You could check if the process is responding using C#. Just modify the answer from the other question to check for the name of the application (or the process id, if possible with that C# API) and sum up the waiting times.
This might be a bit imprecise, because Windows will give a process a grace period until it declares it "not responding", but depending on the waiting times might be enough to make your point.
I'm not sure how your remote file management platform works. In the easiest scenario where you can access files directly on the platform you could just create a simple script that opens files, navigates the file system and lists containing dirs/files.
In case it's something more intricate, I would suggest to use a tool like wireshark to capture the network traffice, filter out the relavant packets and do some analysis on their delay. I'm not sure if this can be done directly in wireshark otherwise I'd suggest to export it as a spreadsheet csv and do you analysis on that.
Related
Yesterday, my customer played with the IIS settings and he changed the number of Working Process to 2 which made my Web Application ran very weird, the Session State sometimes was lost, sometimes was recovered and took me a day to find out what happened. So, the question is Multiple Working Process is useful in what situation?
It can be useful for scaling a web app vertically. Especially poorly designed ones that do too much work in the web processes or one where processes crash frequently so you always have a hot one. It isn't an option that should be exercised lightly as you have found out, but it is good to have when you need it.
The reason your user sessions started to fail was because you are using the default in-process session state module. This is fairly easy to fix as well -- just run the session state out of process using either the session state service or a database. Note that some behavior of the session state changes when you do this as well, so you will need to test carefully that you don't break something else.
I got a dedicated server running both IIS 7.5 and SQL Server 2010. Server CPU load is often near 100%. The SQL server does not take too much but the w3wp process is taking a significant amount of CPU (often 70+%).
I'd like to find out, what is causing this pressure:
* Too many requests of static files (a CDN could be added)
* Too many ajax requests (I am thinking about comet/web sockets anyways)
* Single asp.net pages consuming too much processing power (should be easy to optimize)
Where would you start looking to find out where to start optimizing?
The easiest possible way is to profile the app in production. Not sure if that is possible in your case. Some options:
look into the logs and look at the duration of the requests. Long requests are likely to put load on the system
Remote debug w3wp with Visual Studio and pause the debugger 10 times to see where it stops most. That is the hot spot
Use XPerf or PerfView to capture (managed) stacks. This has almost no impact on production performance
A good starting point would be to fire up the development tools (F12 in IE / Chrome) and look at the timings under the network tab. That will show you a waterfall-style diagram for how the page has loaded and should help you identify any particularly slow-loading static files which might be sensibly moved off to a cdn, any unnecessary requests being made, how much time is being spent getting the actual page itself, etc.
After that, profile the application with a performance profiler. A good profiler like ANTS Performance Profiler will let you look at things like execution time / hit counts for different methods, as well as what database queries are being run and how long they’re taking. A new version of ANTS (currently in EAP) will also group that activity by http request so you can see if specific pages need optimisation or are being hit too many times.
You'd also do well to check that caching is working as you intend it so that users aren’t unnecessarily re-requesting pages.
There's also a nice article on ASP.NET performance which you might want to read at http://aspalliance.com/1533_ASPNET_Performance_Tips.7.
Disclaimer: I work for Red Gate which makes ANTS.
I found an easy way to see what's going on on the server.
Nevertheless, the professional way is probably to go and use a profiling tool.
What did I do?
In IIS Console you can get a list of all current worker threads and if you choose one you can see what this thread is working on. So I was able to see that the thread was handling 100 requests in parallel, 70 of those were tracing back to the same ajax call.
The immediate solution was to reduce the frequency of that call (from every 10 to every 30 seconds). The next step will be to further optimize the call on the server side since I do have other ajax calls with the same frequency (every 10 seconds) which nearly never showed up in the active requests list since they were so fast.
Probably the easiest way to figure it out would be to install New Relic on the server. The trial lasts 30 days I think so it should give you enough time to get to the bottom of this. It'll show you long-running SQL queries, .NET methods, as well as just about everything else you can think of. It makes it very easy to identify bottlenecks.
By the way, I suggested New Relic because it sounds like your problem is in a production environment. New Relic isn't an incredibly detailed profiler. It gathers enough information to be helpful, but not so much as to slow down the server. That makes it well suited to this purpose.
If, however, you could reproduce the problem in a development environment you might try something like the free Eqatec profiler.
This has been a hard problem to diagnose and I was hoping someone could help me out.
We have an AIR app (www.trainerroad.com) that uses Native Process to read data from a command line that comes from a USB stick. It's a cycling app the data is heart rate, cadence, speed and power.
While someone workouts, we record that data to a sqlite database every second. We have a sqlite connection to the database. We have a timer that ticks every second and that's what triggers our inserts.
We have a few hundred users. A few of them have seen it where the app will run for 45 minutes fine, then hang for 30 seconds to a couple minutes. Then, it will release and the clock in the workout will tick down really fast. That means the timers are all getting hit.
This makes me think that something is hanging the app, but I can't figure out what. The app executed the same code every second for 45 or so minutes, then suddenly freezes. (btw, it isn't EVERY 45 minutes, but it does happen after the workout has been running for a while).
I'm thinking it has to do with SQLite. Maybe the database is being compacted while it's being executed against? I'm stumped and I can't reproduce this locally.
Does anyone have any ideas of how to debug this or what area of my app might be causing this? I know it's a hard one.
update
I ran the memory profiler for 20 minutes while it was running and I didn't see peak memory usage increase at all. I could tell it was garbage collecting by looking at the peak vs current memory usage.
I've added two system.gc() calls every 60 ticks. I'm hoping this doesn't make it gittery, but I'm not doing much while it plays so I think I should be good. Not sure if this is what was causing the problem though.
It looks like the manual garbage collection has fixed the issue. We haven't seen this since we put that upgrade in.
I'm looking to create a webpage that will reflect the status of one of my company's servers automatically. Frequently there will be a minor error that only lasts 2-3 minutes, and it would be great to have this reflected on a self-generated page, which might prevent 50-60 unhappy clients from calling in simultaneously and asking what's wrong.
I'm not quite sure where to begin - would anyone have a suggestions for good resources to study? Programming examples? I'm not referring to the basics of writing an ASP.NET page, of course, but rather process interaction in Windows.
Thanks.
To pull this off, you'd need a separate page that essentially runs server diagnostics, otherwise the page wouldn't know if it was up or down. Also, the page would need to be isolated from the sort of problems that are kill other people's requests, such as cache hit problems, memory starvation, high CPU usage, insufficient bandwidth. So ideally the diagnostics would run in a separate app-pool, separate virtual directory, separate machine.
Many of the interesting diagnostics would require a WMI call, but some you can get from the My.Computer namespace.
Also, are you going to do this on every server, or do you want one web server to display the status of several different servers?
It also depends on the type of errors your servers are encountering.
If they are going down completely, or are losing internet connection, then pinging them after an interval of time will let you know if they are up or not.
If you have a specific process running on a server that becomes unavailable, that can be a little more tricky.
Your best bet is to find a way to do a simple request from the services/applications that are important and see if you get a response, if you do, the server is likely up, if not, then it is likely not.
Anything you can do to reduce the number of support calls you get is a good idea, but I'd also focus some time and try to figure out why your servers are going down so often.
Also, telling your users that the server is down, but not giving a reason why may not give the effect you are looking for. Users will still be confused and frustrated when they can't get their work done.
I know you were looking to build a webpage to display the server diagnostics, but there are plenty of server monitoring tools that produce webpages for an easy dashboard view of the history.
A quick google returned the following link:
http://www.webdesignbooth.com/10-really-useful-server-monitoring-tools/
I was wondering if anyone had experience they could share using the Response.IsClientConnected property as a performance optimization for asp.net websites.
The reason I ask is that I am a bit skeptical on how effective it would be in real life scenarios. I understand the concept of checking the value before performing a large task but I just can't see how useful this would be as clients could disconnect at any point time.
I think the main usage would be for optimizing the delivery of long processes. For example, if you had to generate a huge report or something, you might run the report in a separate thread and then periodically check to see if the user is still connnected. If not, you could kill this long running process so that it is not running needlessly since the user is no longer expecting a response.
This helps to prevent users from starting long processes and then making more requests over and over because they might think it is slow or something. If you were not doing this type of checking, you could tax your server due to all the requests even though all but one is valid. This scenario could be handled by allowing only one user to run one long running task, but it would also help in a multi-user environment as well to make sure you are only spending time serving up requests where the user is still connected and waiting for the response.
Note: I have never actually used this before, this is just based on my very basic understanding of what I have read.
I have used this extensively in my applications and it can give you a huge saving on resources.
Try this: create a page that needs -some- time to complete and try refresh it many many times before it complete. You will see that requests are queued to be executed. Imagine a user that has a slow connection and refreshes his page many many times thinking this will fetch the page (a very common issue from what a site can die out of resources when all users are connected and for some reason it becomes slow).
Now, change it and at the start of each page load, (or sooner at page init) check if HttpContext.Current.Response.IsClientConnected and in the case that he is not connetced throw a threadabord exception. You will see, your site will respond much sooner.
Actually I check if client is connected before any heavy action on the page so as to prevent needless executions. In production environments, I have seen that especially in cases where the system becomes slow, this validation will help much.