This has been a hard problem to diagnose and I was hoping someone could help me out.
We have an AIR app (www.trainerroad.com) that uses Native Process to read data from a command line that comes from a USB stick. It's a cycling app the data is heart rate, cadence, speed and power.
While someone workouts, we record that data to a sqlite database every second. We have a sqlite connection to the database. We have a timer that ticks every second and that's what triggers our inserts.
We have a few hundred users. A few of them have seen it where the app will run for 45 minutes fine, then hang for 30 seconds to a couple minutes. Then, it will release and the clock in the workout will tick down really fast. That means the timers are all getting hit.
This makes me think that something is hanging the app, but I can't figure out what. The app executed the same code every second for 45 or so minutes, then suddenly freezes. (btw, it isn't EVERY 45 minutes, but it does happen after the workout has been running for a while).
I'm thinking it has to do with SQLite. Maybe the database is being compacted while it's being executed against? I'm stumped and I can't reproduce this locally.
Does anyone have any ideas of how to debug this or what area of my app might be causing this? I know it's a hard one.
update
I ran the memory profiler for 20 minutes while it was running and I didn't see peak memory usage increase at all. I could tell it was garbage collecting by looking at the peak vs current memory usage.
I've added two system.gc() calls every 60 ticks. I'm hoping this doesn't make it gittery, but I'm not doing much while it plays so I think I should be good. Not sure if this is what was causing the problem though.
It looks like the manual garbage collection has fixed the issue. We haven't seen this since we put that upgrade in.
Related
I don't know if this is a common issue but in my symfony2 project when I load for the first time it takes around 4 seconds (Profiler saying) after a direct refresh , it takes around 100ms.
I know that it needs to create the cache that's why it takes a long time.
But when I wait around 10 mins and refresh, it takes again around 4s to load.
Does the cache get deleted and symfony2 tries to recreate it ?
How can I fix this issue?
It depends on what your setup is.
Could be the symfony cache being dropped, could be a mysql connection timing out and for some reason being slow to reconnect, could be the mysql cache being dropped.
You could lookup in the profiler where the time is used, maybe that gives you a clue.
Yesterday, my customer played with the IIS settings and he changed the number of Working Process to 2 which made my Web Application ran very weird, the Session State sometimes was lost, sometimes was recovered and took me a day to find out what happened. So, the question is Multiple Working Process is useful in what situation?
It can be useful for scaling a web app vertically. Especially poorly designed ones that do too much work in the web processes or one where processes crash frequently so you always have a hot one. It isn't an option that should be exercised lightly as you have found out, but it is good to have when you need it.
The reason your user sessions started to fail was because you are using the default in-process session state module. This is fairly easy to fix as well -- just run the session state out of process using either the session state service or a database. Note that some behavior of the session state changes when you do this as well, so you will need to test carefully that you don't break something else.
I got a dedicated server running both IIS 7.5 and SQL Server 2010. Server CPU load is often near 100%. The SQL server does not take too much but the w3wp process is taking a significant amount of CPU (often 70+%).
I'd like to find out, what is causing this pressure:
* Too many requests of static files (a CDN could be added)
* Too many ajax requests (I am thinking about comet/web sockets anyways)
* Single asp.net pages consuming too much processing power (should be easy to optimize)
Where would you start looking to find out where to start optimizing?
The easiest possible way is to profile the app in production. Not sure if that is possible in your case. Some options:
look into the logs and look at the duration of the requests. Long requests are likely to put load on the system
Remote debug w3wp with Visual Studio and pause the debugger 10 times to see where it stops most. That is the hot spot
Use XPerf or PerfView to capture (managed) stacks. This has almost no impact on production performance
A good starting point would be to fire up the development tools (F12 in IE / Chrome) and look at the timings under the network tab. That will show you a waterfall-style diagram for how the page has loaded and should help you identify any particularly slow-loading static files which might be sensibly moved off to a cdn, any unnecessary requests being made, how much time is being spent getting the actual page itself, etc.
After that, profile the application with a performance profiler. A good profiler like ANTS Performance Profiler will let you look at things like execution time / hit counts for different methods, as well as what database queries are being run and how long they’re taking. A new version of ANTS (currently in EAP) will also group that activity by http request so you can see if specific pages need optimisation or are being hit too many times.
You'd also do well to check that caching is working as you intend it so that users aren’t unnecessarily re-requesting pages.
There's also a nice article on ASP.NET performance which you might want to read at http://aspalliance.com/1533_ASPNET_Performance_Tips.7.
Disclaimer: I work for Red Gate which makes ANTS.
I found an easy way to see what's going on on the server.
Nevertheless, the professional way is probably to go and use a profiling tool.
What did I do?
In IIS Console you can get a list of all current worker threads and if you choose one you can see what this thread is working on. So I was able to see that the thread was handling 100 requests in parallel, 70 of those were tracing back to the same ajax call.
The immediate solution was to reduce the frequency of that call (from every 10 to every 30 seconds). The next step will be to further optimize the call on the server side since I do have other ajax calls with the same frequency (every 10 seconds) which nearly never showed up in the active requests list since they were so fast.
Probably the easiest way to figure it out would be to install New Relic on the server. The trial lasts 30 days I think so it should give you enough time to get to the bottom of this. It'll show you long-running SQL queries, .NET methods, as well as just about everything else you can think of. It makes it very easy to identify bottlenecks.
By the way, I suggested New Relic because it sounds like your problem is in a production environment. New Relic isn't an incredibly detailed profiler. It gathers enough information to be helpful, but not so much as to slow down the server. That makes it well suited to this purpose.
If, however, you could reproduce the problem in a development environment you might try something like the free Eqatec profiler.
Is anyone aware of an application I might be able to use to record or data-log network delays? Or failing that, would it be feasible to write such a program?
I work for a big corporation who have recently deployed a remote file management platform which is causing severe productivity issues for staff in our branch. We work direct to a server, and every time a file is saved now, there is a significant delay (generally between 5-15 seconds, but sometimes timing out all together). Everything is just extremely unresponsive & slow, and it makes people avoid saving files so often, so quite frequently, when crashes occur, quite a bit of work is lost.
And these delays don't only occur on save operations. They also occur on navigating the network file structure. 2-3 seconds pause / outage between each folder-hop is incredibly frustrating, and adds up to a lot of time when you add it all up.
So when these delays occurs, it freezes out the rest of the system. Clicking anywhere on screen or on another application does nothing until the delay has passed its course.
What I am trying to do is to have some sort of data-logger running which records the cumulative duration of these outages. The idea is to use it for a bit, and then take the issue higher with evidence of the percentage of lost time due to this problem.
I suspect this percentage to be a surprising one to managers. They appear to be holding their heads in the sand and pretending like it only takes away a couple of minutes a day. Going by my rough estimates, we should be talking hours lost per day (per employee), not minutes. :/
Any advice would be greatly appreciated.
You could check if the process is responding using C#. Just modify the answer from the other question to check for the name of the application (or the process id, if possible with that C# API) and sum up the waiting times.
This might be a bit imprecise, because Windows will give a process a grace period until it declares it "not responding", but depending on the waiting times might be enough to make your point.
I'm not sure how your remote file management platform works. In the easiest scenario where you can access files directly on the platform you could just create a simple script that opens files, navigates the file system and lists containing dirs/files.
In case it's something more intricate, I would suggest to use a tool like wireshark to capture the network traffice, filter out the relavant packets and do some analysis on their delay. I'm not sure if this can be done directly in wireshark otherwise I'd suggest to export it as a spreadsheet csv and do you analysis on that.
I'm developing an app for a client and one of his devices (2nd gen iTouch on iOS4) is having issues starting the application. I've run a few allocation/leak tests and I concluded that there isn't anything wrong with my app's code. I noticed that there is an allocation spike at startup and I concluded that it's because of dyld which is dynamically linking the libraries on start up. How would I go about pre-binding the application in xcode4?
OS X forum seemed to be extremely non informative in that they assume you'd be able to find it. :/
Any help would be appreciated.
Thanks!
(I also wish I could make a new tag for "prebinding")
According to Apple, you shouldn't need to prebind your iOS applications. If you are getting big allocation spikes, I'm guessing it's due to your app's architecture rather than the underlying OS itself.
The memory allocated by dyld should pale in insignificance compared to even the most basic allocations made by the earliest stages of the runtime. The Objective-C runtime and other system frameworks/libraries allocate a bunch of internal structures that are required for things to work correctly.
For instance, a quick test of an app that does nothing in main but make one call to NSLog(#"FooBar"); and then sleep (i.e. never even spools up UIApplication) performed 373 allocations for a total of 52K living.
To take it a step further, if you actually start up UIKit, like this...
UIApplicationMain(argc, argv, nil, NSStringFromClass([MyAppDelegate class]));
... you'll see ~600K in ~7800 living allocations once the app reaches quiescent state. This is all unavoidable stuff. No amount of prebinding will save you this. I suggest not worrying about it.
If you're seeing orders of magnitude more memory being allocated then, as Nik Reiman said, it's your application. At the end of the day, the memory allocated by the dynamic linker is totally insignificant.