I have what I consider to be a fairly simple application. A service returns some data based on another piece of data. A simple example, given a state name, the service returns the capital city.
All the data resides in a SQL Server 2008 database. The majority of this "static" data will rarely change. It will occassionally need to be updated and, when it does, I have no problem restarting the application to refresh the cache, if implemented.
Some data, which is more "dynamic", will be kept in the same database. This data includes contacts, statistics, etc. and will change more frequently (anywhere from hourly to daily to weekly). This data will be linked to the static data above via foreign keys (just like a SQL JOIN).
My question is, what exactly am I trying to implement here ? and how do I get started doing it ? I know the static data will be cached but I don't know where to start with that. I tried searching but came up with so much stuff and I'm not sure where to start. Recommendations for tutorials would also be appreciated.
You don't need to cache anything until you have a performance problem. Until you have a noticeable problem and have measured your application tiers to determine your database is in fact a bottleneck, which it rarely is, then start looking into caching data. It is always a tradeoff, memory vs CPU vs real time data availability. There is no reason to make your application more complicated than it needs to be just because.
An extremely simple 'win' here (I assume you're using WCF here) would be to use the declarative attribute-based caching mechanism built into the framework. It's easy to set up and manage, but you need to analyze your usage scenarios to make sure it's applied at the right locations to really benefit from it. This article is a good starting point.
Beyond that, I'd recommend looking into one of the many WCF books that deal with higher-level concepts like caching and try to figure out if their implementation patterns are applicable to your design.
Related
On my new job there is a web-application written in Visual Basic .NET with usage of ASP.NET Webforms framework for producing and rendering of webpages.
It runs on a Windows server and requires Microsoft IIS web-server as an application host. The project is developed with Microsoft Visual Studio 2010
as a development environment and uses Intersystems Cache database. The application has a layered architecture (Interface -, Business -, Data access layer).
We use Firefox (78.1.0.esr(64-bit) as browser (internal policy).
Users complain that they don't know when a page is loading / request is being processed.
Apparently in the past Firefox visualized an hourglass when the page was loading.
What is the easiest way to visualize an hourglass for each request (independent of the page)?
It's a very large application.
For postbacks, you could use the asp:UpdateProgress control.
See: https://learn.microsoft.com/en-us/dotnet/api/system.web.ui.updateprogress?view=netframework-4.8
It allow you to display anything you want during the time the postback is being processed. I assume you could also use a css-class which turns your pointer into a hourglass if you wanted..
Well, as noted you don't mention any performance issues for what task?
And the browser display (or not) is a HUGE MASSIVE different issue here.
Browsers even Edge, or Firefox, and in fact all browsers I know of STILL display a loading animation. and that includes FireFox.
Edge browser: (based on Chrome engine now)
And FireFox
I am not aware of any changes in this area. However, it is BEYOND LAUGHABLE that some comments here suggest using ajax or some such!!! - I can provide some links to truck driving school, or maybe fast food training, as that is a laughable and gut busting suggestion here.
In fact?
Introduction of ajax calls WILL FOR SURE remove any browser "waiting" or circle delay animated icon that MOST ALL browsers have when a page is loading.
Next up:
Performance:
Well, does a page on the web site without any data load fast? In other words, do you have (or can you add a test page to the site - "hello world" on the page.
does that page load fast, or do all pages load slow. Or do pages with data operations and pages that pull data are slow?
And did the site run fast at one time, and is now with more data is it running slow? Or was it always slow? YOU REALLY need to answer the above questions.
Since you using a post-sql (or non sql database), then data in that system is actually saved in a format VERY similar to JSON, or XML. Just that the multi-value format used in that Cache database was invented in 1970, but that database uses strings for the data store.
In effect like say using google, or even SharePoint? You can have millions and millions of documents. Such computers even with low processor and low memory can easy say run a motor vehicles department for a whole country - 100 million people say.
So what they do AMAZING:
If you need to pull a patient record, or say a customer invoice? Such systems even with say 1 billion records can pull that WHOLE invoice out of the data base with ONE disk operation and seek. They are blistering fast WHEN retrieving what amounts to a master/child record compared to a sql system.
However, while those systems can say represent a whole invoice with one "string" (just like you can with XML or JSON?). What such systems do VERY poorly is "row" processing. So, get a record, modify that record, save it back? Fantastic performance.
However, row processing, or say using sql statements to update a lot of rows? That system is REALLY slow. And in fact, things are even worse since the ODBC drivers are in fact translating sql statements into "no sql, and into the string base database system).
So, I would ensure the data files are sized correctly. So, if that system used to run fast, but now is slow? The data files (their base size) need to be re-sized, and thus the huge mass of what we call "linked frames" will be reduced, and performance should increase huge. But you have to check the Cache database. (assuming that you have a few years of experience with that database system (so years of experience with a multi-value database is a basic requirement here).
So, do all web pages load slow - even ones without data? Then the can't be the database.
Or, is this occurring for some database pages that involve say updates?
And did the developers use the "data cube" like data objects, or do they use the SQL (odbc) translator for dealing with the database?
And was the system fast at one time? or was it always slow.
And do pages without much (or any) database operations run fast).
And VERY imporant:
Does the site run fast with say 5 users, but then run slow with 50 users?
(but then again, I can't imagine you not asked these super simple questions that anyone would when attempting to evaluate performance). I mean, you can have say "average" doctors, but when you have a special medical mystery, then you need a Dr. House here - the best of the best). Same goes with computers. It is possible you are making the assumption that the original developers of the software and system were drunken unemployed rodeo clowns, but then again, maybe they did a REALLY good job, and the database or software been out grown due to server high loads during peak times. But then again, perhaps worse that these "basic" questions about performance have not yet been asked by you?
Out of one of our computing science classes, there was only about 2-3 out of a class of about 80 that would "naturally" write the fastest running code. (these days, unfortantly tracking things like code execution time and other metrics is often not even considered anymore, but it should be). Same goes for say formula 1 racing. You can take two teams, each with a 400 million dollar budget, but one team BLOWS AWAY the competition, and yet all the teams hire the best talent money can buy. So, hiring some developers to build software? Sure, lots of developers. Hiring developers with top notch performance in mind? Well, then that's when you seek out the big guns, the guru's - the top talent.
and even other basic questions such as :
Does the system run well on the test or development box?
Did the system run well at one time, but is not slower over time?
Or was it always slow? (and then what prompted this issue and idea that performance now needs to be fixed and addressed compared to no one doing anything 5 years ago?).
And when the site was becoming slow, were the developers of the site contacted? (or why, or why not?).
But, talking about a cube like "no-sql" database? And THEN ALSO introduction of a simple browser question of which all have that "animation" icon? That is two VERY different questions here. Have to really wonder how both of this issues could be mixed up and introduced to the same post and question? (yikes!!!!).
All current browsers I am aware of do have a "wait" type of animation. But such animation have VERY little to do with application and database performance optimizing. Toss in that you using a so called post-relational, or so called multi-value database, then you introducing an area of expertise that most posters here likely don't have. (so you getting silly suggestions about ajax and the like).
I have 10+ years of experience on those multi-value databases, and as noted, they are not fast at row processing, but pulling, update and save of a record? Then such systems can easy beat sql based systems performance wise. So, the fine art of performance? It is without question a question and process for the "top dogs" talent wise in our industry when attempting to deal with performance.
So, was the system fast at one time, and now is it slow?
Are all pages - even those without data from databased slow?
Or are only some pages with data operations slow?
Are they using Cache data objects, or are they using the database provider and sql?
so, what type of data provider(s) are being used in this application?
After a watching a few videos regarding DynamoDB and its best practices, I decided to give it a try; however, I cannot help but feel what I'm doing may be an anti-pattern. As I understand it, the best practice is to leverage as few tables as possible while also taking advantage of GSIs to do some 'heavy' lifting. Unfortunately, I'm working with a use case that doesn't actually have strictly defined access patterns yet since we're still in early development.
Some early access patterns that we may see are:
Retrieve the number of wins for a particular game: rock paper scissors, boxing, etc. [1 quick lookup]
Retrieve the amount of coins a user has. [1 quick lookup]
Retrieve all the items that someone has purchased (don't care about date). [Not sure?]
Possibly retrieve all the attributes associated with a user (rps wins, box wins, coins, etc). [I genuinely don't know.]
Additionally, there may be 2 operations we will need to complete. For example, if the user wins a particular game they may receive "coins". Effectively, we'll need to add coins to the user "coins" attribute & update their number of wins for the game.
Do you think I should revisit this strategy? Additionally, we'll probably start creating 'logs' associated with various games and each individual play.
Designing a DynamoDB data model without fully understanding your applications access patterns is the anti-pattern.
Take the time to define your entities (Users, Games, Orders, etc), their relationship to one another and your applications key access patterns. This can be hard work when you are just getting started, but it's absolutely critical to do this when working with DynamoDB. How else can we (or you, or anybody) evaluate whether or not you're using DDB correctly?
When I first worked with DDB, I approached the process in a similar way you are describing. I was used to working with SQL databases, where I could define a few tables and rely on the magic of SQL to support my access patterns as my understanding of the application access patterns evolved. I quickly realized this was not going to work if I wanted to use DynamoDB!
Instead, I started from the front-end of my application. I sketched out the different pages in my app and nailed down the most important concepts in my application. Granted, I may not have covered all the access patterns in my application, but the exercise certainly nailed down the minimal access patterns I'd need to have a usable app.
If you need to rapidly prototype your application to get a better understanding of your acecss patterns, consider using the skills you and your team already have. If you already understand data modeling with SQL databses, go with that for now. You can always revisit DynamoDB once you have a better understanding of your access patterns and determine that your application can benefit from using a NoSQL databse.
I have a ASP.NET web application (.NET 2008) using MS SQL server 2005. I want to increase the performance of the web site. Does anyone know of an article containing steps to do that, step by step, in SQL (indexes, etc.), and in the code?
Performance tuning is a very specific process. I don't know of any articles that discuss directly how to achieve this, but I can give you a brief overview of the steps I follow when I need to improve performance of an application/website.
Profile.
Start by gathering performance data. At the end of the tuning process you will need some numbers to compare to actually prove you have made a difference. This means you need to choose some specific processes that you monitor and record their performance and throughput.
For example, on your site you might record how long a login takes. You need to keep this very narrow. Pick a specific action that you want to record and time it. (Use a tool to do the timing, or put some Stopwatch code in you app to report times. Also, don't just run it once. Run it multiple times. Try to ensure you know all the environment set up so you can duplicate this again at the end.
Try to make this as close to your production environment as possible. Make sure your code is compiled in release mode, and running on real separate servers, not just all on one box etc.
Instrument.
Now you know what action you want to improve, and you have a target time to beat, you can instrument your code. This means injecting (manually or automatically) extra code that times each method call, or each line and records times and or memory usage right down the call stack.
There are lots of tools out their that can help you with this and automate some of it. (Microsoft's CLR profiler (free), Redgate - Ants (commercial), the higher editions of visual studio have stuff built in, and loads more) But you don't have to use automatic tools, it's perfectly acceptable to just use the Stopwatch class to time each block of your code. What you are looking for is a bottle neck. The likely hood is that you will find a high proportion of the overall time is spent in a very small bit of code.
Tune.
Now you have some timing data, you can start tuning.
There are two approaches to consider here. Firstly, take an overall perspective. Consider if you need to re design the whole call stack. Are you repeating something unnecessarily? Or are you just doing something you don't need to?
Secondly, now you have an idea of where your bottle neck is you can try and figure out ways to improve this bit of code. I can't offer much advice here, because it depends on what your bottle neck is, but just look to optimise it. Perhaps you need to cache data so you don't have to loop over it twice. Or batch up SQL calls so you can do just one. Or tighten your query filters so you return less data.
Re-profile.
This is the most important step that people often miss out. Once you have tuned your code, you absolutely must re-profile it in the same environment that you ran your initial profiling in. It is very common to make minor tweaks that you think might improve performance and actually end up degrading it because of some unknown way that the CLR handles something. This is much more common in managed languages because you often don't know exactly what is going on under the covers.
Now just repeat as necessary.
If you are likely to be performance tuning often I find it good to have a whole batch of automated performance tests that I can run that check the performance and throughput of various different activities. This way I can run these with every release and record performance changes each release. It also means that I can check that after a performance tuning session I know I haven't made the performance of some other area any worse.
When you are profiling, don't always just think about the time to run a single action. Also consider profiling under load, with lots of users logged in. Sometimes apps perform great when there's just one user connected, but when they hit a certain number of users suddenly the whole thing grinds to a halt. Perhaps because suddnely they are spending more time context switching or swapping memory in and out to disk. If it's throughput you want to improve you need to be figuring out what is causing the limit on throughput.
Finally. Check out this huge MSDN article on Improving .NET Application Performance and Scalability. Specifically, you might want to look at chapter 6 and chapter 17.
I think the best we can do from here is give you some pointers:
query less data from the sql server (caching, appropriate query filters)
write better queries (indexing, joins, paging, etc)
minimise any inappropriate blockages such as locks between different requests
make sure session-state hasn't exploded in size
use bigger metal / more metal
use appropriate looping code etc
But to stress; from here anything is guesswork. You need to profile to find the general area for the suckage, and then profile more to isolate the specific area(s); but start by looking at:
sql trace between web-server and sql-server
network trace between web-server and client (both directions)
cache / state servers if appropriate
CPU / memory utilisation on the web-server
I think First of all you have to find your Bottlenecks and then try to improve those.
This helps you to perform exactly where you have serios problem.
An in addition you needto improve your Connection to DB. For exampleusing a Lazy , Singletone Pattern and also create Batch request instead of single requests.
It help you to decrease DB connection.
Check your cache and suitable loop structures.
another thing is to use appropriate types, forexample if you need int donot create a long and etc
at the end ypu can use some Profiler (specially in SQL) andcheckif your queries implemented as well as possible.
I just finished working on a project for the last couple of months. It's online and ready to go. The client is now back with what is more or less a complete rewrite of most parts of the application. A new contract has been drafted and payment made for the additional work involved.
I'm wondering what would be the best way to start reworking this whole thing. What are the first few things you would do? How would you rework the design in a way that you stay confident that the stuff you're changing does not break other stuff?
In short, how would you tackle drastic application design changes efficiently (both DB and code)?
Presuming that you have unit tests in place, this is just refactoring.
If you don't have unit tests in place, then
Write unit tests for the parts you're likely to keep.
Write unit tests for the parts you're going to change.
Run the tests. The "keep" should pass. The "change" should fail.
Start refactoring until the tests pass.
This is NOT-A-NEW thing in software and people have done this and written a lot about this.
Try reading
Working Effectively with Legacy
Code
Refactoring Databases:
Evolutionary Database Design
The techniques explained here are invaluable to sustain any kind of long running IT projects.
Database design is different from application design in this regard.
Very often, client rethinking changes the application completely, but changes little, if anything, in the fundamental underlying data model of the enterprise. The reason for this is that clients tend to think in terms of business processes, but not in terms of fundamental data. Business processing and data processing are tightly coupled. Data storage is less tightly coupled.
In the days of classical database design, designers learned how to exploit this pattern, by dividing their database design into (at least) two layers: logical design and physical design. There are any number of times that a change of business process requires a complete rewrite of the application, and a major rework of the database physical design, but requires few, if any, changes to the logical design.
If your database design didn't separate out the layers like this, it's hard to tell what gets affected and what doesn't. Start with your tables and columns. Ask yourself if any of the changes require removing any column from the table it's in, or require inventing new columns. If the answer is no, you're in luck. Next, look at the constraints placed on the database (things like PRIMARY KEY, FOREIGN KEY, UNIQUE and NOT NULL). These constraints might be tightened or loosened by the client's changes. If not, you're in luck. If you didn't declare any constraints in the database, and chose to do all your integrity protection in application code, you're probably out of luck.
You still have a fair amount of work to do in terms of changing the indexes on the tables, and the way the application works with the data. But you've salvaged part of the investment in the old system.
The application itself is much more vulnerable to client changes in process than the database. If your database design was completely driven by your application design, you may be out of luck.
If it's THAT drastic of a change it might be best to just start over. I've worked on a number of projects that have gone through some drastic changes.
Starting over gives you a chance to use experience learned since the last project and provide a more efficent product.
I would recommend against trying to re-work the old site into the new site, you'll probably spend more time fiddling around changing things than you would have if you had just re-written it.
Best of luck to you !
How would you rework the design in a way that you stay confident that the stuff you're changing does not break other stuff? In short, how would you tackle drastic application design changes efficiently (both DB and code)?
Tests, code complexity/coverage metrics, and a continuous integration system. Run them early and often, so you know which parts are the riskiest and where to start writing.
These will become your safety nets when you have to make potentially problematic changes. If something does break, your CI system will tell you, and you won't have spent weeks down some rabbit hole before you realize there's a problem.
Sometimes you do things better the second time around so just try and stay positive. Plus you will have more domain knowledge this time around.
New to .net and was wondering if there is a performance gain to keeping an instance of, for example a DAL object in scope?
Coming from the Coldfusion world I would instanciate a component and store it in the application scope so that every time my code needed to use that component it would not have to be instanciated over and over again effecting performance.
Is there any benefit to doing this in ASP.Net apps?
Unless you are actually experiencing a performance problem, than you need not worry yourself with optimizations like this.
Solve the business problems first, and use good design. As long as you have a decent abstraction layer for your data access code, then you can always implement a caching solution later down the road if it becomes a problem.
Remember that any caching solution increases complexity dramatically.
NO. In the multi-tier world of .asp this would be considered a case of "premature optimization". Once a sites suite of stubs, scripts and programs has scaled up and been running for a few months then you can look at logs and traces to see what might be cached, spawned or rewritten to improve performance. And as the infamous Jeff Atwood says "Most code optimizations for web servers will benifit from money being spent on new and improved hardware rather than tweaking code for hours and hours"
Yes indeed you can and probably should. Oftentimes the storage for this is in the Session; you store data that you want for the user.
If it's a global thing, you may load it in the Application_Start event and place it somewhere, possibly the HttpCache.
And just a note, some people use "Premature Optimisation" to avoid optimising at all; this is nonsense. It is reasonable to cache in this case.
It is very important to do the cost benefit analysis before caching any object, one must consider all the factors like
Performance advantage
Frequency of use
Hardware
Scalability
Maintainability
Time available for delivery (one of the most important factor)
Finally, it is always useful to cache object which are very costly to create or you are using very frequently i.e. Tables's Data (From DB) or xml data
Does the class you are considering this for have state? If not, (and DAL classes often do not have state, or do not need state), then you should make it's methods static, and then you don't need to instantiate it at all. If the only state it holds is a connection string, you can also make that property field a static property field, and avoid the requirement of instantiating it that way.
Otherwise, take a look at the design pattern called Flyweight