I have a page that shows statistics for users, this cannot be cached because each user has different statistics and there are many thus the real time query must be made.
What the way to avoid database server overload when user will click F5's to refresh or to ask different queries in short time intervals ?
I think #Jens A. is halfway there - this is a perfect case for caching, calculate the stats, stick them into the cache with a fixed expiry time and then only calculate them if they're not in the cache. By having the expiry time set to an appropriate value (5 minutes, less?) the stats will still be reasonably up to date and will change (update) at a reasonable without having to be calculated every time if the pages are being refreshed continuously.
You could store the generated statistics in your database for some time, and just show the old values if the statistics are requested again.
Related
My data is stocks in news so everyday once the data changes in my page but in order to keep the page up to date when the data is updated, I used revalidate:1 in getStaticProps it's working fine now but is this comparatively better than using getServerSideProps?
Note: My data updates only once in a day (every day morning)
We have to consider whether the data fetched per user is personalised in nature, if its personalised towards the user currently logged in, its better to use getServerSideProps as it fetches the data during each request and pre-renders it every time, which doesn't involve the usage of caching in any way.
In the case of ISR, it's better suited for rendering a page with content common to all users. Background regeneration ensures traffic is served uninterruptedly, always from static storage, and the newly built page is pushed only after it's done generating. Even when revalidating, the visitor first receives the cached version and only then the updated version. This caching strategy is commonly known as “stale-while-revalidate.”
Having a revalidate score of 1s is low , knowing that u get the update only once a day, it would be better to have it revalidate once every 60s or 120s.
Server-side rendering would be slow on the first render and might decrease the performance when compared to ISR.
This question relates to WordPress's wp-cron function but is general enough to apply to any DB-intensive calculation.
I'm creating a site theme that needs to calculate a time-decaying rating for all content in the system at regular intervals. This rating determines the order of posts on the homepage, which is paged to allow visitors to potentially view all content. This rating value needs to be calculated frequently to make sure the site has fresh content listed in the proper order.
The rating calculation is not heavy but the rating needs to be calculated for, potentially, 1,000s of items and doing that hourly via wp-cron will start to cause problems for sites with lots of content. Ignoring the impact on page load (wp-cron processes requests on page loads once a certain interval has been reached), at some point the script will reach a time limit. Setting up the site to use "plain ol' cron" will solve the page loading issue but not the timeout one.
Assuming that I have no control over the sites that this will run on, what's the best way to handle this rating calculation on a regular basis? A few things that came to mind:
Only calculate the rating for the most recent 1,000 posts, assuming that the rest won't be seen much. I don't like the idea of ignoring all old content, though.
Calculate the first, say, 100 or so, then only calculate the rating for older groups if those pages are loaded. This might be hard to get right, though, and lead to incorrect listing and ratings (which isn't a huge problem for older content but something I'd like to avoid)
Batch process 100 or so at regular intervals, keeping track of the last one processed. This would cycle through the whole body of content eventually.
Any other ideas? Thanks in advance!
Depending on the host, you're in for a potentially sticky situation. Let me outline a couple of ideal cases and you can pick/choose where you need to.
Option 1
Mirror the database first and use a secondary app (WordPress or otherwise) to do the calculations asynchronously against that DB mirror. When they're done, they can update a static file in the project root, write data to a shared Memcached instance, trigger a POST to WordPress' admin_post endpoint to write some internal state, whatever.
The idea here is that you're removing your active site from the equation. The last thing you want to do is have a costly cron job lock the live site's database or cause queries to slow down as it does its indexing.
Option 2
Offload the calculation entirely to a separate application. Tracking ratings in real time with WordPress is a poor idea as it bypasses page caching and triggers an uncachable request every time a new rating comes in. Pushing this off to a second server means your WordPress site is super fast, and it also means you can have the second server do the calculations for you in the first place.
If you're already using something like Elastic Search on the site, you can add ratings as an added indexing facet. Then just update posts as ratings change, and use the ES API to query most popular posts later.
Alternatively, you can use a hosted service like Keen IO to record and aggregate ratings.
Option 3
Still use cron, but don't schedule it as a cron job in WordPress. Instead, write a WP CLI routine that does the reindexing for you. Then, schedule real cron jobs to process the job.
This has the advantage of using PHP's command line version, which can be configured to skip the timeouts and memory limits imposed on the FPM/CGI/whatever version used to serve the site. It also means you don't have to wait for site traffic to trigger the job - and a long-running job won't block other cron events within WordPress from firing.
If using this process, I would set the job to run hourly and, each hour, run a batch of 1/24th of the total posts in the database. You can keep track of offsets or even processed post IDs in the database, the point is just that you're silently re-indexing posts throughout the day.
We have a very active web page which has lots of ajax and regular updates via jquery. It can load a huge amount of data (< 100k per minute) every user in peak situations and we had 2,000 people online during the last peak.
What we would like to do is count the number of concurrent users. If over 500 (and not a registered user) then bad luck, hit the road!
Has anyone got a class or some other process? Our server recycles every hour so I am thinking of an application level variable that adds one to the current count if successful (gold users are exempt from the test but are added to the quota so we may have 600 users).
Has anyone else played with this idea?
TIA
Just some ideas...
application.lock()
application('visitors') = application('visitors') + 1
application.unlock()
You should stress-test this solutions up to the numbers you want to allow. It will probably work is my fair guess.
Consider counting the ajax url page instead, that gives a more accurate estimate of the load. When going for session's you will not know when I've left. Counting via the Ajax line gives a more accurate number of visitors.
Just suggestion: in GLOBAL.ASA on Session OnStart you could increase count of running Sessions in some global (Application) variable.
Do not forget to decrease it in GLOBAL.ASA on Session OnEnd
We've noticed lately that as our site is growing, our data in Google Analytics is getting less reliable.
One of the places we've noticed this most strongly is on the "Realtime Dashboard".
When we were getting 30k users per day, it would show about 500-600 people on line at a time. Now that we are hitting 50k users per day, it's showing 200-300 people on line at a time.
(Other custom metrics from within our product show that the user behavior hasn't changed much; if anything, users are currently spending longer on the site than ever!)
The daily totals in analytics are still rising, so it's not like it's just missing the hits or something... Does anyone have any thoughts?
The only thing I can think of is that there is probably a difference in interpretation of what constitutes a user being on line.
How do you determine if the user is on line?
Unless there is an explicit login/logout tracking, is it possible that it assumes that a user has gone if there is no user generated event or a request from the browser within an interval of X seconds?
If that is the case then it may be worth while adding a hidden iframe with some Javascript code that keeps sending a request every t seconds.
You can't compare instant measures of unique, concurrent users to different time-slices of unique users.
For example, you could have a small number of concurrent unique users (say 10) and a much higher daily unique users number like 1000, because 1000 different people were there over the course of the day, but only 10 at any given time. The number of concurrent users isn't correlated to the total daily uniques, the distribution over the course of the day may be uneven and it's almost apples and oranges.
This is the same way that monthly unique and daily uniques can't be combined, but average daily uniques are a lower bound for monthly uniques.
I would like to be able to display time based on the user's current time zone. I will be storing times in UTC format in my database, but wish to normalize these to the client's time zone. So far the option that seems most viable is to capture the clients time and then deteremine the difference between that and current UTC and use that delta to normalize times when rendering. I would like to know if there are any more straight forward options available to detect and normalize a UTC time to that of the requesting client's machine.
I would use the javascript dateObject.getTimezoneOffset(). Even if their time isn't set accurately, hopefully they've set their timezone:
http://www.w3schools.com/jsref/jsref_getTimezoneOffset.asp
You can use this value to calculate an offset between the server's time and the client's time. (as Jason helpfully pointed out in a comment below)
In addition to, or instead of using getTimezoneOffset, you should consider permitting the user to specify preferred timezone in their profile. For instance, a user may be visiting in one timezone, yet may prefer to see time displayed in his home timezone.
Also, if this were as simple as calling a JavaScript function, then sites would never have to ask for your timezone.
A much better approach is to store per-user the exact time zone the user is in (and optionally) allow them to select if they observe daylight savings time (if it's applicable to their time zone.) You can of course, automatically fill in the DST information (based on the time zone) if you wanted.
You'd have to make this part of the sign-up or possibly in their "options".