So I've attached the resource usage of my wordpress website over the past 30 days.
You can see the I/O usage has been getting higher and more frequent. I think this is a problem that has caused a massive drop in visits to my site.
I asked my host why this is and he said backs up usually contribute largely to this. Only thing is, I backup once a month not every day.
I've tried optimising my database, disabling plugins but I don't understand why it keeps getting higher.
I have a Analytics plugin that refreshes every hour but I've had that all year and I/O usage only started getting high recently.
The only thing I can think of is Wp Super Cache and CloudFlare not working well together. I've tried different configurations but hasn't helped.
Any help would be appreciated.
I think this is a pretty standard IO log, Over time your db does get a lot bigger and so does your users who end up using a lot of IO. I dont think there is anything to panic right away, but obviously if this is a very huge difference from what you are used to see normally then i think you should look into it seriously. I take caching very seriously and i usually use W3 total cache for this kind of performance optimization. Its a bit tricky in the begining but once you are used to it, it very easy.
I know you might just want to improve the IO, for which mostly you just need caching but here are somethings that i would do to get the most performance out of a site.
1) If you are using a VPS or dedicated server install memcache or something like Redis, and then configure your plugin according to it. You might have to enable it in your php.ini file but once installed you will see the difference. It will execute the code and give you a save the results in the RAM, on the next request instead of executing the php code it will just hand over the same results. Now it depends on your website, and whether you want to cache it or not. You can setup individual pages to use caching as well.
2) If your plugin has options to automatically minify and combine html/css/js files then use it, if not then you should minify and combine them into a single file or as less number of files as possible and then manually upload to your server. It will reduce a lot of time that is spent on requesting a file and waiting for getting the response back. Its usually in milliseconds but if you have a lot of files then it does add up to seconds + unnecessary load on the server.
3)If your plugin has gzip feature, then enable it. It will allow your users to download the gzipped css and js files instead of the original large files. This will enormously reduce the number of bits a browser have to download on every attempt.
4) Enable caching of files on the browser, your plugin might already have this, but if not then you will have to set some headers which will tell the browser to cache the css and js files in the user browser. So the next time when the user goes to the next page on your website, instead of calling the css/js files from the server it loads them directly from the Cache.
5) Upload your css/js/images files to a CDN, that way whenever someone requests a file it will use the shortest route to get your users browser.
6) If your site is not just a personal blog and is making serious money or you just want to please all the huge growing number your users. Then i would suggest you look into auto scaling server platforms, where you set some triggers and the number of servers automatically increase when facing a lot of users / IO and once the number of users go back to normal it automatically scales down. One of the big boys for this sort of service would be AWS beanstalk, microsoft azure. Or you can use beanstalkd with digital ocean which is a cheap alternative.
7) Wordpress is quite compatible with facebook's HHVM which is an opensource virtual machine designed to use php as just in time (JIT). Php is an interpreted language i.e its written in C/C++ (you can checkout the code at github), so when ever you refresh a page, hundred's of line of php code is interpretted by C++ and then compiled and executed. What HHVM does is that it compiles the code and keep it in memory, so when someone else requests the same page it already has a compiled version so it just executes and serves it. So it removes 30-40% of the compiling time from every request, which in turn makes your site 30-40% faster. Now PHP7 is already out last month and it does have a lot of performance upgrades, so if you are still not sure about HHVM you should definitely try upgrading to PHP7.
Related
My company uses SilverStripe v3.1.21, along with the Subsite module to display and administer a number of clients' websites that sell products. This results in close to 200 subsites and a page count in the tens of thousands. The websites are very slow to load and tools such as Google's PageSpeed tell us page speeds are poor. We've already done step like combining and minimising the JS and compressing resources such as imaging, which gave some improvements, however the pages remain slow. The system was handed to us in this state and further hardware upgrades are not on the table as an option, nor are gaining additional resources for redevelopment.
We've taken a look at the static publish module (https://github.com/silverstripe/silverstripe-staticpublisher) and found that when we generating static pages the pages become fast and get a good score on the various tools, however the process to regenerate all of these pages takes over 14 hours, which is unacceptable given these products are updated from an external source daily. We also find that the regeneration process is a memory hog, as the module builds all of the pages in memory before dumping to file, causing the process to crash. We've had to alter the process to go subsite-by-subsite just to make it run.
We then took a look at the static publishing queue module (https://github.com/silverstripe/silverstripe-staticpublishqueue), which seemed to address our issues by having it queue pages as needed for regeneration, making it much more responsive to changes. However, the module seems to be very buggy and often crashes when generating pages.
Has anyone had experience using these modules (or similar) with larger sites and may be able to provide any pointers or ideas on how to implement static publishing successfully?
We are using staticpublishqueue currently on several sites. The only problem we've had with it is crashing due to long builds and poor locking. Or to be precise it doesn't actually crash but keeps spawning more and more instances until the server becomes unresponsible.
I think we have a fix for this in our fork. At least we haven't had any problems after using the modified locking. You could try installing the fork instead of the official version. If this fixes things for you maybe we should make a pull request :)
First of: We only use staticpublishqueue, I don't have any experience in regards to the sub site module. So I can't speak for your exact combination.
We are using staticpublishqueue on a huge site. Setup: We have multiple servers running the SilverStripe Website. They share a MySQL Database and use Redis as a session store.
One great thing about staticpublishqueue: you can run it in parallel. So the servers all run an instance of staticpublishqueue and publish into a shared folder, which is then synced to a nginx load balancer in front of the actual webservers. Works quite nice, but it does not scale indefinitely. At some point the staticpublishqueue instances start to pick the same record to render and waste resources. I think about 6 is the max for us.
Couple of things we learned regarding staticpublishqueue:
do not run to many instances at the same time (see above)
make sure it has enough ram
make sure it runs as the same user as the website
the record look it uses is not compatible with a MariaDB Galera Cluster
If possible switch to SilverStripe 3.6.x and PHP7. The performance gain is huge.
We are migrating away from staticpublishqueue to Cloudflare (or maybe another CDN). Why? Because if a page that is requested has not been rendered yet the server will render it for each request individually and then throw it away. Until the que does a separate render for the cache. Total waste of resources, especially if you purge your cache after a sitewide layout change or something.
I know next to nothing about Drupal but I do have a question. We had a site, written in straight HTML and PHP, that loaded the main page in 1-2 seconds and made 25 requests to the server to get the data it needed. A new Drupal version of the site takes 5-6 seconds to load the main page, which is no more complicated than the old page, and makes 127 requests (I'm watching Firebug NET) to the server to get the data it needs.
Is this typical?
Thanks.
Yep a 3x performance hit is natural to Drupal, or most of large scale PHP application framework. Bootstraping Drupal is a costly operation as it requires loading a lot of files. Drupal is also known to perform too much DB queries in order to produce a single page.
The first step is to enable page caching and JS/CSS aggregation. This can be done from the administration page at Administration >> Configuration >> Performance (in Drupal 7).
But a 1-2 seconds load time on a lightweight PHP site is a sign of a either overloaded or badly tuned hosting. You should ensure you site is running in a recent PHP version (PHP is getting faster and faster with each version). Also enable APC (or any other opcache), even with the default settings it can greatly improve Drupal's performances. With APC, try increasing the shared memory size (eg. apc.shm_size = 64 in php.ini).
You should also try profiling your site to identify the actual bottle necks. With Drupal making several requests per page, the DB quickly becomes the bottle neck. Drupal support using multiple slave servers for read queries.
About the database, Drupal uses an internal cache which by default is stored in the database. So this cache does not deal well with overloaded database. Drupal's cache is pluggable. It can be configured to use memcache, redis or mongodb for its storage. This could greatly reduce the load on the database.
Yes drupal is slow.
Thats why we use caching mecahnisms if ur page is making too many requests
See if u can aggregate ur CSS and JS(This will reduce number of
HTML calls. u can do this from admin)
Use CDN
use memcache or varnish cache
use page cache in apache.
Note:-please provide some actual data split up with some load testing tools
How much requests are sent to server? it also matters but drupal has solutions for it. Drupal combine all css files in to a single file to make server calls low, and similarly for js files.
But the speed also matters on server side code, database operations. Drupal is a powerful system which makes complex things easy (and yes easy things complex) and provides such capabilities so that a user can make a complete portal without a line of coding. But all these features come by the cost of performance. Internally drupal do lots of operations and it makes it slow.
Those operations includes views and block operations and the more complex the view / block / form is, the more operations there will be, and hence it will take more time.
Also if the site contents are increased then it will be become more slow. Because drupal consider every content as a node, and for all of your content types (for example news, cms pages, testimonials and so on) data is stored in a single node table (some other tables are also used, but your main contents are stored in node table). So when the contents are increased, the load on that single table is increased, which cause slow database operations, because the more big your table size is, the more operation time it will be taking.
I may be wrong, but Drupal is slow :P
I have a blog I made. Recently, I've been noticing some performance problems. I'm getting about 400ms waiting times for the index page. I think this is quite high. When I first deployed it(with less features, but still) I recall index load times of something like 80ms.
Now I would profile it, but the problem is that this only happens in my production environment. In my test environment, the index page only takes 10ms.
How do I go about profiling my production application? I use Apache+mono+mod_mono on Arch Linux with MongoDB. I have a similar test environment except I use xsp.
I'm unsure of where to look: my code, Apache's configuration, or MongoDB? How can I profile my production server to figure out why it's so much slower than my development environment?
Tough to be specific without details, but here's a shot at a general guide:
First I would recommend using something like Firebug for Firefox - there are equivalents in other browsers, but this is my old go-to tool for this kind of thing. Enable it and got the Net Panel view for a waterfall diagram that will show you a list of every object that is loading on a page (you might have to refresh) - it will also have a blue showing the render event (when the page becomes visible).
The waterfall should make it pretty obvious where the slow pieces of the page are and armed with that information you can go to the next stage - figuring out why particular pieces are slow.
If plugins are not your thing, or you suspect that it could be something local to yur machine causing the issue, then take a look at: http://www.webpagetest.org/
That will give you the ability to remote test from different locations, different browsers, speeds etc. and give you similar detailed results.
If it is a static file being fetched, look at network problems, Apache as a cause. If it is dynamically generated then look at Apache, ASP, mongodb etc.
For Apache, what do the access logs say is the response time for the index page? Assuming Apache 2 or newer, make sure you have %D (and %T if you like) being logged so you can see the time taken to serve the page (from the Apache perspective) at the required level of details. For more info on that, take a look at the LogFormat directive.
I can't help on the ASP/Mono side, not my thing, but adding debug statements at various points to track the index page generation (assuming it is dynamically generated) would be a pretty standard approach.
For the database, MongoDB by default logs only "slow" queries that take >100ms - if you are trying to track down a sub-100ms response time issue via the logs you will need to adjust that or you will likely get very little. That can be done as follows:
> db.setProfilingLevel(0,20) // leave profiling off, slow threshold=20ms
You can also adjust it as a startup parameter (--slowms) to the mongod process. More information on profiling, which may also help but has overheads, can be found here:
http://www.mongodb.org/display/DOCS/Database+Profiler
I'd suggest you have a look a Sam Saffrons Mini Profiler. If you use it in your site, it allows you to turn on profiling on production.
By adding sufficient instrumentation to your code, you should then be able to identify which bit is taking the time and then focus your efforts there.
We have a web application where the users from whole world would upload there file at a same time. We want an effecient, robust upload system. Max file size would be 50 MB.
There would be atleast 1lac users uploading at same time.
Please suggest which is the better upload system, FTP or HTTP?
Currently we have http based upload where in we do get some errors like connection problem, session time out, time out error, etc...
Even suggest me for any 3rd party ftp upload tools if you come accross.
I will suggest you yo go with HTTP, because it is much favorable in terms of user convenience.
If you are having critical issues with Large file upload then Please have a look at Darren johnstone's Large File upload library for ASP.Net.
Still If you have to go with FTP then I will suggest you to go with using some Client Side reach technology which runs under browser like FLASH, JAVA Applets ( or might be SilverLight )
Depends on what you're doing.
every user I ever met knew how to use a browser, but the standard random user doesn't even have an FTP client installed. So usually HTTP uploads aren't really problematic. I wouldn't wanna upload huge files, but 50megs isn't that bad yet. If you want an FTP upload you probably would go for a java applet, so your users are guaranteed to have even the software needed to upload their files. Any PHP/WebFTP things will just run you into the same problems again.
Sometimes if I don't know if things I want to do work well, its a good time to look at how others are doing it. Gmail for example has a fabulous upload system. imageshack, millions and millions of users are uploading their stuff their, basically thats all the page does, and all of them use "normal" HTTP, with a little bit of JavaScript sugar to display the progress.
edit: here is an example with PHP: (although u seem to be using asp, it might still help)
http://www.devpro.it/upload_progress/
I have a weird situation with my sharepoint css.
It is deployed as part of a .wsp solution and up until now everything has been fine.
The farm it deploys too has a couple of webfront ends and a single apps server and SQL box.
The symptom is that if I deploy the solution, then use a webbrowser to view the page it has no styles, and if I access the .css directly I see the first 100 or so bytes of the .css.
However if I go into sharepoint designer and look at the file it is looks fine, and if I check it out and publish it (customising the file but not actually changing anything in it) then the website works fine and the css downloads completely.
There is some fairly complex caching on the servers Disk based and object caches. as far as I can tell I have cleared these (and an issreset should clear them anyway... shouldn't it?)
I have used this tool to clear the blobcache from the whole farm http://blobcachefarmflush.codeplex.com/
The problem you're describing is one I've encountered before. Let me share what I know, what I suspect, and how I'd go about troubleshooting your scenario.
First off, it sounds like you suspect caching as a potential problem source. In the case of the MOSS publishing feature set, you really have three different cache mechanisms in operation: the object cache, the BLOB cache, and the page output cache. The only mechanism that should be in-play, assuming it's turned on with default settings, is the BLOB cache. Neither the object cache nor the page output cache should be touching stand-alone stylesheets like you have.
You've tried flushing the cache the flush using the farm-level BLOB cache flush feature, and that will instruct MOSS to dump all BLOB cache data. You can verify this by reviewing the file system to ensure that only the three .bin folders remain following a flush.
To your specific question regarding an IISRESET: no, and IISRESET actually won't clear the BLOB cache. The contents of the BLOB cache persist beyond the life of the application pool servicing the web application. You either need to use a feature to clear out the cache (as you have been), or perform a manual file delete. I don't recommend the latter unless you absolutely have no other course of action. If you do elect to go the manual route to try it, ensure that you shutdown the W3SVC service before deleting files out of the file system. If you don't, the actual file deletion process can get into a race condition with cache repopulation and lead to corruption. After you've deleted files with a stopped W3SVC, you can start the W3SVC back up again.
For more information on the internals of the BLOB cache and how it operates, I'll point you to a blog article of mine: http://sharepointinterface.com/2009/06/18/we-drift-deeper-into-the-sound-as-the-flush-comes/
To see if the BLOB cache is a factor in the behavior you're seeing, you can modify the web.config for your web application(s) and adjust the file pattern to remove CSS from the list of file types in the <BlobCache> element and then restart IIS (or at least recycle the app pool).
Another possibility, based on experience, is that you're seeing something other than BLOB cache abnormalities. The key observation for me comes in the form of you observing that a direct request for the CSS stylesheet returns only the first 100 bytes or so.
Do you, by any chance, have any intelligent network hardware (that is, intrusion detection hardware or anything that might be performing application/layer-7 filtering) between the WFE and you, the caller? Intrusion detection and IPS systems are the source of many of the types of problems you're seeing, and they're one of my first stops whenever I see "oddball" behavior like you're describing. In the case of one of my clients, I saw a problem meeting your description (CSS and JS files getting truncated) due to an intervening Juniper firewall with active IPS. Turning off IPS (to test) cleared things up immediately. After that, the networking team sought an update from Juniper to correct the issue to ensure that IPS could remain active.
Try turning off BLOB caching (or removing the CSS extension from the file pattern) to see if that makes a difference. If not, talk to your network team to see if something is happening to the response stream coming back to you. That's where I'd start; hopefully, one of those two things will do the trick for you.
Small side note: if you have a free moment and are up to it, I'd like to hear about your experience with the BlobCacheFarmFlush solution you pulled down from CodePlex. I authored it, and I'd love to hear your thoughts -- good or bad :-)
Sean (sean#sharepointinterface.com)