My user took 1.6 min to download a 900KB static file from firebase hosting.
May I know is there any way to optimize this?
Thanks for Doug Stevenson on pointing out, the main reason for slow TTFB and content download speed is because Fastly CDN performs poorly in my country (Malaysia). Switched to Cloudflare to resolve the problem.
The image below shows that test on Cloud Harmony, Fastly CDN has a really bad result especially downlink for > 256KB
There really is no way to optimize the time it takes for Firebase Hosting to serve static content, other than taking steps to reduce the amount of data to transfer (maybe compressing it differently), or splitting the data among more than one concurrent request.
It will take as long as it takes to transfer the content the user. The user's internet connection speed makes a huge difference, and if their connection is slow, that situation can't be improved.
Related
I have a dashboard in which I use Firebase Real Time Database. I have also a backend that publishes output to the front-end, which runs in batches. Every time when there is a batch, I notice that the Load peaks to almost 100% (most of it is writing but there is also some considerable loading). This is causing my front-end dashboard to slow down.
My question is what I could do to avoid this issue? Is there a way to scale the Load up, such that its less likely to approach the 100%? What is the Firebase recommended way to handle this?
This type of spiky load is indeed commonly caused by backend processes that run in batches.
The Firebase backend storage layer runs pretty much as a single threaded process for each database, handling each (read or write) request from clients in turn. So while it is processing one request, any other requests are awaiting their turn.
This means that if you have a particularly large read or write request, it keeps other clients from getting their requests served. For this reason you'll want to take care to divide the interactions with the database (especially from the backend process) into small operations as to not interfere with the other clients.
If the backend process also needs to read a considerable part of the database for its work, consider if you can make it read from a backup of the database instead of the from the live database. The backups are made out-of-band, so don't interfere with clients, so if you can use the backup as the source for reading the data, it will significantly reduce the read load that the backend process causes.
Site that have nice but raising number of visits / hits, heavily operates on few database has 0.6 sec / req average serving time, most of that time is spend (about 30% / 0.2 sec/avg) on "first byte" so working on solving that problem.
Another way to improve speed that I considering is using CDN. I am not very sure can CDN in this case help:
-it all dynamic content
-have a lot of images
-already have internal caching (front end), (and caching on client side if that matters)
So, please tell me what you would suggest, should I try with CDN and how much improvement I can expect? (traffic is mostly US)
As #matthias says accept some of your other answers...
Whether a CDN will make much difference depends on how far they are from the server and what the latencies are like.
A CDN will give you more connections but you can do this by adding extra hostnames.
There are probably other ways of improving page load times, run a test on webpagetest.org and post the results.
I read this article Optimize Parallel Downloads to Minimize Object Overhead and write a test demo
But the result is not I expected，look at the waterfall figure multiple domain's images indeed parallel download but the total time no less.
Who can tell me why,thanks
multiple domain image download
single domain image download
What I think the problem with your test is that the network latency is so low (assuming its a local server), that your network performance isn't even playing a huge role here. If you look at the time difference between those images it doesn't' even register in HTTPWatch. So the browser may be spending more time parsing, processing and rendering the downloads than it does actually downloading (just a guess).
I would hit this test site with something that will show off latency more. If these sites are reachable on the internet, you can just use http://www.webpagetest.org/ and hit them from around the world...
~Sniff
P.S. Check out Amdahl's law, it may have a little bearing here here ;-).
I have an asp.net web site with 10-25k visitors a day (peaks of over 60k before holidays). Pages/visit is also high, since it's a content site.
I have a few specific pages which generate about 60% of the traffic. These pages are a bit complex and are DB heavy (sql server 2008 r2 backend).
I was wondering if it's worth "caching" a static version of these pages (I hear this is possible) and only re-render them when something changes (about once in 48hs).
Does this sound like a good idea? Where would be the best place to implement this?
(asp.net, iis, db)
Update: Looks like a good option for me is outputcache with SqlDependency. I see a reference to some kind of SQL server notification for invalidating the cache, but I only see talk of SQL server 2005. Has this option been deprecated by Microsoft? Any new way to handle this?
Caching is a broad term that can happen at a number of different points. The optimum solution may be a combination of some or all.
For example, you can add page, or output caching as described here, which caches output on the web server, which I think is what you were referring to.
In addition, you can cache the data in memory using something like memcached, so that your data is more available to the web server as it builds the page, but you need to look at cache hit rate to know for sure that you are caching the right data.
Also, although slightly off the topic of improving db heavy pages, you can cache static resources that change infrequently like images, css and include files using a content delivery network. Any CDN will almost certainly have a higher bandwidth and a cheaper data plan than your own connection because of the economies of scale, so the more of your content you can serve from there the better, in general.
Your first question was "I was wondering if it's worth "caching" a static version of these pages". I guess the answer to that depends on whether there is a performance problem at the moment, and where the cause of that problem is. If the pages are being served quickly and reliably, then quite possibly it's not worth implementing caching. If there is a performance problem, then where is it? Is it in db read time, or is it in the time spent building the page once the data has been returned?
I don't have much experience in caching, but this is what I would try to do:
I would look at your stats and run some profiles, see which are the most heavily visited pages that run the most expensive SQL queries. Pick one or two of the most expensive pages.
If the page is pseudo static, that is, no data on it such as your logged in username, no comments, etc etc, you can cache the entire page. You can set a relatively long cache as well, anything from 1 min to a few hours.
If the page has some dynamic real time content on it, such as comments, you can identify the static controls and cache those individually. Don't put a page wide cache on.
Good luck, sounds like a cache could improve performance.
Caching may or may not help. For example, if a site has low traffic and if the caching is enabled, the server processes to create the cache before serving the request. And because the traffic is low, there can be enough delay between successive requests. So the cached version may even expire and the server again creates a new cached version. This process makes the response even slower than normal.
Read more: Caching - the good, the bad.
I have myself experienced this issue.
If the traffic is good, caching may help you have better load times.
Cheers
Aditya
I have a DB with user accounts information.
I've scheduled a CRON job which updates the DB with every new user data it fetches from their accounts.
I was thinking that this may cause a problem since all requests are coming from the same IP address and the server may block requests from that IP address.
Is this the case?
If so, how do I avoid being banned? should I be using a proxy?
Thanks
You get banned for suspicious (or malicious) activity.
If you are running a normal business application inside a normal company intranet you are unlikely to get banned.
Since you have access to user accounts information, you already have a lot of access to the system. The best thing to do is to ask your systems administrator, since he/she defines what constitutes suspicious/malicious activity. The systems administrator might also want to help you ensure that your database is at least as secure as the original information.
should I be using a proxy?
A proxy might disguise what you are doing - but you are still doing it. So this isn't the most ethical way of solving the problem.
Is the cron job that fetches data from this "database" on the same server? Are you fetching data for a user from a remote server using screen scraping or something?
If this is the case, you may want to set up a few different cron jobs and do it in batches. That way you reduce the amount of load on the remote server and lower the chance of wherever you are getting this data from, blocking your access.
Edit
Okay, so if you have not got permission to do scraping, obviously you are going to want to do it responsibly (no matter the site). Try gather as much data as you can from as little requests as possible, and spread them out over the course of the whole day, or even during times that a likely to be low load. I wouldn't try and use a proxy, that wouldn't really help the remote server, but it would be a pain in the ass to you.
I'm no iPhone programmer, and this might not be possible, but you could try have the individual iPhones grab the data so all the source traffic isn't from the same IP. Just an idea, otherwise just try to be a bit discrete.
Here are some tips from Jeff regarding the scraping of Stack Overflow, but I'd imagine that the rules are similar for any site.
Use GZIP requests. This is important! For example, one scraper used 120 megabytes of bandwidth in only 3,310 hits which is substantial. With basic gzip support (baked into HTTP since the 90s, and universally supported) it would have been 20 megabytes or less.
Identify yourself. Add something useful to the user-agent (ideally, a link to an URL, or something informational) so we can see your bot as something other than "generic unknown anonymous scraper."
Use the right formats. Don't scrape HTML when there is a JSON or RSS feed you could use instead. Heck, why scrape at all when you can download our cc-wiki data dump??
Be considerate. Pulling data more than every 15 minutes is questionable. If you need something more timely than that ... why not ask permission first, and make your case as to why this is a benefit to the SO community and should be allowed? Our email is linked at the bottom of every single page on every SO family site. We don't bite... hard.
Yes, you want an API. We get it. Don't rage against the machine by doing naughty things until we build it. It's in the queue.