My website goes 503 Service Unavailable error several times a day. Sometimes it gives Request Time Limit for specific requests.
My website isn't receiving huge traffic for it to go busy.
The trace error logs shows a lot of error requests sometimes 503 or 500. It wasn't like it a few months ago. No changes were made in the code/settings/design of the website.
I found some logs that trace to +Ahrefsbot and Googlebot web crawler so I suspect it's crawling my website too fast. I can't block them as they come in different ip addresses.
Please help
My website is hosted under networksolutions so I can't perform any modifications or analyze what's going behind my website (server-side)
If you're sure it's crawlers that're causing your problems you could try using a /robots.txt file.
You can learn more about it here, but to block them completely use this:
User-agent: *
Disallow: /
You can adjust Google's crawl rate through Google Webmaster tools.
There's an existing answer here that should be of help when dealing with Google's bots.
That said, I honestly doubt it's robots causing spikes in traffic large enough to bump you offline. Try analyzing your logs to see what the actual cause is. Maybe you're using more memory than your host allows.
Related
What status code should I set when updating and reloading the website for search engines to understand?
Currently, I use the code 503, but Google registers it as a site error!
503 is the most appropriate HTTP response code for the scenario you describe. There are no 4xx or 3xx errors that indicate "try again later".
But lets step back:
It shouldn't be a problem if Google occasionally fails to crawl your website. If your business model is significantly impacted by this, I would suggest that there is something a bit wrong with your model. (And besides, that is a matter for discussion between you and Google!)
It would be more a problem if your website's (real) users see 503's.
There is an obvious solution for you. Reimplement your mechanisms for reloading your website so that you don't need to send a 5xx response while reloading.
For example, implement a pair of sites and flip between them using a load balancer or similar. That way you can be updating one site while the other site is serving requests. When the new site is ready switch it into production in the loadbalancer.
There are probably other ways to do this ... depending on how your site works.
Google Pagespeed: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fsuper-zava.co.il%2F
My URL: https://super-zava.co.il/
I talked to my host support (BlueHost) and they told me that the problem is not related to their server. I didn't touch anything.
It seems like it is related to your ISP.
The problem seems to be related with the time that Google's services is taking to fetch the page, so it's either Google's fault or your ISP's, as mentioned above. Google's saying the first request took ~ 5 seconds for them (with the first byte taking around ~0,35 to be received).
The page loads just fine here, perhaps they have changed Google's IP priority.
I reckon it's not because of you or your site.
From me, the latency to your server is around 158 ms, and is taking around 1 second to load the page.
What you can do is to put your site behind a WAF like Cloudflare.
as reported past week on pagespeed insights discuss google groups:
https://groups.google.com/forum/#!topic/pagespeed-insights-discuss/luQUtDOnoik
it seems that this is a problem (or a change in programming api) of PageSpeed Insights web itself
We are uploading a new version of our website.
For various reasons, some pages that exists on the older version still aren't ready for the new version and we need to temporarily take them off.
Which HTTP status should we return for these pages considered they will be up and running again within several days.
Is using ServiceUnavailable = 503 only for these pages the right way or will it have negative impact on the entire website?
(Using ASP.NET in case it's related in some way...)
The status code 503 seems to be the best choice here:
The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.
It shouldn’t be relevant that it’s not due to overload or maintenance in your case. What is relevant is that it’s your fault (hence a status code from the 5xx class), and that it’s temporary (hence 503), so there’s no need to let them know the real reason.
While 503 is typically used for the whole site, I can see no reason why it shouldn’t be used for specific pages only. A possible drawback: If a bot successively crawls a few documents that give 503, it might think that the whole site is affected and stop crawling for now.
If you know when the page will be available again, you can send the Retry-After header:
When sent with a 503 (Service Unavailable) response, Retry-After indicates how long the service is expected to be unavailable to the client.
(FWIW, the Googlebot seems to support this.)
In the post Website outages and blackouts the right way (by a Google employee at the time), my assumptions are confirmed as far as Google Search is concerned: 503 should also be used for specific pages only; crawling rate might be affected if Googlebot gets many 503 answers.
The Situation
I have come across some very suspicious PUT and GET requests in my IIS server logs. After Googling the requesters address, I have found information linking the IP's to known hacking teams. After each PUT there is an immediate GET for the same resource that was attempted to be uploaded to my server.
Question 1:
Would this be considered a remote code execution attack?
Additional Testing Completed By Me:
The IIS logs show that the response given for the PUT request was 412 'Invalid file type all files are not uploaded'
I have turned on Failed Request Tracing and attempted to upload text files using CURL and this is the same response I am provided with and was not able to upload a file.
Question 2:
What can I do to help prevent these type of attacks from being successful?
I can turn on IIS request filtering, but I am concerned that if I deny PUT my IIS application may be negatively impacted for any future web services.
Question 1: Would this be considered a remote code execution attack?
It is impossible to determine the intentions of the attacker from the information given. They could be looking to gain code execution, or they may simply settle for uploading their own content to your server for you to host, or to try and deface your site with their content.
Question 2: What can I do to help prevent these type of attacks from being successful?
Server configuration and patching. The best advice I could give you is to reduce the attack surface - only enable the features you need. If you're not using PUT in your application, then disable it. Only reenable it if needed. Make sure you have the latest updates for your OS installed.
Security is a wide subject. You need everything from secure code when developing applications to rigorous security testing after.
I have been trying to find ways to speed up the load time of my site, so I have turned to CloudFlare to see if I can improve my load time.
My site is thelocalgolfer.com and I host it with hostmonster. I took three consecutive gtMetrix tests w/o cloudflare enabled and then enabled cloudflare ran three
consecutive gtMetrix tests w/ cloudflare enabled. You will see that with
cloudflare enabled it takes on average 21 seconds of wait time on the initial load. I have spent hours on the phone with hostmonster tech support trying to troubleshoot the problem and they said they have exhausted all options on their side.
Also to note when cloudflare is enabled one of the errors I have been
getting is
Error : cURL error 6: Resolving host timed out: www.thelocalgolfer.com
in the middle of the page after the page loads. The page still takes about 21 seconds
Try it yourself I still have it enabled (for now).
Here are the gtmetrix results with CloudFlare enabled:
http://gtmetrix.com/reports/www.thelocalgolfer.com/C3Yv7xNW
http://gtmetrix.com/reports/www.thelocalgolfer.com/Y35wcjzO
http://gtmetrix.com/reports/www.thelocalgolfer.com/x82tUdhH
Without Cloudflare enabled:
http://gtmetrix.com/reports/www.thelocalgolfer.com/NevlWuVV
http://gtmetrix.com/reports/www.thelocalgolfer.com/GDiEPUnG
http://gtmetrix.com/reports/www.thelocalgolfer.com/CcuvxYdq
By the way I have gone back and forth with CloudFlare and they have been less than helpful, they tell me to tweak this option or that option, and they take 24-48 hours to respond.
I am hoping that someone has experience with this issue and can help me out!
Thanks,
Neil
I actually posted some information in your support ticket relative to the issue. Doing a curl against the www record, with no CloudFlare in the middle, is returning a very large response time.
So for anyone that runs in to a similar problem, I was running simplepie on my site which was causing a loopback situation where the page was calling an RSS feed on the same domain.