Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I have a project in which I have implemented hundreds of scrapy spiders.
Now I collided with the following problems:
time to time websites changes their DOM/API - so spider stop working or not all info collected
websites became unavailable or moved to another domain - spider stop working.
Since there are a lot of spiders it not easy to monitor the status of each spider.
is there any framework that provides the ability to monitor scrapy spiders?
status of running spider(s)
show when spider(s) stop working etc.
I have looked into scrapinghub/zyte but not sure if it good for our purpose because we need something that can be running locally.
Scrapydweb is basically just like Scrapinghub but you run it locally.
It can alert you when spiders fail with email/slack if I remember correct. It's a bit less user-friendly than scrapinghub since you have to manage servers and so on. But overall I think it was a good platform when I used it.
Airflow is a very handy platform for scheduling and monitoring.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I have an ASP.NET application which is configured to run on my local IIS.
Anyone knows any Free Record/Replay Web Testing Tool so that I can test the pages with e.g. 10 concurrent users?
(I don't have access to Visual Studio Test/Ultimate edition to use its Web Testing tools.)
Thanks
You may checkout the soapUI/loadUI suite.
check Selenium tool http://seleniumhq.org/
and seleniumgrid http://seleniumgrid.thoughtworks.com/
Selenium Grid allows you to run Selenium tests in parallel, cutting
down the time required for running acceptance tests to a fraction of
the total time it currently takes.
Automate web application UI testing with Selenium
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I am looking for a third-party service or tool that can trigger or hit a web page at scheduled times. Does anyone know of any? I currently use my windows scheduler to hit these pages, but there are gaps in this since I don't run my computer 24 / 7.
There are lots of free website monitoring services out there that check your site's availability by testing whether a given URL responds properly. One that I use is http://mon.itor.us/ Give it the URL of your page and that will do the trick.
One other alternative is Pingdom. They offer a free account for monitoring one web site. You can set Pingdom to send email or SMS if your site goes down and you can configure the service to hit your page for example once in every 5 minutes. You can set the check resolution quite freely.
Some other alternatives are MonitorUs (which RichieHindle already mentioned), SiteUptime and HyperSpin.
I created an open source app called Pinger. You can do unlimited URLs with intervals of your choosing. The docs has instructions for getting running on Heroku quickly:
https://github.com/austinthecoder/pinger
Only problem is, currently, you can only set an interval, not a specific time.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Is there any free web based cron that can do scheduled jobs ....
http://www.mywebcron.com/
but why would you want a web based cron over a standard crontab?
You can do (for free, unless your load's very heavy) periodic cron-like "sending of mails" with Google's App Engine, with Java or Python -- but I don't know of a free service that will let you do that with asp.net (or other .NET code).
Here's an couple of old articles on how to implement a cron job with ASP.NET: Simulate a Windows Service using ASP.NET to run scheduled jobs and Scheduled ASP.NET Task (ASP.NET Cron Jobs).
You basically just add something to the cache, and when the item expire you can use that event to trigger your code. Simple.
You can also try www.thecloudblocks.com which includes a web UI and RESTful API
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
What are tools you would recommend for analyzing VPS performance? Specifically I run several Wordpress blogs on a VPS and would like to find some tools to help me find issues worth looking into (so I can address them).
I use New Relic for getting some data on where to focus some attention on our Rails applications and find it very useful.
I have a feeling there might be issues with memory use. I would love to see something that showed memory use over time so I could see what my memory use looks like. I am considering upgrading the VPS and this would be one useful piece of data.
"top" is always a good place to start in the terminal.
but http://scoutapp.com/ is a really good monitoring app, and easy to implement on vps servers
Try Munin, is a great monitoring app with a web interface, http://munin-monitoring.org/
cloudstats.me
works free for three VPS
New Relic has a free server monitoring service that will show you CPU, memory, network I/O, disk I/O, and disk capacity, as well as a process list a la top.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
What do you use to monitor the uptime / performance of your websites, specifically those based on a PHP/MySQL platform like Wordpress?
I'm looking for something that alerts me if the site is down, or performing too slowly, and has some useful (not volumeous!) charts showing me any potential problems, and what to do about them.
Thanks!
We along with the usual Nagios, we use Pingdom. It comes with lots of default checks.
For example it also reports how fast your website is or since they employ tests from different locations you get a nice graph how accessible your website was. To put some sense into it, add a reference check (e.g. google) and see how you perform.
Aside from HTTP etc. you can also check other services (mail, database, etc.). If they are not reachable from the outside, you can always create a script that outputs a standard "OK" and have Pingdom check on that, and report back if the output changed.
I should add that Pingdom is not a free service. But we've been using them for 10 months now and they haven't troubled us. :)
Try looking at Zabbix
http://www.zabbix.com/
WEB performance monitoring
WEB availability monitoring
Support of POST and GET methods
Try out Insping also
performance monitoring
availability monitoring
e-mail and SMS alerts