Start Scrapyd as service - web-scraping

I would like to start scrapyd as service but when I start scrapyd,
if I close the SSH session the service scrapyd close automatically.
When I try to start as service like this I have an error :
root#vps:~# service scrapyd start
scrapyd: Failed to start scrapyd.service: Unit scrapyd.service failed to load: No such file or directory.
And when i try to start daemon scrapyd the CURL request return :
{"status": "error", "message": "Use \"scrapy\" to see available commands", "node_name": "vps"}
Can someone help me to start my scrapyd as service please !

daemon --chdir=/home/Crawler scrapyd
I need to set the --chdir to load the service on the Scrapy folder !

I could not get my scrapyd start through /etc/rc.local or crontab so I found a work around. I am sure there will be a better way but for mean time this worked for me.
I created a python file start.py.
import os
os.system('/usr/bin/python3 /home/ubuntu/.local/bin/scrapyd > /home/ubuntu/scrapy.log &')
And simply call the python file start.py through crontab.
Add this to crintab file by cronat -e.
#reboot /usr/bin/python3 /home/ubuntu/start.py
Basically what I did is execute the scrapy through shell by calling the python filr through crontab.
If anyone finds something better please comment.

Related

Linux runlevels and chkconfig

I know this an old question but I am having a tough time in understanding the linux runlevels w.r.t gracefully stopping and starting a service. For suppose consider myscript service is configured to auto-start after a system shutdown/reboot.
Below is the output of chkconfig command (RHEL 6.5x):
[testfolk#jomohost ~]$ sudo chkconfig --list|grep myscript
myscript 0:off 1:off 2:on 3:on 4:off 5:on 6:off
I understand that myscript service auto-starts on system startup. But what takes care of it's gracefull stop/shutdown task. What if myscript is a delicate service and need to be stopped at any cost before any reboot/shutdown or the application gets corrupted. Is taken care of by default ?
Can I configure it something like below to do a graceful auto stop:
[testfolk#jomohost ~]$ sudo chkconfig --list|grep myscript
myscript 0:on 1:off 2:on 3:on 4:off 5:on 6:on
My assumption of above configuration is the myscript service gets auto stopped on invoking init 6 or init 0. Is this a valid config, I can be completely wrong, please advise. I searched as many articles as possible but all of them were talking about auto-start of a service after a reboot or shutdown but not auto-stop. I am not sure if I am understanding this whole thing correctly, pls help if you have any info!
Thanks in advance for your time!

How can I stop the http server, downloaded using 'npm install http-server"?

How can I stop the http server, downloaded using 'npm install http-server" comand in terminal (console) and launched then?
Simply Ctrl+C, if you read the output after you launch it, you should see:
Starting up http-server, serving xxx
Available on:
http://<some ip>:<some port>
Hit CTRL-C to stop the server
Its built on node so Kill the node process for stopping it if it stuck. You can find all the node process ids and see what I'd your server have and kill that.

exec function blocks other requests during the execution

I can't get http response when a exec or system function is excuting.
A specific request launch the exec function sudo find / -name toto, this take a while
During the execution, the client send any other requests
I don't get responses
Did you have a similar problem ? I tried shell_exec, exec, system...
I tried also this case :
A request launch a infinite loop
A second request launch an other script who return 'toto'
It work fine. What's wrong with exec ?
Thank you for your ideas

Boot script execution order (rc.local)?

With some great help from another user on here I've managed to create a script which writes the necessary network configurations required to /etc/network/interfaces and allow public access to a DomU server.
I’ve placed this script in the /etc/rc.local file, and executed chmod u+x /etc/rc.local to enable it.
The server is a DomU Ubuntu server on the a host (Dom0). And rc.local doesn't seem to be executing before the network is brought up at boot/creation time.
So the configuration changes are being made to the /etc/network/interfaces file, but are not active once the boot process completes. I have to reboot once more before the changes take effect.
I've tried adding /etc/init.d/networking restart to the the end of the rc.local script (before exit 0), but with no joy.
I also tried adding the script to the S35networking file, but again without success.
Any advice or suggestions on getting this script to execute before the network device is brought up would be greatly appreciated.?

Monit fails to execute the complete command in start program

I'm using monit 5.4 on Mac 10.7.4 machine. When i tried to execute a example configuration
check process syslogd with pidfile /var/run/syslogd.pid
start program = "/etc/init.d/sysklogd start"
stop program = "/etc/init.d/sysklogd stop"
if 5 restarts within 5 cycles then timeout
from monit wiki page, I get the following error.
'syslogd' process is not running
'syslogd' trying to restart
'syslogd' start: /etc/init.d/sysklogd
'syslogd' failed to start
Monit does not take the complete command given in the "start program" of the monitrc file. It just takes the first word in the command and tries to execute it and fails. Is this a known issue? If yes, does it have a workaround? If not, what am i missing here and how to get it working?
Thanks in advance.
Try this (from http://mmonit.com/wiki/Monit/FAQ#execution)
start program = "/bin/bash -c '/etc/init.d/blah start'"
Does /etc/init.d/sysklogd actually exist?
On 10.8 I have /etc/init.d/syslog and manually running /etc/init.d/syslog restart works fine.

Resources