I am having difficulty in a running python script that calls module "requests" in crontab. This was fine a few days ago and then I had to change my authentication for Google (to send emails), then "requests" stopped working in crontab. The python script runs fine in a terminal but will not execute in crontab. "requests" is available and when I type "pip3 show requests" the following is displayed (note I replaced my user name with "user"):
$pip3 show requests
Name: requests
Version: 2.27.1
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me#kennethreitz.org
License: Apache 2.0
Location: /home/user/.local/lib/python3.6/site-packages
Requires: certifi, idna, urllib3, charset-normalizer
A simplified version of the python file I would like to execute in crontab is:
#!/usr/bin/env python...
# -*- coding: utf-8 -*-
import requests
print ('End of code')
The file test_request.py executes fine in a terminal.
I created a bash script called test_request.sh based on directions from this stack overflow page:
"ImportError: No module named requests" in crontab Python script
That bash script is this:
#!/bin/bash
echo test_request.sh called: `date`
HOME=/home/user/
PYTHONPATH=/home/user/.local/lib/python3.6/site-packages
cd /home/user/Documents/bjg_code/
python ./test_request.py 2>&1 1>/dev/null
When I try to run the bash script in a terminal or in crontab I receive this error:
$bash test_request.sh
test_request.sh called: Sat Jun 11 14:18:46 EDT 2022
Traceback (most recent call last):
File "./test_request.py", line 4, in <module>
import requests
ImportError: No module named requests
Any advice would be welcomed and appreciated.
Thank you in advance.
Related
My project structure looks like this:
/definitions (for all dagster python definitions)
__init__.py
repositories.py
/exchangerates
pipelines.py
...
...
workspace.yaml
I've tried running the grpc server using various methods, especially the following (started in project root):
dagster api grpc -h 0.0.0.0 -p 4000 -f definitions/repositories.py
dagster api grpc -h 0.0.0.0 -p 4000 -m definitions
dagster api grpc -h 0.0.0.0 -p 4000 -m definitions.repositories
The first command yields the following error:
dagster.core.errors.DagsterImportError: Encountered ImportError: attempted relative import with no known parent package while importing module repositories from file C:\Users\Klaus\PycharmProjects\dagsterexchangerates\definitions\repositories.py. Consider using the module-based options -m for CLI-based targets or the python_package workspace.yaml target.
The second and third command yield the following error:
(stacktrace comes before this)
ModuleNotFoundError: No module named 'definitions'
How can this be solved?
EDIT:
I've uploaded the current version of the example I'm working on to GitHub: https://github.com/kstadler/dagster-exchangerates
EDIT2:
Reflected changes in directory structure
sorry about the trouble - there are a couple of options here to get your server running.
To get it working with the '-f' option, the relative imports need to be replaced with absolute imports. That would look like this:
-from .pipelines import exchangerates_pipline
-from .partitions import year_partition_set
+from definitions.pipelines import exchangerates_pipline
+from definitions.partitions import year_partition_set
(This is the same error you would get if you tried to run python definitions/repositories.py directly).
I'm still digging into why exactly the third '-m' option isn't working the way I'd expect it to. Curiously the following command seems to work for me which should be close to identical:
python -m dagster.grpc -h 0.0.0.0 -p 4000 -m definitions.repositories
Incidentally, your example contains a workspace.yaml that should cause dagit and other Dagster processes to automatically start up the gRPC server for you for that module - so depending on your goal you may not need to run dagster api grpc yourself at all.
When I try to run my sh script with a spider, it displays only one warning and does not parse, but when I run the spider on my own, the parsing goes fine
my sh file
#!/bin/bash
# shellcheck disable=SC2164
cd /var/www/scrapy_parser/avito/avito/spiders
scrapy crawl avito -L WARNING
cd /var/www/scrapy_parser/info/info/spiders
scrapy crawl info_v1 -L WARNING.
sh output:
WARNING: /usr/local/lib/python3.6/site-packages/scrapy/extensions/feedexport.py:210:
ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in
favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details
exporter = cls(crawler)
What can I do to fix this?
In general, I solved this problem by logging, in the end it turned out that the site that I parse blocked me as a bot and the sh script did not read the pre-installed proxies in the system, I solved everything by adding the proxy setting directly to the sh script before starting the spider
I am trying to run an .exe that sits on a remote machine with PSEXEC, but it does not run properly.
The exe is a converted python script:
usage: WindowsUpdates3.exe [-h] [-v] [-m] [-i] [-u] [-r] [-s SKIP]
List, download and update Windows clients
optional arguments:
-h, --help show this help message and exit
-v, --version Show program version and exit
-m, --list-missing List missing updates
-i, --list-installed List installed updates
-u, --update List and install missing updates
-r, --reboot Reboot after installing updates if needed
-s SKIP, --skip SKIP Skips these KB numbers
I am able to run the .exe as intended with a PSEXEC console session:
PSEXEC \\<hostname> cmd
Navigate to the exe my.exe -i, run it and it works the same way if I execute it on the machine locally.
When I try to execute the file directly some fucntionality does not work
PSEXEC.exe \\<hostname> -h "C:\WindowsUpdates\WindowsUpdates3.exe" -i
...
C:\WindowsUpdates\WindowsUpdates3.exe exited on <hostname> with error code 0.
I am able to get the help-menu (-h) and the version (-v) of the exe with the above command.
The other arguments don't return anything but code 0 and the --update argument throws a com_error: -2147024891, which translates to access denied...
How can this be as I have the same privileges as if I spawn a cmd terminal?
OK so I have paramiko v2.2.1 and I am trying to login to a machine and restart a service. Inside the service scripts it basically starts a process via nohup. However if I allow paramiko to disconnect as soon as it is done the process started terminates with a PIPE signal when it writes to stdout.
If I start the service by ssh'ing into the box and manually starting it there is no issue and it runs in the background fine. Also if I add long sleep 10 before disconnecting (close) paramiko it also seems to work just fine.
The service is started via a init.d script via a line like this:
env LD_LIBRARY_PATH=$bin_path nohup $bin_path/ServerLoop.sh \
"$bin_path/Service service args" "$#" &
Where ServerLoop.sh simply calls the service forever in a loop like this so it will never die:
SERVER=$1
shift
ARGS=$#
logger $ARGS
while [ 1 ]; do
$SERVER $ARGS
logger "$SERVER terminated with exit code: $STATUS. Server has been restarted"
sleep 1
done
I have noticed when I start the service by ssh'ing into the box I get a nohup.out file written to the root. However when I run through paramiko I get no nohup.out written anywhere on the system ... ie this after I manually ssh into the box and start the service:
root#ts4700:/mnt/mc.fw/bin# find / -name "nohup*"
/usr/bin/nohup
/usr/share/man/man1/nohup.1.gz
/nohup.out
And this is after I run through paramiko:
root#ts4700:/mnt/mc.fw/bin# find / -name "nohup*"
/usr/bin/nohup
/usr/share/man/man1/nohup.1.gz
As I understand it nohup will only redirect the output to nohup.out if "If standard output is a terminal" (from the manual), otherwise it thinks it is saving the output to a file so it does not redirect. Hence I tried the following:
In [43]: import paramiko
In [44]: paramiko.__version__
Out[44]: '2.2.1'
In [45]: ssh = paramiko.SSHClient()
In [46]: ssh.set_missing_host_key_policy(AutoAddPolicy())
In [47]: ssh.connect(ip, username='root', password=not_for_so_sorry, look_for_keys=False, allow_agent=False)
In [48]: stdin, stdout, stderr = ssh.exec_command("tty")
In [49]: stdout.read()
Out[49]: 'not a tty\n'
So I am thinking that nohup is not redirecting to nohup.out when I run it through paramiko because tty is not returning a terminal. I don't know why adding a sleep(10) would fix this though as the service if run on the command line is quite verbose.
I have also noticed that if the service is started from a manual ssh its tty in the ps ax output is still set to the ssh tty ... however if the process is started by paramiko its tty in the ps ax output is set to "?" .. since both processes are run through nohup I would have expected this to be the same.
If the problem is that nohup is indeed not redirecting the output to nohup.out because of the tty is there a way to force this to happen or a better way to run this sort of command via paramiko?
Thanks all, any help with this would be great :)
I've been struggling with this one for a while and I'm thoroughly baffled.
I have this postinst Debian script that is supposed to start a service once installation (of the service executable) is complete. As best I can tell, the service does start successfully, but it then immediately and mysteriously quits. Restarting the service from the command-line works fine once Synaptic concludes.
I tried writing a dummy package to verify this. The dummy package installs /etc/init/service-dummy.conf and a symbolic link to that file, named /etc/init.d/service-dummy (just like the original service). The contents of service-dummy.conf are the same as service.conf. The dummy starts the service...and then the service keeps on running. So I can't even reproduce my problem!
The postinst script does this:
#!/bin/sh
set -e
case "$1" in
configure)
# (instructions which config, make and install the freshly installed source code)
ldconfig
echo "Install concluded"
if [ -e "/etc/init/service-dummy.conf" ]; then
echo "Starting service-dummy root service" | tee service.log
service service-dummy restart | tee --append service.log
else
echo "service-dummy.conf not installed"
fi
echo "Postinst complete"
;;
*)
echo "postinst called with unknown argument '$1'" >&2
;;
esac
# exit 1 to ensure installer stalls
exit 1
Synaptic displays the log:
...
Starting service-dummy root service
stop: Unknown instance:
service-dummy start/running, process 9207
Postinst complete
dpkg: error processing service-dummy (--configure):
subprocess installed post-installation script returned error exit status 1
...
It's as if upstart needed to be refreshed?
I tried more things and then I did get it to work, sort of: I try starting the service, then abort the script with an exit 1, and when the script runs a second time (postinst, same parameters, so I detect the second run otherwise), I start the service again, and this time it sticks.
A key clue is in the log:
Postinst complete (aborting script)
dpkg: error processing service-dummy (--configure):
subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Errors were encountered while processing:
service-dummy
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up service-dummy ...
service-dummy postinst configure
Starting service-dummy a second time
stop: Unknown instance:
service-dummy start/running, process 4034
Postinst complete (aborting script recovery attempt)
So I guess my question now becomes:
How do I force ldconfig to not defer its processing?
Found the right clue here: http://lists.debian.org/debian-glibc/2008/07/msg00169.html
Turns out apt-get temporarily prevents use of ldconfig by replacing it with something else. The solution to my problem is to very simply call ldconfig.real instead of ldconfig in the script.