I'm developing a cluster using mpi4py and when i run a basic script to test the cluster the result is weird:
the result should be this (script executed in c):
My code:
from mpi4py import MPI
world_comm = MPI.COMM_WORLD
world_size = world_comm.Get_size()
name = MPI.Get_processor_name()
my_rank = world_comm.Get_rank()
print('World Size: {0} | Name: {1} | Rank: {2}'.format(world_size,name,my_rank))
And the hostfile:
no02 slots=3
no01 slots=5
How could I get around this problem?
I managed to solve it as follows: I reinstalled mpi4py, but this time I passed the MPICC env before installation:
env MPICC=/usr/bin/mpicc pip3 install mpi4py --no-cache-dir
But you need to make sure the libopenmpi-dev library is installed and then run the which mpicc command and put it as a path in the env in the above command
Related
I am trying to install and configure Globus Connect Personal for Linux (i have a CentOS 8), following this tutorial. However, when I try to set up Globus connect personal by running ./globusconnectpersonal -start i get this error
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'gc.py'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = ''
sys.base_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
sys.base_exec_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
sys.executable = ''
sys.prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_
Subprocess pid 1722896 exited, rc=1
Traceback (most recent call last):
File "./gc-ctrl.py", line 369, in <module>
start(debug=False)
File "./gc-ctrl.py", line 191, in start
send2clients(fds[2:], mesg.encode('utf-8'))
AttributeError: 'bytes' object has no attribute 'encode'
does anybody know what this could mean?
I think there needs to be PYTHONHOME and PYTHONPATH. I created a conda environment with just the correct version of python in it. Then I ran ./globusconnectpersonal inside the conda environment.
Using a conda environment also works for the non-GUI form of globus.
I have not tried setting the paths manually.
I ran into the same problem when I was using Python3.8 in a Miniconda environment. When I disabled conda with:
conda deactivate
Then I could run "globusconnectpersonal -start" with my native Python2.7. I don't know if it was because the client needed Python2 or if conda was interfering, but his resolved the problem for me.
I am unable to setup & run a simple darkflow program. Infact can't even configure darkflow library:
from darkflow.net.build import TFNet
==> ModuleNotFoundError: No module named 'darkflow'
My Target is to run the following program:
from darkflow.net.build import TFNet
import cv2
options = {"model": "cfg/yolo.cfg", "load": "bin/yolo.weights", "threshold": 0.1}
tfnet = TFNet(options)
imgcv = cv2.imread("./test/dog.jpg")
result = tfnet.return_predict(imgcv)
print(result
Please suggest steps so that I could configure darkflow on Jupyter Notebook (with no GPU) and run the above code
Fixed by creating the file in ipynb file in darkflow directory (downloaded from github) and executing the following from the notebook:
!python3 setup.py build_ext --inplace
!pip install -e .
!pip install .
I have installed both requests and requests_ntlm modules using "sudo python3 -m pip install requests" (and requests_ntlm respectively) and both installs were successful.
When I then attempt to do "from requests import HttpNtlmAuth", I get an error stating "cannot import name 'HttpNtlmAuth'. I do not get this error on my "import requests" line.
When I do a "sudo python3 -m pip list", I see both are installed and are the latest versions.
I've not encountered this error before, only "cannot import module", so I'm unfamiliar with how to resolve this.
EDIT 1: Additional information. When I run this script from command line as "sudo", it works. Because I am running my python script from within a PHP file using "exec", I don't particularly want to run this as a root user. Is there a way around this, or possibly running the exec statement with sudo?
the HttpNtlmAuth class is in the requests_ntlm package so you'll need to have:
import requests
from requests_ntlm import HttpNtlmAuth
Then you'll be able to instantiate your authentication
session = requests.Session()
session.auth = HttpNtlmAuth('domain\\username','password')
session.get(url)
Firstly, I'm a newbie R, AWS and python guy. So I'm trying to get a python script with embedded R code running in AWS Lambda using rpy2. I created a Lambda package on an EC2 instance following the instructions here (modified for using python 3.4). It seems that there is something funky happening with loading the R libs using ctypes, as per the following error received in the console:
OSError: lib/libRrefblas.so: undefined symbol: xerbla_
The test file (py_test.py) looks like this:
import os
import ctypes
for file in os.listdir('lib'):
if os.path.isfile(os.path.join('lib', file)):
ctypes.cdll.LoadLibrary(os.path.join('lib', file))
os.environ["R_HOME"] = os.getcwd()
os.environ["R_USER"] = os.path.join(os.getcwd(), 'rpy2')
os.environ["R_LIBS"] = os.path.join(os.getcwd(), 'library')
os.environ["LD_LIBRARY_PATH"] = os.path.join(os.getcwd(), 'lib')
import sys
sys.path.append(os.path.join(os.getcwd(),'rpy2'))
import rpy2
from rpy2 import robjects
def test_handler(event, context):
robjects.r('''
f <- function(r, verbose=FALSE) {
if (verbose) {
cat("I am calling f().\n")
}
2 * pi * r
}
print(f(3))
''')
test_handler(None,None)
I have lib/libRrefblas.so in my virtual environment. I have scoured google looking for answers but have come up empty. Any suggestions would be greatly appreciated!
If you can get the traceback, that could help, but I suspect the problem is that it's looking for xerbla_ in the wrong place. Is xerbla_ defined in the path to RLIBS? Maybe in libR.so?
Turns out the BLAS that ships with R is corrupt. Your best bet is to make sure that BLAS and Lapack are installed on the machine you are building R on and see if you can get it to build with those libraries instead.
So steps would be to uninstall R, then run
yum -y install lapack-devel.x86_64 lapack.x86_64
yum -y install blas -devel
yum -y install R.x86_64
Check to see if it has still installed with libRrefblas.so. If it has - try deleting that file and see if it will default to the system BLAS. If you get a error because it is still looking for libRrefblas.so
rm lib/libRrefblas.so
cp /usr/lib64/libblas.so.3 lib/
mv lib/libblas.so.3 lib/libRrefblas.so
I am testing an R package called eutradeflows on travis. The package contains test programmed with testthat and I would like to see the output of devtools::test() in travis.
There was a line in the main travis log saying :
Status: 4 NOTEs
See ‘/home/travis/build/stix-global/eutradeflows/eutradeflows.Rcheck/00check.log’
for details
From this answer, I learned that its possible to display a file in the travis log. In .travis.yml I have asked travis to print that file after the test:
- cat /home/travis/build/stix-global/eutradeflows/eutradeflows.Rcheck/00check.log
But it doesn't contain the result of testthat tests.
How can I display the output of testthat tests in travis?
This is particularly important since I have skip instructions in the tests and I would like to know which tests have been skipped.
To display the result of testthat tests,
add this to .travis.yml:
r_binary_packages:
- devtools
- roxygen2
after_success:
- Rscript -e 'devtools::install();devtools::test()'
I used a slightly heavier weight fix to ensure that build and test only happened once. In the build matrix I put the following in for the script
script: |
R CMD build .
R CMD INSTALL RMINC*.tar.gz
R CMD check --as-cran --no-install RMINC*.tar.gz
cat RMINC.Rcheck/00check.log
Rscript -e "\
testr <- devtools::test(); \
testr <- as.data.frame(testr); \
if(any(testr\$error) || any(testr\$warning > 0)) \
stop('Found failing tests') \
"
pass=$?
if [[ $pass -ne 0 || $(grep -i "WARNING\|ERROR" RMINC.Rcheck/00check.log) != "" ]]; then
(exit 1)
else
(exit 0)
fi
This runs R CMD check and properly displays build output, failing on warnings or errors in either check or test phases.