At work I received an old shiny application bundle. In there there is both a manifest.json file. Coming from python, the file looks like it could rebuild the original dependecies. The question is, how do install it in my current setup? I would like to have the equivalent of a poetry install or pip install -r requirements.txt
Thanks to the comment, I made my own solution that worked well.
First create a requirements.txt with a python script.
"""
Read a manifest.json and output a requirements.txt file
"""
import json
file = 'manifest.json'
out = 'requirements.txt'
with open(file) as json_file:
data = json.load(json_file)
with open(out, 'w') as f:
for pkg_name in data['packages'].keys():
pkg_version = data['packages'][pkg_name]['description']['Version']
res = f'{pkg_name} {pkg_version} \n'
f.write(res)
Then install all the old dependencies with this bash script
while IFS=" " read -r package version;
do
Rscript -e "devtools::install_version('"$package"', version='"$version"')";
done < "requirements.txt"
I am trying to create a layer of simple-salesforce (Python Library) in AWS lambda, and trying to use it (import it) from my python code. I am having windows machine.
Though I read that there might be issues due to compilation windows so I install ubuntu1804 from windows store and then went ahead with creating zip for lambda layers. (zip is created for python folder with structure "python/lib/python3.6/site-packages/......")
I am using Python 3.6. I went through few articles for this issue but could find any resolution. this Video helped me creating a layer for Pandas & requests in AWS successfully with minor tweaks in pip commands I used
sudo python3 -m pip install simple-salesforce -t build/python/lib/python3.6/site-packages
Exactly same process i used for Simple salesforce and I am getting below error is as below:
Unable to import module 'lambda_function': /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/python/lib/python3.6/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)
Edit: --
Another approach I tried using .whl though this was not giving above error but giving error as "request module not found" and when I add request module layer it gives error authlib not found. (request layers work fine if I comment salesforce related things. Even tried uploading as simple layer same authlib issue I got)
Edit :
Lambda code I am using is as below
the code I am using is basic code which doesnt have any logic with empty imports
import json
import pandas as pd
import requests as req
from simple_salesforce.format import format_soql
def lambda_handler(event, context):
#TODO
I also received the same error while installing pysftp on lambda which uses cryptography library(python)
the error was similiar to (required by /opt/python/lib/python3.6/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)
The solution that worked for me is
1] pip uninstall cryptography
2] pip install cryptography==3.4.8
The following github link explains it in detail
https://github.com/pyca/cryptography/issues/6390
AWS lambda functions are like virtual environments, they do not come with the .so files which are kernel level packages. When installing the python packages you have to make sure the system dependent files are installed with it. This can be achieved by passing the argument --platform to pip install.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
simple-salesforce
I changed my code to not use simple_salesforce library and work out all the logic with Requests ( using Salesforce REST APIs).
This is not really ideal but I could get it working as I had some deliveries to meet.
I have the Atom editor installed on my Mac, with the Beautify Package. Works great for HTML,etc, but when I try to Beautify SQL it says that it needs SQLFormat, which is not installed by default. When I follow the github link it shows that SQLFormat is Python, install instructions are downalod zip and 'pip install sqlparse'. Do I need to install Python to load sqlparse? Anyone have some simple instructions on how to load on a mac?
first install sqlparse with pip ( to do this you need to have python)
pip install sqlparse
Once it's installed note down the python version with which its installed. e.g. in my case it was python3.7 / python3
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: sqlparse in ./Library/Python/3.7/lib/python/site-packages (0.4.2)
make sure /usr/local/bin/ is on your PATH variable by running echo $PATH if not add it your PATH variable.
then create a file sqlformat under /usr/local/bin/ (i.e. /usr/local/bin/sqlformat) using any editor of your choice and paste below snippet, make sure to use correct version of python and save the file
#!/bin/bash
FLAGS="$f $p $#"
exec python3 -m sqlparse $FLAGS
finally do change the file permission by
chmod 755 /usr/local/bin/sqlformat
The first lines I run when launching my rstudio session are:
library(reticulate)
use_python("/usr/local/lib/python3.6/site-packages")
However, when I run py_config() it shows as still using the default python 2.7 installation.
This is an issue because I'm unable to import any modules that were installed for python3. Any idea why this isn't working? I followed documentation fairly closely.
I observed that neither the technique "use_python('path')" nor the tactic of Sys.setenv(RETICULATE_PYTHON = 'path') in .RProfile worked for me (of course I am sure it must have worked for others.)
In any case the line at terminal,
which -a python python3
did produce two paths to choose from (one for python2 and one for python3 installed on my mac), so then I was able to create a ".Renviron" file in my home directory with this single line in it:
RETICULATE_PYTHON="/usr/local/bin/python3"
After I restarted RStudio, library(reticulate) activates the desired python3, and repl_python() opens a python3 interactive window, etc. etc.
It worked for me:
Sys.setenv(RETICULATE_PYTHON = "/usr/bin/python3")
library(reticulate)
It seems important that you set RETICULATE_PYTHON before you first use reticulate.
use_python("path/to/python3") definitely does not work, although the Reticulate Python version configuration article says so. Don't believe it! :-)
I have tried to set the interpreter with the current Reticulate version (1.13), and the package gave me a very honest answer:
> library("reticulate")
> repl_python()
Python 2.7.15 (/usr/bin/python)
Reticulate 1.13 REPL -- A Python interpreter in R.
> use_python('/usr/bin/python3', require=T)
ERROR: The requested version of Python ('/usr/bin/python3') cannot be
used, as another version of Python ('/usr/bin/python') has already been
initialized. Please restart the R session if you need to attach
reticulate to a different version of Python.
Error in use_python("/usr/bin/python3", require = T) :
failed to initialize requested version of Python
Luckily, putting a .Renviron file containing the line RETICULATE_PYTHON="/path/to/python3" into the user's home directory does work:
> library("reticulate")
> py_config()
python: /usr/bin/python3
libpython: /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so
pythonhome: /usr:/usr
version: 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0]
numpy: [NOT FOUND]
NOTE: Python version was forced by RETICULATE_PYTHON
Finally, here comes the added value of my answer:
You can configure the Reticulate Python interpreter for all users by adding the RETICULATE_PYTHON line to the global Renviron file. It is usually found in the etc subdirectory of R's home directory. You can find out where R's home is by running the R.home() function in the R interpreter. In my case (Ubuntu 18.04.3 LTS) it was /usr/lib/R, so I edited /usr/lib/R/etc/Renviron. You obviously need admin rights to do this.
The only thing that work for me on Mac OSX, perform the following commands in the terminal:
touch $HOME/.Renviron
Then open it, I use vim, so my command is then the following:
vim $HOME/.Renviron
Add the following (for anaconda):
RETICULATE_PYTHON="/anaconda3/bin/python"
Otherwise, in the terminal type: which python3 and enter your output path
RETICULATE_PYTHON="your path from which python3"
It worked for me:
Sys.setenv(RETICULATE_PYTHON = "C:\ProgramData\Anaconda3")
library(reticulate)
repl_python()
Install numpy in the python environment you wish to use.
pip install numpy
I found that Reticulate refuses to use a version of python that does not have numpy installed.
I figured this out by running:
library("reticulate")
py_discover_config()
Reticulate skipped the first two version of python listed and used the third. I noticed it printed out the numpy version so it was probably looking for it as a requirement. It is not mentioned in the docs and should probably be added as a common problem.
For windows users, after creating a virtualenv python :
virtualenv python
this worked to use the virtualenv
Sys.setenv(RETICULATE_PYTHON = "python/Scripts/python.exe")
library(reticulate)
py_config()
EDIT:
This works for any path, e.g.
Sys.setenv(RETICULATE_PYTHON = "C:/Users/UserA/Anaconda3/envs/myEnv/python.exe")
To avoid having to do it each time, add the Sys.setenv line to RProfile:
file.edit(file.path("~", ".Rprofile"))
file.edit(".Rprofile")
You have to discover the path to your the .exe python file and then specify it in the .Renviron file.
The best is to create a conda environment to your project and then specify it as your reticulate path.
library(reticulate)
conda_create("environment name", packages = c("pandas", "matplotlib", "seaborn"), conda = "auto") # creates conda environment for your project
conda_python("environment name") # returns the .exe python path for your environment
Keep this path for the .Renviron file!!
To edit the .Renviron file:
library(usethis)
edit_r_environ()
Then, the .Renviron file will open. Paste the .exe path in it as follows:
RETICULATE_PYTHON="YOUR ENVIRONMENT PATH/python.exe"
Remember to paste it with / instead of "\".
Restart the R session and use the following command to check if everything is right:
library(reticulate)
py_config()
If everything is right, you should get something like:
python: YOUR ENVIRONMENT PATH/python.exe
libpython: YOUR ENVIRONMENT PATH/python39.dll
pythonhome: YOUR ENVIRONMENT PATH
version: 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: YOUR ENVIRONMENT PATH\lib\site-packages\numpy
numpy_version: 1.21.2
After this it should work :)
Posting this as an answer for visibility. None of the other methods worked for me. Fortunately you can select your interpreter from Tools→Global Options→Python.
It will ask you to restart R Studio. But when you do and use reticulate::py_config() it will give the correct version.
I am using Sublime Text 2 (2.0.1, build 2217) on Mac OS X (10.7.2).
I have installed the SASS Build package, but every time I try to build the SASS file, I get the following error:
[Errno 2] No such file or directory
My path seems to be correct, and copying the cmd directly and pasting it into the terminal works, I'm really confused why it just isn't working from within ST2.
Most likely such situation happened due to different Ruby versions installed on a machine.
For example I have 1.8.7 installed with Mac OS X by default and later I installed 1.9.3 using rvm for multiuser environment (i.e. into /usr/local/rvm)
I set current version in rvm but GEM_HOME still points to gems from 1.8.7
The only straight and dumb solution I've found - hack environment variables for Sublime Text 2. This can be done creating .py file with any name in folder:
~/Library/Application Support/Sublime Text 2/Packages/User/
See more information in Rob Dodson's blog post:
http://robdodson.me/blog/2012/05/14/hacking-the-path-variable-in-sublime-text/
Example .py that works for me:
import os
LOCAL = '/usr/local/bin:/usr/local/sbin:'
RVM = '/usr/local/rvm/bin:'
RVMGEMS = '/usr/local/rvm/gems/ruby-1.9.3-p392/bin:'
GEMHOME = '/usr/local/rvm/gems/ruby-1.9.3-p392'
# Sublime's default path is
# /usr/bin:/bin:/usr/sbin:/sbin
os.environ['PATH'] += ':'
os.environ['PATH'] += LOCAL
os.environ['PATH'] += RVM
os.environ['PATH'] += RVMGEMS
os.environ['GEM_HOME'] = GEMHOME
print 'PATH = ' + os.environ['PATH']