Lambda function failing with /lib64/libc.so.6: version `GLIBC_2.18' not found - python-3.6

I am trying to create a layer of simple-salesforce (Python Library) in AWS lambda, and trying to use it (import it) from my python code. I am having windows machine.
Though I read that there might be issues due to compilation windows so I install ubuntu1804 from windows store and then went ahead with creating zip for lambda layers. (zip is created for python folder with structure "python/lib/python3.6/site-packages/......")
I am using Python 3.6. I went through few articles for this issue but could find any resolution. this Video helped me creating a layer for Pandas & requests in AWS successfully with minor tweaks in pip commands I used
sudo python3 -m pip install simple-salesforce -t build/python/lib/python3.6/site-packages
Exactly same process i used for Simple salesforce and I am getting below error is as below:
Unable to import module 'lambda_function': /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/python/lib/python3.6/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)
Edit: --
Another approach I tried using .whl though this was not giving above error but giving error as "request module not found" and when I add request module layer it gives error authlib not found. (request layers work fine if I comment salesforce related things. Even tried uploading as simple layer same authlib issue I got)
Edit :
Lambda code I am using is as below
the code I am using is basic code which doesnt have any logic with empty imports
import json
import pandas as pd
import requests as req
from simple_salesforce.format import format_soql
def lambda_handler(event, context):
#TODO

I also received the same error while installing pysftp on lambda which uses cryptography library(python)
the error was similiar to (required by /opt/python/lib/python3.6/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)
The solution that worked for me is
1] pip uninstall cryptography
2] pip install cryptography==3.4.8
The following github link explains it in detail
https://github.com/pyca/cryptography/issues/6390

AWS lambda functions are like virtual environments, they do not come with the .so files which are kernel level packages. When installing the python packages you have to make sure the system dependent files are installed with it. This can be achieved by passing the argument --platform to pip install.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
simple-salesforce

I changed my code to not use simple_salesforce library and work out all the logic with Requests ( using Salesforce REST APIs).
This is not really ideal but I could get it working as I had some deliveries to meet.

Related

installation lemon Could NOT find ILOG COIN SOPLEX

I am trying to install a github repository and on running as follows
/home/user/.local/bin/python3 -m pip install -e /home/user/repository_folder_name
I get the following error
ERROR: Command errored out with exit status 1:
command: /home/user/.local/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/user/PyMatching/setup.py'"'"'; __file__='"'"'/home/user/PyMatching/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/user/PyMatching/
Complete output (54 lines):
running develop
running egg_info
writing src/PyMatching.egg-info/PKG-INFO
writing dependency_links to src/PyMatching.egg-info/dependency_links.txt
writing requirements to src/PyMatching.egg-info/requires.txt
writing top-level names to src/PyMatching.egg-info/top_level.txt
reading manifest file 'src/PyMatching.egg-info/SOURCES.txt'
writing manifest file 'src/PyMatching.egg-info/SOURCES.txt'
running build_ext
-- pybind11 v2.4.dev4
CMake Warning (dev) at lib/lemon/CMakeLists.txt:6 (PROJECT):
Policy CMP0048 is not set: project() command manages VERSION variables.
Run "cmake --help-policy CMP0048" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
The following variable(s) would be set to empty:
LEMON_VERSION
This warning is for project developers. Use -Wno-dev to suppress it.
-- Could NOT find ILOG (missing: ILOG_CPLEX_LIBRARY ILOG_CPLEX_INCLUDE_DIR)
-- Could NOT find COIN (missing: COIN_CBC_LIBRARY COIN_CBC_SOLVER_LIBRARY COIN_CGL_LIBRARY COIN_CLP_LIBRARY COIN_OSI_LIBRARY COIN_OSI_CBC_LIBRARY COIN_OSI_CLP_LIBRARY)
-- Could NOT find SOPLEX (missing: SOPLEX_LIBRARY SOPLEX_INCLUDE_DIR)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/user/PyMatching
Error: could not load cache
There is a similar question here. But I have tried commenting out #CMAKE_POLICY(SET CMP0048 OLD)
and made sure that I am using the latest version of cmake.
I have no idea how to find these development libraries or find alternatives. Please suggest how to resolve this. Also, this is an issue only on the unix server where I am trying to run this. On my local Windows computer, installation went smoothly.
I'm the developer of PyMatching and I've now added it to the Python package index (see here) so you can install the latest version from PyPI with pip:
pip install pymatching
Since pip will fetch the wheels, which have been built for various platforms, this should hopefully fix your issue, as you won't need to build it yourself.
If you still need to build your own local copy of the PyMatching source code for some reason, just cloning the latest version of the repo might also solve your problem, as I've switched to a more recent version of the Lemon C++ library since you posted.
If this doesn't fix the problem, you could also create an issue on GitHub and/or let me know which version of your operating system you're using.

Compiling Jaeger gRPC proto files with Python

I'm currently playing around with Jaeger Query and trying to access its content through the API, which uses gRPC. I'm not familiar with gRPC, but my understanding is that I need to use the Python gRPC compiler (grpcio_tools.protoc) on the relevant proto file to get useful Python definitions. What I'm trying to do is find out ways to access Jaeger Query by API, without the frontend UI.
Currently, I'm very stuck on compiling the proto files. Every time I try, I get dependency issues (Import "fileNameHere" was not found or has errors.). The Jaeger query.proto file contains import references to files outside the repo. Whilst I can find these and manually collect them, they also have dependencies. I get the impression that following through and collecting each of these one by one is not how this was intended to be done.
Am I doing something wrong here? The direct documentation through Jaeger is limited for this. The below is my basic terminal session, before including any manually found files (which themselves have dependencies I would have to go and find the files for).
$ python -m grpc_tools.protoc --grcp_python_out=. --python_out=. --proto_path=. query.proto
model.proto: File not found.
gogoproto/gogo.proto: File not found.
google/api/annotations.proto: File not found.
protoc-gen-swagger/options/annotations.proto: File not found.
query.proto:20:1: Import "model.proto" was not found or had errors.
query.proto:21:1: Import "gogoproto/gogo.proto" was not found or had errors.
query.proto:22:1: Import "google/api/annotations.proto" was not found or had errors.
query.proto:25:1: Import "protoc-gen-swagger/options/annotations.proto" was not found or had errors.
query.proto:61:12: "jaeger.api_v2.Span" is not defined.
query.proto:137:12: "jaeger.api_v2.DependencyLink" is not defined.
Thanks for any help.
A colleague of mine provided the answer... It was hidden in the Makefile, which hadn't worked for me as I don't use Golang (and it had been more complex than just installing Golang and running it, but I digress...).
The following .sh will do the trick. This assumes the query.proto file is a subdirectory from the same location as the script below, under model/proto/api_v2/ (as it appears in the main Jaeger repo).
#!/usr/bin/env sh
set +x
rm -rf ./js_out 2> /dev/null
mkdir ./js_out
PROTO_INCLUDES="
-I model/proto \
-I idl/proto \
-I vendor/github.com/grpc-ecosystem/grpc-gateway \
-I vendor/github.com/gogo/googleapis \
-I vendor/github.com/gogo/protobuf/protobuf \
-I vendor/github.com/gogo/protobuf"
python -m grpc_tools.protoc ${PROTO_INCLUDES} --grpc_python_out=./python_out --python_out=./python_out model/proto/api_v2/query.proto
This will definitely generate the needed Python file, but it will still be missing dependencies.
I did the following to get the Jaeger gRPC Python APIs:
git clone --recurse-submodules https://github.com/jaegertracing/jaeger-idl
cd jaeger-idl/
make proto
Use the files inside proto-gen-python/.
Note:
While importing the generated code, if you face the error:
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
Do:
pip3 install --upgrade pip
pip3 install --upgrade protobuf

Setting up Yocto on my Ubuntu (Ubuntu 18.04.3 LTS/bionic) with error "ImportError: No module named '_sqlite3'"

my first posting on setting up Yocto development environment
on my Ubuntu system (Ubuntu 18.04.3 LTS/bionic), based on the information enclosed in the document from
this web link (https://www.yoctoproject.org/docs/current/brief-yoctoprojectqs/brief-yoctoprojectqs.html).
All is well until... ~/poky/build$ bitbake core-image-sato
which results in this error:
File "/usr/local/lib/python3.5/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named '_sqlite3'
Below is my effort to proceed past this error, which didn't resolve the
error above. Please be generous and provide some guidance. I searched for
relevant posting locations; any advice on a better place is appreciated.
Thank you.
------------------------------------------------
A web search on this error () results in:
How to Use SQLite in Ubuntu | Chron.com
with
~/poky/build$ sudo apt-get install sqlite3 libsqlite3-dev
which tells me this:
Reading package lists... Done
Building dependency tree
Reading state information... Done
libsqlite3-dev is already the newest version (3.22.0-1ubuntu0.1).
sqlite3 is already the newest version (3.22.0-1ubuntu0.1).
The following packages were automatically installed and are no longer
required:
linux-headers-5.0.0-23 linux-headers-5.0.0-23-generic linux-image-5.0.0-23-generic linux-modules-5.0.0-23-generic
linux-modules-extra-5.0.0-23-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
So, evidently sqlite3 exists on my system. Here are the SO references that I checked:
[ImportError: No module named '_sqlite3' in python3.3][1]
[importerror no module named '_sqlite3' python3.4][2]
[ImportError: No module named _sqlite3 (even after doing eveything)][3]
[ImportError: No module named _sqlite3][4]
[1]: https://stackoverflow.com/questions/20126475/importerror-no-module-named-sqlite3-in-python3-3
[2]: https://stackoverflow.com/questions/24052137/importerror-no-module-named-sqlite3-python3-4
[3]: https://stackoverflow.com/questions/35889383/importerror-no-module-named-sqlite3-even-after-doing-eveything
[4]: https://stackoverflow.com/questions/2665337/importerror-no-module-named-sqlite3
I have just kicked off a build verifying the Brief-Quickstart steps verbatim on an otherwise fresh Ubuntu 18.04 install. There is not even SQLite installed at all, yet the build proceeds nicely. So the chances are pretty high the python installation in your development host is busted in some way or the other. Yet, there might be reasons for it:
you maybe selected python 3.5 explicitly because some other thing you did requires it
you maybe selected python 3.5 implicitly because you forwarded from on old installation, installed something that depended on it, or similar.
In any case, I'd guess that now tinkering with the link might break things somewhere else on your machine, which should be avoided IMHO.
So what are your options now? My advice would be to start building in a container, in the simplest for that requires no more than installing docker and kicking off docker run -it ubuntu:bionic /bin/bash - at least to verify things are generally working.
In the longer term you might want to make a specialized container for this with one or two additions:
1) have all the needed packages set up already
2) using a standard user instead of root.
This is the way I do things personally. An alternative would be to use the prepared things by CROPS as it is a known good solution, and it significantly reduces problems originating from host system pecularities.

IPython magic extension not found

I want to look at a database schema using %load_ext schemadisplay_magic then %schema but schemadisplay_magic will load. I get an error message saying:
No module named 'schemadisplay_magic'
IPython version 5.8.0
I have tried looking it up, but can't find anything about it anywhere.
%load_ext sql
%load_ext schemadisplay_magic
I expected to be able to load %schema, but i just get an error message.
ModuleNotFoundError: No module named 'schemadisplay_magic'
It seems as though this magic is the creation of innovationOUtside and hasn't been published on PyPi -- here's the git repository
you can install it using
pip install git+https://github.com/innovationOUtside/ipython_magic_sqlalchemy_schemadisplay.git#egg=ipython_magic_sqlalchemy_schemadisplay
I was testing it in a virtualenv with a bunch of stuff already installed, so this may not be an issue in a clean environment, but I found that I also needed to upgrade sqlalchemy-schemadisplay beyond the 1.3 that's on PyPi. I used this command:
pip install -U git+https://github.com/fschulze/sqlalchemy_schemadisplay.git#egg=sqlalchemy-schemadisplay
Trying to use the ipython magic with sqlalchemy-schemadisplay 1.3 results in this error:
TypeError: create_schema_graph() got an unexpected keyword argument 'restrict_tables'

pypy failing to load jupyter notebook... why?

I am trying to load jupyter notebook with PyPy. I have done it already on different machines and I never got any problem. This time, however, (on an iMac with OSX 10.11.16) I am getting the following error message:
This is the script I use to load the notebook with PyPy:
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(start_ipython())
I run it from bash: pypy script.py notebook
If I run it without notebook, the IPython session starts as it is supposed to, so the issue must be related to the notebook but I can figure it out. Of course, zeromq is installed, but I have no idea what headers in the error message refers to... Any clue on how to solve this?
P.S. I have used pypy -m pip install jupyter to install it.
"zeromq headers" means what is necessary for developing with, as opposed to only using, the zeromq library. It may be in some "zeromq-dev" package or similar, depending on what distribution system you use on your OS/X.
Normally you'd get such an error when trying to do pip install, but in this case, "zmq" is written using the old CFFI style, deprecated since 2015; as a result it requires the headers at run-time.

Resources