error: db type is dbm.gnu, but the module is not available in windows - python-3.6

I installed python3.6 in windows machine. And get below error while open my.db file.
my.db file created by my program in ubuntu16.04 in python3.6, using shelve module.
In [1]: import shelve
In [2]: db = shelve.open("etc/my.db")
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-2-b4828c8ee6e1> in <module>()
----> 1 db = shelve.open("etc/my.db")
c:\Python36\Lib\shelve.py in open(filename, flag, protocol, writeback)
241 """
242
--> 243 return DbfilenameShelf(filename, flag, protocol, writeback)
c:\Python36\Lib\shelve.py in __init__(self, filename, flag, protocol, writeback)
225 def __init__(self, filename, flag='c', protocol=None, writeback=False):
226 import dbm
--> 227 Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
228
229
c:\Python36\Lib\dbm\__init__.py in open(file, flag, mode)
89 elif result not in _modules:
90 raise error[0]("db type is {0}, but the module is not "
---> 91 "available".format(result))
92 else:
93 mod = _modules[result]
error: db type is dbm.gnu, but the module is not available
Please help, how can I install a missing module in windows.

As mentioned by #alexander-p , the problem might be the __pycache__ folders in your source code.
In my case was that I did replace a venv folder with a virtual environment by a new folder with the same name but a newer version of Python (~3.8~ → 3.9), and using at the same time PyCharm (that used the venv setup).
Deleting all the Python caches (and restarting PyCharm just in case) solved the problem.
You can do so with:
$ find . -name __pycache__ | xargs rm -Rv
If the venv folder is inside the same folder where your source code is placed, better to execute:
$ find . -name __pycache__ | grep -v venv | xargs rm -Rv
So the cache inside the venv/ folder is not deleted.

Related

How do I cross-compile a Qt-project with a recipe in yocto

I am trying to Cross-compile a qt-project from a recipe. I have created a recipe file but when I try to bitbake it. I am met with error
Here is my recipe file
DESCRIPITION = "my_project File Transfer"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
SRC_URI = "git://git#bitbucket.org/johndoe/my_ui.git;protocol=ssh;rev=master"
S = "${WORKDIR}/git/my_project"
RDEPENDS_${PN} ="bash"
inherit qmake5
require recipes-qt/qt5/qt5.inc
do_install_append() {
## Creating Folder Structure
install -d ${D}/opt/my_project/bin
install -d ${D}/home/root/my_project
install -d ${D}/home/root/my_project/font
install -d ${D}/home/root/my_project/Images
install -d ${D}/home/root/my_project/Qml
###compile the project
oe_runmake INSTALL_ROOT=${D} install
#### Copying files
install -m 0755 ${S}/font/* ${D}/home/root/my_project/font/
install -m 0755 ${S}/Images/* ${D}/home/root/my_project/Images/
install -m 0755 ${S}/Qml/* ${D}/home/root/my_project/Qml/
}
FILES_${PN} = "/home/root/my_project"
The error that I see is
Sstate summary: Wanted 335 Found 327 Missed 8 Current 1958 (97% match, 99% complete)
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: myproject-project-1.0-r0 do_configure: Error calling /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/recipe-sysroot-native/usr/bin/qmake -makefile -o Makefile /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/git/myproject/myproject.pro --
ERROR: myproject-project-1.0-r0 do_configure: Function failed: do_configure (log file is located at /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/temp/log.do_configure.20982)
ERROR: Logfile of failure stored in: /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/temp/log.do_configure.20982
Log data follows:
| DEBUG: Executing shell function qmake5_base_preconfigure
| DEBUG: Shell function qmake5_base_preconfigure finished
| DEBUG: Executing shell function do_configure
| NOTE: qmake prevar substitution: ' '
| Could not find qmake spec 'linux-oe-g++'.
| Error processing project file: /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/git/myproject/myproject.pro
| ERROR: Error calling /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/recipe-sysroot-native/usr/bin/qmake -makefile -o Makefile /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/git/myproject/myproject.pro --
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_configure (log file is located at /home/blue/yacto/rpi-qt5/build/tmp/work/all-poky-linux/myproject-project/1.0-r0/temp/log.do_configure.20982)
ERROR: Task (/home/blue/yacto/poky-warrior-21.0.1/meta-rpi_custom/recipes-custom/myproject-project/myproject-project_1.0.bb:do_configure) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4242 tasks of which 4241 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/blue/yacto/poky-warrior-21.0.1/meta-rpi_custom/recipes-custom/myproject-project/myproject-project_1.0.bb:do_configure
I know that in order to cross-compile. I had to run qmake from my cross-compile tool-chain location and then run make command on it.
I am guessing that is what's missing in my recipe. So my question is, Do I add that in my do_configure
if thats the case can anyone help me or point me how do I populate my do_configure
Is it as simple as source /opt/poky/2.7.1/environment----
then qmake
I am drawing a blank at this step
Please let me know what am I doing wrong
Edit1: remove inherit allarch from recipe
I would just leave inherit qmake5 and put DEPENDS + = "qtbase qtxyz ...", qtxyz would be the list of other dependency modules.

Get Access to Airflow Hooks within a jupyter notebook

I have Airflow running using the postgres backend - all fine. Additionally I have running a Jupyter server on the same host where Airflow runs. Now I thought I can just access the airflow hooks from a notebook.
import pandas as pd
import numpy as np
import matplotlib as plt
from airflow.hooks.mysql_hook import MySqlHook
mysql = MySqlHook(mysql_conn_id = 'mysql-x')
sql = "select 1+1"
mysql.get_pandas_df(sql)
But I get this exception message:
OperationalError: (sqlite3.OperationalError) no such table: connection
[SQL: SELECT connection.password AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted AS connection_is_extra_encrypted
FROM connection
WHERE connection.conn_id = ?]
[parameters: ('mysql-x',)]
(Background on this error at: http://sqlalche.me/e/e3q8)
But what makes me get suspicious is that it not only does not find the connection_id (which I clearly can see in the Airflow UI). It also says: sqlite3.OperationalError - it very much looks like it is not even connected to the same postgres database. I have checked os.environ["AIRFLOW_HOME"] which seems to be correct.
EDIT 1:
After I start the jupyter notebook server after airflow, such that all environment variables are set, I get a different error:
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py in __get__(self, instance, owner)
351
352 def __get__(self, instance, owner):
--> 353 retval = self.descriptor.__get__(instance, owner)
354 # detect if this is a plain Python #property, which just returns
355 # itself for class level access. If so, then return us.
/usr/local/lib/python3.7/site-packages/airflow/models/connection.py in get_password(self)
188 "Can't decrypt encrypted password for login={}, \
189 FERNET_KEY configuration is missing".format(self.login))
--> 190 return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
191 else:
192 return self._password
/usr/local/lib/python3.7/site-packages/cryptography/fernet.py in decrypt(self, msg, ttl)
169 except InvalidToken:
170 pass
--> 171 raise InvalidToken
InvalidToken:
You can use this dockerfile to reproduce it:
FROM apache/airflow
USER root
# install mysql client
RUN apt-get update && apt-get install -y mariadb-client-10.3 unzip
# install mssql client and tools
RUN apt-get install -y curl gnupg libicu-dev libicu63
RUN curl https://packages.microsoft.com/keys/microsoft.asc -o key
RUN apt-key add < key
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/msprod.list
ENV ACCEPT_EULA=Y
RUN apt-get update && apt-get install -y mssql-tools msodbcsql17 unixodbc unixodbc-dev unzip libunwind8
RUN curl -Lq 'https://go.microsoft.com/fwlink/?linkid=2108814' -o sqlpackage-linux-x64-latest.zip
RUN mkdir /opt/sqlpackage/ && unzip sqlpackage-linux-x64-latest.zip -d /opt/sqlpackage/
RUN chmod a+x /opt/sqlpackage/sqlpackage && ln -sfn /opt/sqlpackage/sqlpackage /usr/bin/sqlpackage
RUN ln -sfn /opt/mssql-tools/bin/sqlcmd /usr/bin/sqlcmd
# install notebooks
RUN pip install jupyterlab pandas numpy scikit-learn matplotlib pymssql
#RUN cat /entrypoint.sh
# start additional notebok server
# this is a dirty hack but for the sake of this prototype good enough
RUN sed -i -e's/\# Run the command/airflow scheduler \& \njupyter notebook --ip=0.0.0.0 --port=9000 --NotebookApp.token="" --NotebookApp.password="" \& \n/' /entrypoint
EXPOSE 9000
# switch back to airflow user
USER airflow
RUN airflow initdb
RUN alias ll='ls -al'

Homebrew: lack permissions when installing

I'm trying to write my first formula, for the Rserve package in R: https://www.rforge.net/Rserve/ .
So far I have this:
# Documentation: https://docs.brew.sh/Formula-Cookbook
# https://www.rubydoc.info/github/Homebrew/brew/master/Formula
class Rserve < Formula
desc "Rserve acts as a socket server (TCP/IP or local sockets) which allows binary requests to be sent to R."
homepage "http://www.rforge.net/Rserve/"
url "https://cran.r-project.org/src/contrib/Rserve_1.7-3.tar.gz"
sha256 "f6d636c736c3f16487d9987e54554fe0c55257b9bc0671b43e536d832e513027"
depends_on "r"
depends_on "gettext"
def install
system("whoami")
system("#{HOMEBREW_PREFIX}/bin/R CMD install --configure-args=\"CPPFLAGS=-L#{HOMEBREW_PREFIX}/opt/gettext/lib LDFLAGS=-I#{HOMEBREW_PREFIX}/opt/gettext/include\" .")
end
test do
# `test do` will create, run in and delete a temporary directory.
#
# This test will fail and we won't accept that! For Homebrew/homebrew-core
# this will need to be a test that verifies the functionality of the
# software. Run the test with `brew test Rserve`. Options passed
# to `brew install` such as `--HEAD` also need to be provided to `brew test`.
#
# The installed folder is not in the path, so use the entire path to any
# executables being tested: `system "#{bin}/program", "do", "something"`.
system "false"
end
end
The error I get is:
==> Downloading https://cran.r-project.org/src/contrib/Rserve_1.7-3.tar.gz
Already downloaded: /Users/kwilliams/Library/Caches/Homebrew/downloads/690e0934dcba3770ba80b743f7b2c9cee0250434ed17cc7949bc1eee74d5b170--Rserve_1.7-3.tar.gz
==> Verifying 690e0934dcba3770ba80b743f7b2c9cee0250434ed17cc7949bc1eee74d5b170--Rserve_1.7-3.tar.gz checksum
tar xf /Users/kwilliams/Library/Caches/Homebrew/downloads/690e0934dcba3770ba80b743f7b2c9cee0250434ed17cc7949bc1eee74d5b170--Rserve_1.7-3.tar.gz -C /private/tmp/d20190114-99089-11isfza
cp -pR /private/tmp/d20190114-99089-11isfza/Rserve/. /private/tmp/rserve-20190114-99089-b57ji6/Rserve
chmod -Rf +w /private/tmp/d20190114-99089-11isfza
==> whoami
kwilliams
==> /usr/local/bin/R CMD install --configure-args="CPPFLAGS=-L/usr/local/opt/gettext/lib LDFLAGS=-I/usr/local/opt/gettext/include" .
* installing to library ‘/usr/local/lib/R/3.5/site-library’
Error: ERROR: no permission to install to directory ‘/usr/local/lib/R/3.5/site-library’
==> Formula
Path: /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/rserve.rb
==> Configuration
HOMEBREW_VERSION: 1.9.2-5-g44f4f36
ORIGIN: https://github.com/Homebrew/brew
HEAD: 44f4f36c0db693000410fe664b250a90325f4e32
Last commit: 4 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: de1ecc1d981de9d5165ea9e96242c32023d14d7c
Core tap last commit: 4 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_DEV_CMD_RUN: 1
HOMEBREW_ENABLE_AUTO_UPDATE_MIGRATION: 1
HOMEBREW_LOGS: /Users/kwilliams/Library/Logs/Homebrew
CPU: octa-core 64-bit kabylake
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 10.0 build 1000
Git: 2.17.2 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 11.0.1, 1.8.0_192
macOS: 10.13.6-x86_64
CLT: 10.1.0.0.1.1539992718
Xcode: 10.1
XQuartz: 2.7.11 => /opt/X11
The permissions on /usr/local/lib/R/3.5/site-library should allow me to install there, though:
% ls -al /usr/local/lib/R/3.5/site-library
total 0
drwxr-xr-x 2 kwilliams admin 64 Jan 14 13:38 ./
drwxr-xr-x 3 kwilliams admin 96 Jan 14 13:38 ../
If I put myself in a directory where the Rserve tarball has been freshly expanded, then manually run the command /usr/local/bin/R CMD install --configure-args="CPPFLAGS=-L/usr/local/opt/gettext/lib LDFLAGS=-I/usr/local/opt/gettext/include" ., I don't get the error about "no permission to install to directory".
What's different about the environment that Homebrew is running that command, compared to me running it in a shell, that causes it to fail when run automatically?
My brew doctor output is clean, except for a warning that my docker keg is unlinked, which I assume is unrelated.
Figured it out - there's a sandbox (sandbox-exec, I think) in place that restricts writing to the filesystem outside the prefix directory:
https://discourse.brew.sh/t/potential-permission-issues-in-a-formula-im-writing/3707
I'll need to figure out the correct way to write files to /usr/local/lib/R/3.5/site-library, or symlink them to there from the Cellar directory, or whatever it is.

How does zsh interpret non-absolute paths in shebangs? (WAS: Why does python3 -i permit non-absolute paths in shebang?)

I recently discovered the -i argument to Python, which drops into interactive mode after the script completes. Pretty neat!
$ cat test.py
#!python3 -i
x=5
print('The value of x is ' + str(x))
$ ./test.py
The value of x is 5
>>> print(str(x+1))
6
>>>
zsh: suspended ./test.py
However, when I tried to copy this script to a version that terminates on completion, it fails:
$ cat test1.py
#!python3
x=5
print('The value of x is ' + str(x))
$ ./test.py
/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python: can't open file '
x=5
print('The value of x is ' + str(x))
': [Errno 2] No such file or directory
From some further reading, I discovered that I had originally made a mistake, and #!/usr/bin/env python3 is the correct shebang.
However, I'm curious why a non-absolute path to python3 succeeds only when I give the -i flag. I guess this must be something to do with how zsh interprets non-absolute shebangs, but I don't know enough to know how to investigate that.
System setup: MacOS 10.12.6, iTerm2 3.1.6, zsh 5.2. which python3 gives /usr/local/bin/python3, and that directory is on $PATH.
Interestingly, I don't get the same behaviour on sh:
$ sh
sh-3.2$ cat test.py
#!python3
x=5
print('The value of x is ' + str(x))
sh-3.2$ ./test.py
sh: ./test.py: python3: bad interpreter: No such file or directory
I got some comments suggesting that this is something to do with CWD or permissions. python3 is not in my CWD, and both files have execute permission:
$ ls -al | grep 'py' | awk '{print $1, $10}'
-rw------- .python_history
-rwxr-xr-x test.py
-rwxr-xr-x test1.py
Your kernel will not execute the script unless the interpreter is
specified as an absolute path, or
specified as a path relative to the current working directory
Then if the kernel refuses to execute the script, your shell might take over and try to execute it anyway, interpreting the shebang line according to its own rules (like finding the executable in the $PATH for example).
zsh does attempt to do this. sh does not.
However the way zsh interprets the shebang (and probably subsequent lines) is really really strange. It looks like it always expects a single argument after the command name. See what it does:
$ cat test.py
#!python3 -b -i
x=5
print('The value of x is ' + str(x))
$ strace -f -e execve zsh
execve("/bin/zsh", ["zsh"], 0x7ffd35c9e198 /* 78 vars */) = 0
host% ./test.py
strace: Process 5510 attached
[pid 5510] execve("./test.py", ["./test.py"], 0x558ec6e46710 /* 79 vars */) = -1 ENOENT (No such file or directory)
[pid 5510] execve("/usr/bin/python3", ["python3", "-b -i", "./test.py"], 0x558ec6e46710 /* 79 vars */) = 0
[pid 5510] execve("/usr/lib/python-exec/python3.4/python3", ["/usr/lib/python-exec/python3.4/p"..., "-b -i", "./test.py"], 0x7fffd30eb208 /* 79 vars */) = 0
Unknown option: -
usage: /usr/lib/python-exec/python3.4/python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
[pid 5510] +++ exited with 2 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=5510, si_uid=1000, si_status=2, si_utime=0, si_stime=0} ---
host%
+++ exited with 2 +++
See how ["python3", "-b -i", "./test.py"] are passed as arguments. It seems highly counterintuitive to me to lump the two switches -b and -i together, but that's what zsh does. Python obviously doesn't understand this.
When there are no arguments, the exact behaviour depends on whether there is a space after the program name, but is strange in either case. Check it with strace yourself because you are not going to believe me.
It is my understanding that zsh handling of the shebang line is just buggy.

Grabbing .jar application output stream to console after console was closed and new one opened on Oracle Solaris 11

On Oracle Solaris 11 console when ps -ef | grep java command is issued I can see running some java process PID, which was started on other console window and then it (console window) was closed (.jar application output then was visible). Is it some way to grab again that application output without restarting .jar file?
Application was started like this (as a root user):
java -jar SomeFile.jar &
Write output to file is not an option in this case.
Yes, you can do that, but it involves some mad skills with gdb. Here is how to do that in Linux and I believe you can do the same in Solaris (since it has gdb and it has all needed system calls I'm gonna use further).
There are 3 file descriptors for standard streams:
stdin: 0
stdout: 1
stderr: 2
You are interested in stdout and stderr (both are console output), so you need file descriptors with numbers 1 and 2, just keep it in mind.
Now I'm gonna show you how to do what you ask for "okular" application (instead of your "java" application) for stderr stream.
Run "okular" in terminal, like this:
$ okular &
and then close this terminal. This is just to simulate your situation.
Open another terminal
Look for "okular" process:
$ ps aux | grep okular
Output:
joe 27599 2.2 0.9 515644 73944 ? S 23:46 0:00 okular
So "okular" PID is 27599.
Look for open file descriptors of "okular" process:
$ ls -l /proc/27599/fd
Output:
lrwx------ 1 joe joe 64 Feb 18 23:46 0 -> /dev/pts/0 (deleted)
lrwx------ 1 joe joe 64 Feb 18 23:46 1 -> /dev/pts/0 (deleted)
lrwx------ 1 joe joe 64 Feb 18 23:46 2 -> /dev/pts/0 (deleted)
You see that all 3 streams are deleted.
Now let's attach to our process with gdb:
$ gdb -p 27599 /usr/bin/okular
Inside of gdb perform next operations:
(gdb) p close(2)
(gdb) p creat("/tmp/okular_2", 0600)
(gdb) detach
(gdb) quit
Here we invoked 2 system calls:
close(), to close file for stderr stream of our process
creat(), to create new file for stderr stream of our process
p is gdb command, it prints (in our case) system calls return values.
Now all new stderr output of our process will be appended to text file /tmp/okular_2. We can read it constantly this way:
$ tail -f /tmp/okular_2
Conclusion
Ok, that's it, we revived stderr stream. You can do the same for stdout stream, the only difference is that you need to call "close(1)" instead of "close(2)" in gdb. Also, in your case be sure to replace all "okular" words with your "java" word.
The most of answer was inspired by this article.
If you need to revive stdin stream, you can attach it to pipe (FIFO) file, see details here.
Yes, it is possible to snoop any process output with Solaris native tools.
One way would be using dtrace which allows tracing processes even when they are already grabbed by a debugger or similar tool.
This dtrace script will display a given process stdout:
#!/bin/ksh
pid=$1
dtrace -qn "syscall::write:entry /pid == $pid && arg0 == 1 /
{ printf(\"%s\",copyinstr(arg1)); }"
You should should pass the process id of the java application to trace as its first argument, eg. $(pgrep -f "java -jar SomeFile.jar").
Replace arg0 == 1 by arg0 == 2 if you want to trace stderr vs stdin.
Should you want to see non displayable characters (in octal), you might use this slightly modified version:
#!/bin/ksh
pid=$1
dtrace -qn "syscall::write:entry /pid == $pid && arg0 == 1 /
{ printf(\"%s\",copyinstr(arg1)); }" | od -c
Another native way is to use the truss command. The following script will show all writes from your process to any file descriptors, and will include a full detailed trace for both stdout and stderr (3799 is your target process pid):
truss -w1,2 -t write -p 3799
dtrace:
http://docs.oracle.com/cd/E18752_01/html/819-5488/gcgkk.html
truss:
http://docs.oracle.com/cd/E36784_01/html/E36870/truss-1.html#scrolltoc

Resources