I have been trying to use Hydra with PyInstaller and failed.
I have created a very configuration example similar to example in here.
I noticed that hydra packages are not being found by PyInstaller so I created a simple hook file hook-hydra.py with the following code:
from PyInstaller.utils.hooks import collect_data_files, collect_submodules
datas = collect_data_files('hydra')
hiddenimports = collect_submodules('hydra')
That seemed to solve the module imports failures, but then when I tried to run the executable in the command line I got the following error:
Traceback (most recent call last):
File "lib\site-packages\hydra\_internal\utils.py", line 198, in run_and_report
File "lib\site-packages\hydra\_internal\utils.py", line 321, in <lambda>
File "lib\site-packages\hydra\_internal\hydra.py", line 74, in create_main_hydra2
File "lib\site-packages\hydra\_internal\config_loader_impl.py", line 80, in __init__
File "lib\site-packages\hydra\_internal\config_repository.py", line 22, in __init__
File "lib\site-packages\hydra\_internal\sources_registry.py", line 30, in resolve
ValueError: No config source registered for schema pkg, supported types : []
I can't seem to figure it out, any ideas?
I'm using PyInstaller 3.6 and Hydra 1.0.4
After taking a look at PyInstaller, it looks like it's attempting to discover needed packages and it somehow fails to do a good job for Hydra.
Hydra has some built in plugins that are discovered at runtime, including the config sources. The error suggests that the config sources were not packages by PyInstaller.
If PyInstaller is trying to be clever and only include things it sees a direct dependency on it is likely to fail for Hydra.
Try to add all hydra modules explicitly to your PyInstaller config file.
As an alternative packaging method for your app, take a look at the application packaging example here. It shows you the supported method for installing Hydra apps (with a working example).
Hydra does not play nice with PyInstaller because it dynamically reads the plugin packages using pkgutil.walk_packages.
Check out the issue and answers here: Pyinstaller - include programmatically imported modules
As you are not solving this from a library developer's point of view, copying the package using the data field in the .spec file could solve your issue, although it is a bit hacky. Either this or switching to OmegaConf is a good solution in my opinion.
I faced similar issues and ended up having to create a pyinstaller hook named hook-hydra.py and put it in the pyinstaller hooks folder. It could probably be reduced to lower level hydra namespaces. Leveraging other hooks, I did the following:
Only issue I face is inline using compose overrides=[f"environment.root_path={set_root_path}"], complains about EOF .... etc. But have workaround for that. Logging and Fetching configuration works like a charm.
from PyInstaller.utils.hooks import collect_submodules
# Use this to get hydra internals
# "hydra._internal.core_plugins"
# "hydra.plugins"
# 'hydra._internal.core_plugins','hydra.grammar.gen.OverrideParser'
hiddenimports = collect_submodules('hydra')
This works for me, no problems running from command line, using PyInstaller 4.5.1, Hydra 1.2:
from PyInstaller.utils.hooks import collect_data_files, collect_submodules
datas = collect_data_files('hydra', subdir='conf')
hiddenimports = collect_submodules('hydra')
Related
I have a script in Python that imports out of some other packages, the NLTK package.
The OS is Debian Stretch. Executing it directly on Linux everything works as should it be. But running the mentioned script with Sympony - Process, it returns the following error:
Traceback (most recent call last):
File \"/var/www/html/public/_import.py\", line 1, in <module>
import nltk
ModuleNotFoundError: No module named 'nltk'
If simply I just comment "import nltk", all the script works properly even with Symfony Process at all.
I could not leave this problem, as I have resolved it, without an answer for you that might be facing the same issue!
The problem at all was not caused by Symfony - Process, but, unfortunatly, by instalation behavior of the module named: "NLTK";
In my case, the user used to install things on Linux Debian Stretch was the root.
Try (now, you, before changes and check...) to use the following command in your terminal: pip show nltk.
The output is:
Name: nltk
Version: 3.4.5
Summary: Natural Language Toolkit
Home-page: http://nltk.org/
Author: Steven Bird
Author-email: stevenbird1#gmail.com
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.7/site-packages
Requires: six
Required-by:
Take a look at Location. Now it is the right place. But, as default, pip install nltk will place it, in my case with user "root" installing things, at "/root/.local/lib/python3.7/site-packages/nltk" as the base path! So, when the user "www-data" tried to runing it... ModuleNotFoundError: No module named 'nltk'
No module named 'nltk' - because it was not there to the place should it be accessed!
So, as you might guess now, the problem as caused by permission issues! Unfortunatly, no error (logs) output happened that would drive me to mind that would it be related to permission issues!
The solution:
1) Not messing around with the system or executing unecessary changes was considered at first place;
2) Had uninstalled it with pip the module NLTK and I have seeked to find where the other modules were installed;
3) Installed it (NLTK) again, but now, placing it where I want, I mean, where it should be since the begening: among the others modules - as numpy -, that could be accessed easily and without issues!
The command used was:
pip install --target="/usr/local/lib/python3.7/site-packages"
--upgrade nltk
Now, NLTK resides at /usr/local/lib/python3.7/site-packages followed by all the other modules working properly like a charm!
I hope this helps you!
I have been experimenting with PyInstaller for a few days. My operating system is FreeBSD 10.4 and I use Python 3.6. When I issue this command from the terminal, it works all fine:
# pyinstaller my_script.py
But this command has an unprecedented side effect. When I write this:
if __debug__:
print("debug")
I get "debug" printed to the console, which suggests that PyInstaller compiles the script with debug symbols.
While searching for a solution to this problem, I found that running PyInstaller through the Python interpreter with the -O parameter would solve the problem:
[root#bsd-pwb ~]# /usr/local/bin/python3.6 /usr/local/lib/python3.6/site-packages/PyInstaller/__main__.py my_script.py
But this time, I get this:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/PyInstaller/__main__.py", line 21, in <module>
from . import __version__
ImportError: cannot import name '__version__'
What am I doing wrong here? How can I get PyInstaller to compile with "debug" set to "False"?
I recommend you trying your script exactly as it is but using a Python 3.5 Enviroment, there are so many publications where you can realize that pyinstaller is more stable with python 3.5 versions than 3.6. I had several errors as the one you are commenting and at the end they get solved just running my scripts on python<=3.5
From the docs you can actually get it working by using another method: set explicitely the PYTHONOPTIMIZE environment variable to a non-zero value, in your case:
PYTHONOPTIMIZE=1 pyinstaller myscript.py
I am working on windows platform and have set up all my requirements for NSGA2 solver but still it is not working. I have downloaded and installed MPI, MinGW, SWIG, Pyopt, pyoptsparse but still I am unable to use the pyoptsparse driver.
If someone could help on this, it will be of a great help. Thanks
I have pasted the error below
D:\Anaconda2\Scripts\python.exe D:/OpenMDAO/Mitul/Sellar/Sellar_MDF.py
Traceback (most recent call last):
File "D:/OpenMDAO/Mitul/Sellar/Sellar_MDF.py", line 6, in <module>
from openmdao.drivers.pyoptsparse_driver import pyOptSparseDriver
File "d:\anaconda2\lib\site-packages\openmdao\drivers\pyoptsparse_driver.py", line 19, in <module>
from openmdao.core.driver import Driver
File "d:\anaconda2\lib\site-packages\openmdao\core\driver.py", line 15, in <module>
from openmdao.util.options import OptionsDictionary
ImportError: No module named options
it is difficult to get MPI and pyopt-sparse built correctly on windows. But It has been done successfully. You can see the docs here for detailed instructions
That being said... there is clearly something wrong with your OpenMDAO installation. That error implies that things are not being copied correctly into the installation folder. Since you're using anaconda, I suggest that you create a new anaconda env as follows:
conda create --name om2 python=2 numpy scipy matplotlib
then activate it with source activate om2 and try to re-install openmdao via pip. You can get the very latest by doing
pip install git+http://github.com/OpenMDAO/OpenMDAO.git#master
Once you've done that, before you try to get mpi/petsc/pyopt-sparse installed you should run our test suite. Follow our docs on that, and if the tests all pass you can move on to the other more advanced install steps.
I am attempting to convert a python file to an executable file. Easy enough right?
I used pyinstaller on a simple program that doesn't import anything. It worked like a charm. Then, I tried it with another dummy program with imported modules, (PyQt4, sys, matplotlib) that my actual program would have. Here I encountered problems.
This error appeared when I ran the application in the 'dist' folder pyinstaller created.
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
I found another site with a possible solution to this problem, but his situation wasn't exactly the same: http://code.activestate.com/lists/python-dev/118463/
This lead me to trying the QT designer that I downloaded earlier. Perhaps if I could convert the .ui file it produced into a .py file, I would be fine. I could use his solution and all would be well.
That's when I got this error:
File "C:\Anaconda3\Lib\site-packages\PyQt4\uic\pyuic.py", line 26, in module
from PyQt4 import QtCore
mportError: No module named 'PyQt4'
I should also probably mention that all the modules I have are through Anaconda 3
I thought installing pyqt in a conda... project? Would fix the problem. It didn't. To be honest I don't entirely know what those are for.
Now I'm entertaining the idea of just using the c++ files that QT designer makes instead of converting them and importing python to tell the gui what to do.
What do you guys think would solve the errors above?
Short solution / workaround:
In your python file import the missing module explicitly. In your case: import encodings.
Proper solution:
By importing every module separately so you might end up imorting many modules and submodules. In this case you need tell pyinstaller where to find the modules (e. g. using compile flags).
After hearing Peter Wang talk about the Bokeh plotting environment, I had to try it. The problem is I can't even get the examples to work because I can't get the plot server going. I verified the install of all listed dependencies, and installed continuumweb (a module that is curiously absent from the list of dependencies in the Bokeh README on github). However, I still encountered the error below. It looks like this should come out of continuumweb, but I find no trace of such a sub-module within the repo. Am I looking in the wrong place? (I have not found any other instances of this exception.)
If anyone could provide some guidance, I would be grateful...
choct155#choct155-U46E:~/analysis/anaconda/pkgs/Bokeh$ python runserver.py
Traceback (most recent call last):
File "runserver.py", line 3, in <module>
from bokeh.server import start
File "/home/choct155/analysis/anaconda/pkgs/Bokeh/bokeh/server/start.py", line 76, in <module>
import services
File "/home/choct155/analysis/anaconda/pkgs/Bokeh/bokeh/server/services.py", line 4, in <module>
from continuumweb.launch_process import ManagedProcess
ImportError: No module named launch_process
Exception KeyError: KeyError(139639237560272,) in <module 'threading' from '/home/choct155/analysis/anaconda/lib/python2.7/threading.pyc'> ignored
For those who are reading this later, we've had two versioned releases since the original question, and they are all easy to install directly from PyPI via "pip install bokeh". If you are using Anaconda, it's even easier to install via "conda install bokeh".
For some reason, the pip installer did not perform well here. I just started from scratch by cloning the continuumweb repository and installing from source ('python setup.py install' in the cloned directory)...
All is well.