System information
OS Platform and Distribution: Linux CentOS7.2
TensorFlow Serving installed from (source or binary): Source
TensorFlow Serving version: r1.9
Python Version: 3.6.2
Describe the problem
The centos7.2 install python2.7 defaultly, I want to build tensorflow-serving for python3, So I use the pyenv to install a python3.6.2, and in the pyenv virtualenv, I cloned the tensorflow/serving r1.9, to bazel build it from source. But the at last it failed becase of "import mork" error. I saw in the serving/tools/bazel.rc file, it use the /usr/bin/python as python runner, so I changed the file to my python3.6.2, but failed again. with error like
" _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: /home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _Py_FalseStruct"
Exact Steps to Reproduce
I changed the bazel.rc file as follows
build:cuda --crosstool_top=#local_config_cuda//crosstool:toolchain
build:cuda --define=using_cuda=true --define=using_cuda_nvcc=true
build --action_env PYTHON_BIN_PATH="/home/pyenv/.pyenv/versions/tensorflow-serving/bin/python"
build --define PYTHON_BIN_PATH=/home/pyenv/.pyenv/versions/tensorflow-serving/bin/python # tensorflow-serving is a virtualenv from python3.6.2 installed by pyenv
build --spawn_strategy=standalone --genrule_strategy=standalone
test --spawn_strategy=standalone --genrule_strategy=standalone
run --spawn_strategy=standalone --genrule_strategy=standalone
build --define=grpc_no_ares=true
build --define with_hdfs_support=true # support HDFS
# TODO(b/69809703): Remove once no longer required for TensorFlow to build.
build --copt=-DGEMMLOWP_ALLOW_SLOW_SCALAR_FALLBACK
Source code / logs
the built also failed, the error log is as follows
ERROR:
/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/external/org_tensorflow/tensorflow/BUILD:541:1: Executing genrule #org_tensorflow//tensorflow:python_api_gen failed (Exit 1): bash failed: error executing command
(cd /home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving && \
exec env - \
PATH=/home/pyenv/.pyenv/plugins/pyenv-virtualenv/shims:/home/pyenv/.pyenv/shims:/home/pyenv/.pyenv/bin:/usr/local/ffmpeg/bin/:/opt/jdk1.8.0_112/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pyenv/.local/bin:/home/pyenv/bin \
PYTHON_BIN_PATH=/home/pyenv/.pyenv/versions/tensorflow-serving/bin/python \
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api --root_init_template=external/org_tensorflow/tensorflow/api_template.__init__.py --apidir=bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/app/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/bitwise/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/compat/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/data/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/distributions/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/distributions/bijectors/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/errors/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/estimator/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/estimator/export/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/estimator/inputs/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/feature_column/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/gfile/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/graph_util/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/image/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/initializers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/activations/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/densenet/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/inception_resnet_v2/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/inception_v3/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/mobilenet/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/nasnet/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/resnet50/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/vgg16/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/vgg19/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/applications/xception/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/backend/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/callbacks/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/constraints/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/boston_housing/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/cifar10/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/cifar100/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/fashion_mnist/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/imdb/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/mnist/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/datasets/reuters/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/estimator/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/initializers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/layers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/losses/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/metrics/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/models/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/optimizers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/preprocessing/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/preprocessing/image/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/preprocessing/sequence/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/preprocessing/text/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/regularizers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/utils/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/wrappers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/keras/wrappers/scikit_learn/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/layers/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/linalg/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/logging/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/losses/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/manip/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/math/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/metrics/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/nn/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/nn/rnn_cell/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/profiler/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/python_io/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/resource_loader/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/strings/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/builder/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/constants/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/loader/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/main_op/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/signature_constants/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/signature_def_utils/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/tag_constants/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/saved_model/utils/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/sets/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/sparse/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/spectral/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/summary/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/sysconfig/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/test/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/train/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/train/queue_runner/__init__.py bazel-out/k8-opt/genfiles/external/org_tensorflow/tensorflow/user_ops/__init__.py')
Traceback (most recent call last):
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/tf_serving/../org_tensorflow/tensorflow/tools/api/generator/create_python_api.py", line 27, in <module>
from tensorflow.python.util import tf_decorator
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: /home/pyenv/.cache/bazel/_bazel_pyenv/3e832003f40725f2f414b6fe95aca127/execroot/tf_serving/bazel-out/host/bin/external/org_tensorflow/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _Py_FalseStruct
Then how can I build tensorflow-serving r1.9 with python3.6.2 support on CentOS7?
change the tools/bazel.rc to config python path can resolve this problem.
but this seems not true for tensorflow-serving r1.10 or r1.11
Related
I have a GCP Cloud Composer environment with airflow version composer-1.10.0-airflow-1.10.6 and python 3, 3.6 to be precise. I am calling an apache-beam pipeline on Dataflow using a python_operator.PythonOperator, operator. Here is the code snippet
Calling the pipeline function
test_greeting = python_operator.PythonOperator(
task_id='python_pipeline',
python_callable=run_pipeline
)
The pipeline function is as follows
def run_pipeline():
print("Test Pipeline")
pipeline_args=[
"--runner","DataflowRunner",
"--project","*****",
"--temp_location","gs://******/temp",
"--region","us-east1",
"--job_name","job1199",
"--zone","us-east1-b"
]
pipeline_options=PipelineOptions(pipeline_args)
pipe=beam.Pipeline(options=pipeline_options)
small_sum = (
pipe
| beam.Create([18,5,7,7,9,23,13,5])
| "Combine Globally" >> beam.CombineGlobally(AverageFn())
| 'Write results' >> beam.io.WriteToText('gs://******/ouptut_from_pipline/combine')
)
run_result=pipe.run()
run_result.wait_until_finish()
return "True"
When I run this the pipeline execution runs in dataflow but fails with the following error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line 648, in do_work
work_executor.execute()
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py", line 150, in execute
test_shuffle_sink=self._test_shuffle_sink)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py", line 116, in create_operation
is_streaming=False)
File "apache_beam/runners/worker/operations.py", line 1032, in apache_beam.runners.worker.operations.create_operation
File "apache_beam/runners/worker/operations.py", line 845, in apache_beam.runners.worker.operations.create_pgbk_op
File "apache_beam/runners/worker/operations.py", line 903, in apache_beam.runners.worker.operations.PGBKCVOperation.__init__
File "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line 290, in loads
return dill.loads(s)
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 275, in loads
return load(file, ignore, **kwds)
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 270, in load
return Unpickler(file, ignore=ignore, **kwds).load()
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 472, in load
obj = StockUnpickler.load(self)
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 462, in find_class
return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named 'unusual_prefix_162ac8b7030d5bd1ff5f128a26483932d3968a4d_python_bash'
The beam version is Apache Beam Python 3.6 SDK 2.19.0.
I suspect the version of Python 3.6 may be the issue as calling the pipeline directly (as a runner) from my local system works fine and my local system is running python 3.7.
I cant find a way to test this theory though.
It would be helpful to get tips of how to resolve this issue.
WARNING: The wheel package is not available.
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-nu9_zlgc/numexpr/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-nu9_zlgc/numexpr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-wheel-nu9_zlgc/numexpr/pip-egg-info
cwd: /tmp/pip-wheel-nu9_zlgc/numexpr/
ERROR: Failed building wheel for numpy
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-tr7p7knw/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-tr7p7knw/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
cwd: /tmp/pip-wheel-tr7p7knw/numpy
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-wheel-nu9_zlgc/numexpr/setup.py", line 228, in <module>
setup_package()
File "/tmp/pip-wheel-nu9_zlgc/numexpr/setup.py", line 224, in setup_package
setup(**metadata)
File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
_install_setup_requires(attrs)
File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
replace_conflicting=True,
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 782, in resolve
replace_conflicting=replace_conflicting
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1065, in best_match
return self.obtain(req, installer)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1077, in obtain
return installer(requirement)
File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
return fetch_build_egg(self, req)
File "/usr/local/lib/python3.6/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
raise DistutilsError(str(e))
distutils.errors.DistutilsError: Command '['/usr/local/bin/python3.6', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpxhcxiwia', '--quiet', 'numpy>=1.7']' returned non-zero exit status 1.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
error: Command '['/usr/local/bin/python3.6', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpgvr79eoz', '--quiet', 'numexpr>=2.6.2']' returned non-zero exit status 1.
Python Version 3.6.0,
OS : Ubuntu 16 and 18 and
OS : Rhel 7.3
Facing above error while running below commands
sudo python3.6 setup.py install
sudo python3.6 setup.py test
I'm trying to install an R package in my anaconda environment DS-ML i've created with conda skeleton. The environment has R and python installed in it.
The R package is called r-ffbase, and it is on CRAN repository.
I'm on C:\users\public. After typing:
conda skeleton cran ffbase
a skeleton folder named "r-ffbase" has been created successfully.
I get a conda exception while building the package -
conda.exceptions.ResolvePackageNotFound:
- m2w64_c_win-64
I couldn't find any package in any channel I think. Where can I get this package from?
And why do I need this package for?
This is for a windows 10 server (win64), with Anaconda installed with conda 4.7.12, numpy 1.16.5 and R version 3.6.1 in the DS-ML enviroment.
I've tried typing in the conda prompt:
conda build r-ffbase --R=3.6.1 --numpy=1.16.5
I get the following full error message:
(DS-ML) C:\Users\Public>conda build r-ffbase --R=3.6.1 --numpy=1.16.5
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from config.variant
INFO:conda_build.variants:Adding in variants from config.variant
Attempting to finalize metadata for r-ffbase
INFO:conda_build.metadata:Attempting to finalize metadata for r-ffbase
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Leaving build/test directories:
Work:
C:\ProgramData\Anaconda3\conda-bld\work
Test:
C:\ProgramData\Anaconda3\conda-bld\test_tmp
Leaving build/test environments:
Test:
source activate C:\ProgramData\Anaconda3\conda-bld\_test_env
Build:
source activate C:\ProgramData\Anaconda3\conda-bld\_build_env
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\environ.py", line 756, in get_install_actions
actions = install_actions(prefix, index, specs, force=True)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\common\io.py", line 88, in decorated
return f(*args, **kwds)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\plan.py", line 474, in install_actions
txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\core\solve.py", line 117, in solve_for_transaction
should_retry_solve)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\core\solve.py", line 158, in solve_for_diff
force_remove, should_retry_solve)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\core\solve.py", line 275, in solve_final_state
ssc = self._add_specs(ssc)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\core\solve.py", line 555, in _add_specs
explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\resolve.py", line 523, in _get_package_pool
pool = self.get_reduced_index(specs)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\common\io.py", line 88, in decorated
return f(*args, **kwds)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\resolve.py", line 544, in get_reduced_index
explicit_specs, features = self.verify_specs(explicit_specs)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda\resolve.py", line 280, in verify_specs
raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
- m2w64_c_win-64
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Scripts\conda-build-script.py", line 10, in <module>
sys.exit(main())
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\cli\main_build.py", line 445, in main
execute(sys.argv[1:])
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\cli\main_build.py", line 436, in execute
verify=args.verify, variants=args.variants)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\api.py", line 209, in build
notest=notest, need_source_download=need_source_download, variants=variants)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\build.py", line 2343, in build_tree
notest=notest,
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\build.py", line 1334, in build
output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\render.py", line 746, in expand_outputs
for (output_dict, m) in _m.copy().get_output_metadata_set(permit_unsatisfiable_variants=False):
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\metadata.py", line 2047, in get_output_metadata_set
bypass_env_check=bypass_env_check)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\metadata.py", line 721, in finalize_outputs_pass
permit_unsatisfiable_variants=permit_unsatisfiable_variants)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\render.py", line 527, in finalize_metadata
exclude_pattern)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\render.py", line 390, in add_upstream_pins
permit_unsatisfiable_variants, exclude_pattern)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\render.py", line 378, in _read_upstream_pin_files
permit_unsatisfiable_variants=permit_unsatisfiable_variants)
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\render.py", line 154, in get_env_dependencies
channel_urls=tuple(m.config.channel_urls))
File "C:\ProgramData\Anaconda3\lib\site-packages\conda_build\environ.py", line 758, in get_install_actions
raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform win-64: {'m2w64_c_win-64'}
in my recipe/meta.yaml:
change
requirements:
build:
- {{ compiler("c") }} # [unix]
- {{ compiler('m2w64_c') }} # [win]
- {{ compiler("fortran") }} # [unix]
- {{ compiler('m2w64_fortran') }} # [win]
to:
requirements:
build:
- {{ compiler("c") }} # [unix]
- m2w64-gcc [win]
- {{ compiler("fortran") }} # [unix]
- m2w64-gcc-fortran [win]
I just started the PyTorch-Tutorial Deep Learning with PyTorch: A 60 Minute Blitz and I should add, that I haven't programmed any python (but other languages like Java) before.
Right now, my Code looks like
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
print("\n-------------------Backpropagation-------------------\n")
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
dataiter = iter(trainloader)
images, labels = dataiter.next()
def imshow(img):
img = img / 2 + 0.5
npimg = img.numpy()
plt.imshow(np.transpose(npimg, (1, 2, 0)))
imshow(torchvision.utils.make_grid(images))
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
which should be consistent with the tutorial.
If I execute this, I'll get the following error:
"C:\Program Files\Anaconda3\python.exe" C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py
-------------------Backpropagation-------------------
Files already downloaded and verified
Files already downloaded and verified
-------------------Backpropagation-------------------
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Program Files\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\MA\pytorch\deepLearningWithPytorchTutorial\trainingClassifier.py", line 23, in <module>
dataiter = iter(trainloader)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py", line 23, in <module>
dataiter = iter(trainloader)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Program Files\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
Process finished with exit code 1
I already downloaded the *.py and *.ipynb.
Running the *.ipynb with jupyter works fine (but I don't want to programm in the juniper web-interface, I prefer pyCharm) while the *.py in the console (Anaconda prompt and cmd) fails with the same error.
Does anyone know how to fix this?
(I'm using Python 3.6.5 (from Anaconda) and pyCharm, OS: Win10 64-bit)
Thanks!
Bene
Update:
If it is relevant, I just set num_workers=2 to num_workers=0 (both) and then it'll work.. .
Check out the documentation for multiprocessing: programming guidelines for windows. You should wrap all operations in functions and then call them inside an if __name__ == '__main__' clause:
# required imports
def load_datasets(...):
# Code to load the datasets with multiple workers
def train(...):
# Code to train the model
if __name__ == '__main__':
load_datasets()
train()
In short, the the idea here is to wrap the example code inside an if __name__ == '__main__' statement.
Because of different implementation of multiprocessing in Windows, you need to wrap your main code with this block:
if __name__ == '__main__':
For more info, you can check the official PyTorch Windows notes.
I'm on OS X 10.10.4 with Macports and Anaconda (condo 3.15.1).
During the process of building the robotic simulator ARGoS,
https://github.com/ilpincy/argos3
we ran into runtime 'qt' issues running the ARGoS examples.
[FATAL] Can't load library "/usr/local/lib/argos3/libargos3plugin_simulator_epuck.dylib" even after trying to add extensions for shared library (dylib) and module library (so):
/usr/local/lib/argos3/libargos3plugin_simulator_epuck.dylib: dlopen(/usr/local/lib/argos3/libargos3plugin_simulator_epuck.dylib, 1): Library not loaded: #rpath/./libQtOpenGL.4.dylib
I learned that cmake found 'qt' within Anaconda which had runtime problems.
-- Found Qt4: /Users/davidlaxer/anaconda/bin/qmake (found version "4.8.6")
Next, I installed qt4-mac:
sudo port install qt4-mac
---> Computing dependencies for qt4-mac
---> Fetching archive for qt4-mac
---> Attempting to fetch qt4-mac-4.8.7_0.darwin_14.x86_64.tbz2 from http://packages.macports.org/qt4-mac
---> Attempting to fetch qt4-mac-4.8.7_0.darwin_14.x86_64.tbz2.rmd160 from http://packages.macports.org/qt4-mac
---> Installing qt4-mac #4.8.7_0
---> Activating qt4-mac #4.8.7_0
NOTE: Qt database plugins for mysql55, postgresql91, and sqlite2 are NOT installed by this port; they are installed by qt4-mac-*-plugin instead.
---> Cleaning qt4-mac
---> Updating database of binaries
---> Scanning binaries for linking errors
---> No broken files found.
When I installed 'qt' with Macports, ipython qtconsole broke.
David-Laxers-MacBook-Pro:build_simulator davidlaxer$ ipython qtconsole
Traceback (most recent call last):
File "/Users/davidlaxer/anaconda/bin/ipython", line 6, in <module>
sys.exit(start_ipython())
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 120, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 573, in launch_instance
app.initialize(argv)
File "<string>", line 2, in initialize
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 75, in catch_config_error
return method(app, *args, **kwargs)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 321, in initialize
super(TerminalIPythonApp, self).initialize(argv)
File "<string>", line 2, in initialize
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 75, in catch_config_error
return method(app, *args, **kwargs)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/core/application.py", line 369, in initialize
self.parse_command_line(argv)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 316, in parse_command_line
return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<string>", line 2, in parse_command_line
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 75, in catch_config_error
return method(app, *args, **kwargs)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 471, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "<string>", line 2, in initialize_subcommand
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 75, in catch_config_error
return method(app, *args, **kwargs)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 402, in initialize_subcommand
subapp = import_item(subapp)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/utils/importstring.py", line 42, in import_item
module = __import__(package, fromlist=[obj])
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/qt/console/qtconsoleapp.py", line 50, in <module>
from IPython.external.qt import QtCore, QtGui
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/external/qt.py", line 23, in <module>
QtCore, QtGui, QtSvg, QT_API = load_qt(api_opts)
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/external/qt_loaders.py", line 277, in load_qt
result = loaders[api]()
File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/IPython/external/qt_loaders.py", line 184, in import_pyqt4
from PyQt4 import QtGui, QtCore, QtSvg
ImportError: dlopen(/Users/davidlaxer/anaconda/lib/python2.7/site-packages/PyQt4/QtGui.so, 2): Symbol not found: _iconv
Referenced from: /Users/davidlaxer/anaconda/lib//libxml2.2.dylib
Expected in: /opt/local/lib//libiconv.2.dylib
How is this supposed to work?