How to launch an R session with rpy2 that will use virtual framebuffer X11 server - r

I'm running R (3.2.2) on a Debian Jessie VPS thanks to rpy2 (2.6.1) through Python (2.7.10).
I have a Flask (0.10.1) application launching R via rpy2 and then loading R package devEMF (2.0.0) to plot various graphics. The aim is to save vector images created thanks to devEMF.
My sample code is:
robjects.r(u'''
suppressPackageStartupMessages(library(qgraph))
suppressPackageStartupMessages(library(devEMF))
emf(file = "''' + pathToSaveGraphEmf + '''", bg = "white", width = 6.25, height = 5.21)
attach(mtcars)
plot(wt,mpg)
dev.off()
''')
And I get the following error message:
/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py:106: UserWarning: Error in axis(side = side, at = at, labels = labels, ...) :
Can't open connection to X server to read font metric information.
res = super(Function, self).__call__(*new_args, **new_kwargs)
/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py:106: UserWarning: In addition:
res = super(Function, self).__call__(*new_args, **new_kwargs)
/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py:106: UserWarning: Warning message:
res = super(Function, self).__call__(*new_args, **new_kwargs)
/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py:106: UserWarning: no DISPLAY variable so Tk is not available
res = super(Function, self).__call__(*new_args, **new_kwargs)
[2016-01-21 20:03:18 +0000] [21430] [ERROR] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 130, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 171, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py", line 110, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py", line 110, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python2.7/dist-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "/routines/routine2.py", line 124, in keyDriverAnalysis
myCorrelationMatrix, rowName, colName = r2.getCorrelationGraph(myCorrelationMethod, myPreparedData, myGroupUserInput, myDependentVariable, myIndependentVariable, myPathToSaveGraph)
File "../graph.py", line 194, in getCorrelationGraph
''')
File "/usr/local/lib/python2.7/dist-packages/rpy2/robjects/__init__.py", line 317, in __call__
res = self.eval(p)
File "/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
RRuntimeError: Error in axis(side = side, at = at, labels = labels, ...) :
Can't open connection to X server to read font metric information.
devEMF needs X11 and Xft (line 35), when not running on OSX or Windows, so I installed it. But now it is unable to connect to the X server. Certainly because the VPS has no GUI feature.
Hopefully I found this very helping SO question, and I understood that I should use virtual framebuffer X11 server.
I tried the advice from the SO question in a command line R session, and it worked ! (Now it outputs an empty file, but it was fully working two minutes ago. (I had downloaded the file, inserted is in an .xls file and my graph was openable and ungroupable.))
But now I have to make it work within rpy2.
I tried the second answer in the previously cited SO question, but it doesn't work because it gives a trick when X11 is problematic when loading but not needed then.
I think where I am now is: how do I launch an R session thanks to rpy2 that will use virtual framebuffer X11 server ?
EDIT0:
I think I found something useful here. It explains how to configure a Virtual Frame Buffer on headless unix servers.
EDIT1:
Eventually, what I have done:
Launch xvfb thanks to the command from the main environment, and bind it to the desired screen (1) (I don't reminder where I found this precise line, but well it can be analyzed thanks to the xvfb documentation):
Xvfb :1 -screen 0 1024x768x16 &
Attach the tmux session, and export the DISPLAY environment variable linked to the correct screen (1):
export DISPLAY=:1
And it works !
Important note: the environment variable should be set inside the session.
Also: I don't have the "launch at boot time" problem mentioned in the link from EDIT0, probably because I am on a VPS (the xvfb-run folder is still in tmp and it seems to run because I can plot and save in .emf ;-) )

Related

problem in Change Directory in colab for google drive

I want download video from YouTube with youtube_dl with Colab and save it in google drive. I make a directory with the name of video title and save video in that folder. Then I use this code:
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
URL = "https://www.youtube.com/watch?v=QTPP-iaF7BY&t=1955s"
!pip install youtube_dl
import youtube_dl
with youtube_dl.YoutubeDL({"ignoreerrors": True, "quiet": True}) as ydl:
playlist_dict = ydl.extract_info(URL, download=False)
print('\n', playlist_dict['title'], '\n')
import os
new_folder = playlist_dict['title']
path = f"//content//drive//MyDrive//{new_folder}//".replace("'"," ").replace(".","-").replace(":","-")
os.makedirs(path, exist_ok=True)
print('\n', path, '\n')
%cd {path}
But for the URL that I specified in the above code it get this error:
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Transport endpoint is not connected
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Transport endpoint is not connected
The folder you are executing pip from can no longer be found.
ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.
Pillai "Hoeffding's Inequality"
//content//drive//MyDrive//Pillai "Hoeffding's Inequality"//
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-13-dd9eae6c92da>", line 20, in <module>
get_ipython().magic('cd {path}')
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2160, in magic
return self.run_line_magic(magic_name, magic_arg_s)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2081, in run_line_magic
result = fn(*args,**kwargs)
File "<decorator-gen-84>", line 2, in cd
File "/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py", line 188, in <lambda>
call = lambda f, *a, **k: f(*a, **k)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/magics/osm.py", line 288, in cd
oldcwd = py3compat.getcwd()
OSError: [Errno 107] Transport endpoint is not connected
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 1823, in showtraceback
stb = value._render_traceback_()
AttributeError: 'OSError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 1132, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 313, in wrapped
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 358, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/usr/lib/python3.7/inspect.py", line 1502, in getinnerframes
frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
File "/usr/lib/python3.7/inspect.py", line 1460, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File "/usr/lib/python3.7/inspect.py", line 696, in getsourcefile
if getattr(getmodule(object, filename), '__loader__', None) is not None:
File "/usr/lib/python3.7/inspect.py", line 725, in getmodule
file = getabsfile(object, _filename)
File "/usr/lib/python3.7/inspect.py", line 709, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File "/usr/lib/python3.7/posixpath.py", line 383, in abspath
cwd = os.getcwd()
OSError: [Errno 107] Transport endpoint is not connected
with other URL in YouTube I haven't this problem and it downloads and saves correctly in Google Drive.
EDIT
With changing %cd {path} to os.chdir(path) the problem solved. But
I don't understand why %cd {path} work for some and don't work
for others.

Unable to export jupyter notebook to any other format using nbconvert

I am using Python 3.8.5 on Windows 10. I installed Jupyterlab and it works correctly. However nbconvert does not. When I try to export notebook to any format, I get the following error in the terminal:
[W 16:14:06.733 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20200928161400 (::1) 8.46ms referer=http://localhost:8889/notebooks/tester1/testing_epr_py.ipynb
[I 16:14:07.066 NotebookApp] Kernel started: 1e1cd4f1-55e7-4583-b343-98ed9670e8f9, name: python3
[E 16:14:10.773 NotebookApp] Uncaught exception GET /nbconvert/html/tester1/testing_epr_py.ipynb?download=true (::1)
HTTPServerRequest(protocol='http', host='localhost:8889', method='GET', uri='/nbconvert/html/tester1/testing_epr_py.ipynb?download=true', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\traitlets\traitlets.py", line 535, in get
value = obj._trait_values[self.name]
KeyError: 'template_paths'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\tornado\web.py", line 1703, in _execute
result = await result
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\tornado\gen.py", line 209, in wrapper
yielded = next(result)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\notebook\nbconvert\handlers.py", line 93, in get
exporter = get_exporter(format, config=self.config, log=self.log)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\notebook\nbconvert\handlers.py", line 67, in get_exporter
Exporter = get_exporter(format)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\base.py", line 102, in get_exporter
if getattr(exporter(config=config), 'enabled', True):
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\templateexporter.py", line 328, in __init__
super().__init__(config=config, **kw)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\exporter.py", line 114, in __init__
self._init_preprocessors()
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\templateexporter.py", line 494, in _init_preprocessors
conf = self._get_conf()
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\templateexporter.py", line 510, in _get_conf
for path in map(Path, self.template_paths):
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\traitlets\traitlets.py", line 575, in __get__
return self.get(obj, cls)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\traitlets\traitlets.py", line 538, in get
default = obj.trait_defaults(self.name)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\traitlets\traitlets.py", line 1577, in trait_defaults
return self._get_trait_default_generator(names[0])(self)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\traitlets\traitlets.py", line 975, in __call__
return self.func(*args, **kwargs)
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\templateexporter.py", line 521, in _template_paths
template_names = self.get_template_names()
File "C:\Users\draco.malfoy\AppData\Roaming\Python\Python38\site-packages\nbconvert\exporters\templateexporter.py", line 582, in get_template_names
raise ValueError('No template sub-directory with name %r found in the following paths:\n\t%s' % (base_template, paths))
ValueError: No template sub-directory with name 'lab' found in the following paths:
C:\Users\draco.malfoy\AppData\Roaming\jupyter
C:\Program Files\Python38\share\jupyter
C:\ProgramData\jupyter
I also tried to use nbconvert through command line but got the same error. nbconvert version is 6.0.6. Any help is greatly appreciated.
Copying files from "C:\Users\{username}\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\share\jupyter"
to "C:\ProgramData\jupyter" helped me solve this issue, kindly note that you will have to install Pandoc and MiKTex on Windows if you want to convert notebooks.

Loading saved NER transformers model causes AttributeError?

I have trained and saved some NER models using
torch.save(model)
I need to load these model files (extension .pt) for evaluation using
torch.load('PATH_TO_MODEL.pt')
And I get the following error: 'BertConfig' object has no attribute 'return_dict'
For the same, I updated my transformer package to the latest one, but the error persists.
This is the stack trace:
Traceback (most recent call last):
File "/home/systematicReviews/train_mtl_3.py", line 523, in <module>
test_loss, test_cr, test_cr_fine = evaluate_i(test_model, optimizer, scheduler, validation_dataloader, args, device)
File "/home/systematicReviews/train_mtl_3.py", line 180, in evaluate_i
e_loss_coarse, e_output, e_labels, e_loss_fine, e_f_output, e_f_labels, mask, e_cumulative_loss = defModel(args, e_input_ids, attention_mask=e_input_mask, P_labels=e_labels, P_f_labels=e_f_labels)
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/systematicReviews/models/mtl/model.py", line 122, in forward
attention_mask = attention_mask
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/transformers/modeling_bert.py", line 784, in forward
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
File "/home/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/transformers/configuration_utils.py", line 219, in use_return_dict
return self.return_dict and not self.torchscript
AttributeError: 'BertConfig' object has no attribute 'return_dict'
Here is some more information about my system:
- `transformers` version: 3.1.0
- Platform: Linux-4.4.0-186-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
It worked pretty fine until now, but suddenly this bug appears. Any help or hint is appreciated.
Try to save your model with model.save_pretrained(output_dir). Then you can load your model with model = *.from_pretrained(output_dir) where * is the model class (e.g. BertForTokenClassification).
To save model dictionary rather than an entire model is slightly different. Instead of torch.save(model) use torch.save('path_to_the_model/model.pth') and load using torch.load('path_to_the_model/model.pth').

Error in Jupyter Notebook: What could be the reason?

Recently I updated R n Jupyter Lab with homebrew.
Today, I am surprised that R failed to run on ipython notebook with error below:
Error Starting Kernel
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/handlers.py", line 72, in post
type=mtype))
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/opt/anaconda3/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
km.start_kernel(**kwargs)
File "/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 259, in start_kernel
**kw)
File "/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 204, in _launch_kernel
return launch_kernel(kernel_cmd, **kw)
File "/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/launcher.py", line 138, in launch_kernel
proc = Popen(cmd, **kwargs)
File "/opt/anaconda3/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/opt/anaconda3/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/Cellar/r/3.5.3/lib/R/bin/R': '/usr/local/Cellar/r/3.5.3/lib/R/bin/R'
Appreciate your advice.
The interesting part of the error is.
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/Cellar/r/3.5.3/lib/R/bin/R': '/usr/local/Cellar/r/3.5.3/lib/R/bin/R'
I guess you Jupyter kernel is referencing the old installation of R that you have updated.
The first thing to do is to check what is defined in your R kernel.
Check kernel
# list all kernel
jupyter kernelspec list
#Available kernels:
# ir /opt/conda/share/jupyter/kernels/ir
# ...
You should have an ir kernel.
Check the content of the kernel definition, for example:
$ cat /opt/conda/share/jupyter/kernels/ir/kernel.json
# {"argv": ["R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
# "display_name":"R",
# "language":"R"
# }
If the command in argv is incorrect you can remove an reinstall the kernel.
Remove and reinstall the kernel
See the IRkernel installation documentation for more information.
$ jupyter kernelspec remove ir
$ R
# Install IRkernel
> install.packages('IRkernel')
# Or update it if already installed
> update.packages('IRkernel')
# Install the kernel
> IRkernel::installspec()
# Or alternatively if you want to install it system-wide
> IRkernel::installspec(user = FALSE)

PyTorch Tutorial Error Training a Classifier

I just started the PyTorch-Tutorial Deep Learning with PyTorch: A 60 Minute Blitz and I should add, that I haven't programmed any python (but other languages like Java) before.
Right now, my Code looks like
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
print("\n-------------------Backpropagation-------------------\n")
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
dataiter = iter(trainloader)
images, labels = dataiter.next()
def imshow(img):
img = img / 2 + 0.5
npimg = img.numpy()
plt.imshow(np.transpose(npimg, (1, 2, 0)))
imshow(torchvision.utils.make_grid(images))
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
which should be consistent with the tutorial.
If I execute this, I'll get the following error:
"C:\Program Files\Anaconda3\python.exe" C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py
-------------------Backpropagation-------------------
Files already downloaded and verified
Files already downloaded and verified
-------------------Backpropagation-------------------
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Program Files\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\MA\pytorch\deepLearningWithPytorchTutorial\trainingClassifier.py", line 23, in <module>
dataiter = iter(trainloader)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py", line 23, in <module>
dataiter = iter(trainloader)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Program Files\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
Process finished with exit code 1
I already downloaded the *.py and *.ipynb.
Running the *.ipynb with jupyter works fine (but I don't want to programm in the juniper web-interface, I prefer pyCharm) while the *.py in the console (Anaconda prompt and cmd) fails with the same error.
Does anyone know how to fix this?
(I'm using Python 3.6.5 (from Anaconda) and pyCharm, OS: Win10 64-bit)
Thanks!
Bene
Update:
If it is relevant, I just set num_workers=2 to num_workers=0 (both) and then it'll work.. .
Check out the documentation for multiprocessing: programming guidelines for windows. You should wrap all operations in functions and then call them inside an if __name__ == '__main__' clause:
# required imports
def load_datasets(...):
# Code to load the datasets with multiple workers
def train(...):
# Code to train the model
if __name__ == '__main__':
load_datasets()
train()
In short, the the idea here is to wrap the example code inside an if __name__ == '__main__' statement.
Because of different implementation of multiprocessing in Windows, you need to wrap your main code with this block:
if __name__ == '__main__':
For more info, you can check the official PyTorch Windows notes.

Resources