Message
Build of instance 15f7f7e3-408f-414e-a1f3-ba3f83b1b2d5 aborted: Volume aaf13a40-c6a0-4829-9992-69157c914d1e did not finish being created even after we waited 261 seconds or 61 attempts. And its status is downloading.
Code
500
Details
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1992, in _prep_block_device wait_func=self._await_block_device_map_created) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices _log_and_attach(device) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach bdm.attach(*attach_args, **attach_kwargs) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 771, in attach wait_func=wait_func, image_id=self.image_id) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 384, in _create_volume self._call_wait_func(context, wait_func, volume_api, vol['id']) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 731, in _call_wait_func {'volume_id': volume_id, 'exc': exc}) File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 721, in _call_wait_func wait_func(context, volume_id) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1611, in _await_block_device_map_created volume_status=volume_status) nova.exception.VolumeNotCreated: Volume aaf13a40-c6a0-4829-9992-69157c914d1e did not finish being created even after we waited 261 seconds or 61 attempts. And its status is downloading. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2629, in _build_resources block_device_mapping) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2010, in _prep_block_device raise exception.InvalidBDM(six.text_type(ex)) nova.exception.InvalidBDM: Volume aaf13a40-c6a0-4829-9992-69157c914d1e did not finish being created even after we waited 261 seconds or 61 attempts. And its status is downloading. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2250, in _do_build_and_run_instance filter_properties, request_spec, accel_uuids) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2462, in _build_and_run_instance bdms=block_device_mapping, tb=tb) File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2410, in _build_and_run_instance accel_uuids) as resources: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2639, in _build_resources reason=e.format_message()) nova.exception.BuildAbortException: Build of instance 15f7f7e3-408f-414e-a1f3-ba3f83b1b2d5 aborted: Volume aaf13a40-c6a0-4829-9992-69157c914d1e did not finish being created even after we waited 261 seconds or 61 attempts. And its status is downloading.
I Installed OpenStack ussuri in centos 8. everything working fine I tried to launch the instance when launching the instance it showed Build of instance 15f7f7e3-408f-414e-a1f3-ba3f83b1b2d5 aborted: Volume aaf13a40-c6a0-4829-9992-69157c914d1e did not finish being created even after we waited 261 seconds or 61 attempts. And its status is downloading.
Related
I have set up a single server with Google Auth Jupyterhub using Docker. You can find the setup scripts here - https://github.com/deepakputhraya/jupyterhub. The setup works well with multiple users able to login with separate home directories.
The problem arises when I update the Dockerfile or requirements.txt file and restart the server. Users whose accounts are created can log in but cannot access their home folders.
[ec2-user#ip-10-0-1-196 ~]$ sudo docker exec -it jupyterhub /bin/sh
# ls -lah /home
total 64K
drwxr-xr-x 16 root root 4.0K Oct 18 10:17 .
drwxr-xr-x 1 root root 4.0K Sep 4 15:46 ..
drwxr-xr-x 15 abhilash abhilash 4.0K Oct 31 10:51 abhilash
drwxr-xr-x 9 ajay ajay 4.0K Oct 10 09:11 ajay
drwxr-xr-x 8 abhilash abhilash 4.0K Sep 2 11:05 akshay
drwxr-xr-x 7 deepak deepak 4.0K Oct 4 12:20 deepak
Logs:
[I 2019-10-31 14:57:14.985 JupyterHub log:174] 200 GET /hub/spawn-pending/deepak (deepak#127.0.0.1) 38.35ms
[I 2019-10-31 14:57:17.321 JupyterHub spawner:1387] Spawning jupyterhub-singleuser --port=34405 --NotebookApp.default_url=/lab
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 660, in main
return SingleUserNotebookApp.launch_instance(argv)
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 268, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 558, in initialize
return super().initialize(argv)
File "</opt/conda/lib/python3.6/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1676, in initialize
self.init_configurables()
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1349, in init_configurables
connection_dir=self.runtime_dir,
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
[Previous line repeated 1 more time]
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/deepak'
[I 2019-10-31 14:57:18.274 JupyterHub log:174] 302 GET /hub/spawn/deepak -> /hub/spawn-pending/deepak (deepak#127.0.0.1) 1014.98ms
[I 2019-10-31 14:57:18.332 JupyterHub pages:303] deepak is pending spawn
[I 2019-10-31 14:57:18.335 JupyterHub log:174] 200 GET /hub/spawn-pending/deepak (deepak#127.0.0.1) 17.16ms
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py:697> exception=HTTPError()>
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 922, in spawn_single_user
% (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for deepak may contain details.)
[W 2019-10-31 14:57:44.492 JupyterHub user:678] deepak's server never showed up at http://127.0.0.1:34405/user/deepak/ after 30 seconds. Giving up
[E 2019-10-31 14:57:44.530 JupyterHub gen:593] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py:800> exception=TimeoutError("Server at http://127.0.0.1:34405/user/deepak/ didn't respond in 30 seconds",)> after timeout
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 589, in error_callback
future.result()
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
await spawn_future
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 654, in spawn
await self._wait_up(spawner)
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 701, in _wait_up
raise e
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 669, in _wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
timeout=timeout,
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:34405/user/deepak/ didn't respond in 30 seconds
The folder permissions for user akshay is that of abhilash. New users who signup for the first time do not have this issue. Again this only happens when there is an update to Docker image. If I were to restart the server, it neither fixes the issue nor changes the user folder permissions map for other users.
Why are the permissions getting mismatched? How can this be fixed?
I have set up a raspberry webserver running on my school lan network ,other people are connecting with arduino,sometimes when they connect i get this error:
Exception happened during processing of request from ('172.17.17.66', 49153)
Traceback (most recent call last):
File "/usr/lib/pythonz.7/SocketServer.py", line 290, in
_hand1e_request_nobloc k self.process_request(request, client_address) File
"/usr/lib/pythonz.7/SocketServer.py“, line 318, in process_request
self.finish_request(request, client_address) File
"/usr/lib/pythonz.7/SocketServer.py“, line 331, in finish_request
self.RequestHandlerClass(request, client_address, self) File
"/usr/lib/pythonz.7/SocketServer.py", line 652, in __init__ self.hand1e()
File "/usr/lib/pythonz.7/BaseHTTPServer.py“, line 340, in handle
self.handle_one_request() File "/usr/lib/pythonz.7/BaseHTTPServer.py", line
310, in handle_one request self . raw_requestline = self . rfile .
readline(65537) ‘ File "/usr/lib/pythonz.7/socket.py", line 480, in readline
data = self._sock.recv(se1f._rbufsize) error: [Errno 104] Connection reset
by peer
Can someone tell me what means? Is a problem of my server or their socket? If needed i can post my code.
The last line:
error: [Errno 104] Connection reset by peer
Means the client dropped the connection. I'd look into the Arduino code first.
For some odd reason, after last night my jupyter kernel wouldn't start so I went ahead and removed it and pip3 installed it and now everytime I run jupyter notebook Im getting the following long stack trace in the termninal
[I 14:59:43.566 NotebookApp] Writing notebook server cookie secret to /run/user/1000/jupyter/notebook_cookie_secret
[I 14:59:43.885 NotebookApp] Serving notebooks from local directory: /home/dmitri/machine_learning/airbusShipDetection
[I 14:59:43.885 NotebookApp] The Jupyter Notebook is running at:
[I 14:59:43.885 NotebookApp] http://localhost:8888/?token=b98d3bd1ec284b81559e0a865c14d6a0932e0725a99d811d
[I 14:59:43.885 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 14:59:43.885 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=b98d3bd1ec284b81559e0a865c14d6a0932e0725a99d811d
[I 14:59:44.624 NotebookApp] Accepting one-time-token-authenticated connection from 127.0.0.1
[I 14:59:48.412 NotebookApp] Kernel started: 15342e49-8fc3-402c-9964-eb8f4e2324c5
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in <module>
app.launch_new_instance()
File "/home/dmitri/.local/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "/usr/local/lib/python3.5/dist-packages/tornado/platform/asyncio.py", line 132, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.5/asyncio/base_events.py", line 340, in run_forever
raise RuntimeError('Event loop is running.')
RuntimeError: Event loop is running.
ERROR:tornado.general:Uncaught exception in ZMQStream callback
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
self.pre_handler_hook()
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 248, in pre_handler_hook
self.saved_sigint_handler = signal(SIGINT, default_int_handler)
File "/usr/lib/python3.5/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread
ERROR:tornado.general:Uncaught exception in zmqstream callback
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
self.pre_handler_hook()
File "/home/dmitri/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 248, in pre_handler_hook
self.saved_sigint_handler = signal(SIGINT, default_int_handler)
File "/usr/lib/python3.5/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread
[W 15:00:48.619 NotebookApp] Timeout waiting for kernel_info reply from 15342e49-8fc3-402c-9964-eb8f4e2324c5
[I 15:01:49.148 NotebookApp] Saving file at /u-net-model-with-submission.ipynb
^C[I 15:02:46.820 NotebookApp] interrupted
Serving notebooks from local directory: /home/dmitri/machine_learning/airbusShipDetection
1 active kernel
The Jupyter Notebook is running at:
The notebook itself says the kernel is loading, but it never finishes loading.
I had the same problem after upgrading notebook.
Upgrading tornado and ipykernel fixed it for me.
If you use the following commands in your terminal, it should work:
pip install tornado==5.1.1
and
pip install -U ipykernel
After rolling back "tornado" and updating "ipykernel" everything worked.
i tried to start 5 instance on my "litle cloud". I have one controller node (tb22) with nova-api and compute. An i have one compute node (tb23).
I get an error message in nova-schedule:
2014-07-09 13:00:23.858 ERROR nova.scheduler.filter_scheduler
[req-f699a7d3-e3de-40e4-b291-9ae972c7d8f9 admin demo] [instance:
55febf3d-1d56-4381-a6ca-b4b3b37e92e0] Error from last host: tb23 (node
tb23): [u'Traceback (most recent call last):\n', u' File
"/opt/stack/nova/nova/compute/manager.py", line 1305, in
_build_instance\n set_access_ip=set_access_ip)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 393, in
decorated_function\n return function(self, context, *args,
**kwargs)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1717, in spawn\n LOG.exception((\'Instance failed to spawn\'),
instance=instance)\n', u' File
"/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in
exit\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1714, in _spawn\n
block_device_info)\n', u' File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2262, in spawn\n
write_to_disk=True)\n', u' File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 3447, in to_xml\n
disk_info, rescue, block_device_info)\n', u' File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 3263, in
get_guest_config\n flavor)\n', u' File
"/opt/stack/nova/nova/virt/libvirt/vif.py", line 384, in get_config\n
_("Unexpected vif_type=%s") % vif_type)\n', u'NovaException: Unexpected vif_type=binding_failed\n']
Has anybody an idea what the fault is?
Thanks
This is error is caused by configuration mistake in
/etc/neutron/plugins/ml2/ml2_conf.ini
Fix
Edit /etc/neutron/plugins/ml2/ml2_conf.ini on both compute and network node
Change tunnel_type = gre to tunnel_types = gre
Restart these services in network node
service openvswitch-switch restart
service neutron-plugin-openvswitch-agent restart
service neutron-l3-agent restart
service neutron-dhcp-agent restart
service neutron-metadata-agent restart
Restart these services in compute node
service openvswitch-switch restart
service nova-compute restart
service neutron-plugin-openvswitch-agent restart
I really have a serious problem on my site and i fixed some of those.
Recently i encountered this error
We’re sorry, but there seems to be an error…
Here is the full error message:
Display traceback as text
Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 47, in call_object
Module Products.CMFFormController.FSControllerPythonScript, line 105, in __call__
Module Products.CMFFormController.Script, line 145, in __call__
Module Products.CMFCore.FSPythonScript, line 130, in __call__
Module Shared.DC.Scripts.Bindings, line 324, in __call__
Module Products.PloneHotfix20121106.python_scripts, line 63, in _patched_bindAndExec
Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 344, in _exec
Module script, line 42, in content_status_modify
<FSControllerPythonScript at /clkmis/content_status_modify used for /clkmis/news/news-archive/anti-child-labor-portal-advocates-real-time-monitoring-1>
Line 42
Module Products.CMFCore.WorkflowTool, line 244, in doActionFor
Module Products.CMFCore.WorkflowTool, line 573, in _invokeWithNotification
Module zope.event, line 23, in notify
Module zope.component.event, line 24, in dispatch
Module zope.component._api, line 136, in subscribers
Module zope.component.registry, line 321, in subscribers
Module zope.interface.adapter, line 575, in subscribers
Module plone.app.contentrules.handlers, line 189, in workflow_action
Module plone.app.contentrules.handlers, line 132, in execute_rules
Module plone.app.contentrules.handlers, line 99, in execute
Module plone.contentrules.engine.executor, line 27, in __call__
Module plone.contentrules.rule.rule, line 48, in __call__
Module collective.contentrules.mailtolocalrole.actions.mail, line 178, in __call__
Module zope.deprecation.deprecation, line 152, in deprecated_method
Module Products.CMFPlone.patches.securemailhost, line 95, in secureSend
Module Products.MailHost.MailHost, line 342, in _send
Module Products.CMFPlone.patches.sendmail, line 11, in _catch
Module zope.sendmail.mailer, line 46, in send
Module smtplib, line 239, in __init__
Module smtplib, line 295, in connect
Module smtplib, line 273, in _get_socket
Module socket, line 561, in create_connection
error: [Errno 111] Connection refused
Upon figuring out i found out that i had an ownership problem on /var/log/exim4 and /var/spool/exim4
i fixed this error thru chown -Rv Debian-exim /var/log/exim4 and chown -Rv Debian-exim /var/spool/exim4
Now i was able to send emails thru CLI, but the remaining problem is my plone, i have a content rule that sends email to users if the state of the item is changed and i got the error above, i tried removing the content rule and the changing of state was successful, i really think it's about ownership and permission error since the ownership of /var folder was accidentally altered...
Solution:
Changed the owner of /var/log/exim4/ and /var/spool/exim4 to Debian-exim
chown chown -Rv Debian-exim.Debian-exim /var/log/exim4
chown chown -Rv Debian-exim.Debian-exim /var/spool/exim4