I know that we can call one notebook from another using %run <jupyter-notebook>
But is there a way to pass in a string parameter while calling a notebook this way? Or any other way to share information from caller notebook to callee notebook?
I tried executing the following:
%run /root/notebook.ipynb "/root/abc.csv"
inside the notebook, I print sys.argv and I see this on console:
['/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py', '-f', '/root/.local/share/jupyter/runtime/kernel-fa157ce3-83be-4e70-bf95-fe7f83530d4d.json']
I was expecting to see my argument /root/abc.csv in the output for sys.argv according to the documentation. I may be misunderstanding something.
I ended up using papermill python library to do this.
Syntax:
import papermill as pm
pm.execute_notebook('input.ipynb', 'output.ipynb', {'param1': 'value1', 'param2': ['value2', 'value3']})
Related
I've been trying to create custom magic for a Glue Studio Notebook, like the following example (taken from here)
I've adding the Ipython module by running the glue magic
%additional_python_modules IPython
And running this from a cell:
from IPython.core.magic import (register_line_magic,
register_cell_magic)
#register_line_magic
def hello(line):
if line == 'french':
print("Salut tout le monde!")
else:
print("Hello world!")
However, I get this error:
AttributeError: 'NoneType' object has no attribute 'register_magic_function'
Thanks.
I think is related to the fact that if i do
from IPython import get_ipython
get_ipython()
get_ipython() returns None.
This means that this is not running inside IPython, but what then? How can I add a custom magic? My goal is to have a magic to run sql queries in a postgresql database connected using a glue connection.
I am using ipython 6.4.0 on ubuntu 20.04 and using jupyter kernelspec list , I found, there are 2 kernels :
practice_applied_ai
python3
When I open any .ipynb file, it directly opens in "python3" but I want to open it in "practice_applied_ai" because I created virtual environment practice_applied_ai and only in this kernel I can import Tensorflow 2.2.0 for my work.
My question is, Is there any way to change my default kernel without removing any kernel ?
jupyter notebook --generate-config
open the generated config file change
change this line to your desired kernel
#c.MultiKernelManager.default_kernel_name = 'python3'
like
c.MultiKernelManager.default_kernel_name = 'py38'
See this answer on GitHub.
As explained there:
the default kernel name is rarely used. It really only comes into play when a request is received to start a kernel and the kernel name is not specified in the request payload. Since both Notebook and Lab UIs essentially require the user to select a kernel (for new notebooks), it doesn't really come into play.
Put c.MappingKernelManager.default_kernel_name='newDefault' in config file.
To confirm the default is in place, hit the kernelspecs REST API of your running notebook server (e.g., http://localhost:8888/api/kernelspecs) and you should see the default kernel name as the first entry in the returned payload.
Yes this is possible via the .ipynb file itself.
Set the following variables in the metadata, specifically the name which identifies the kernel
"metadata": {
"kernelspec": {
"display_name": "Python 3 (PyTorch 1.6 Python 3.6 CPU Optimized)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/pytorch-1.6-cpu-py36-ubuntu16.04-v1"
},
"language_info": {
"codemirror_mode": {
Not sure how to change the default kernel, but you can change the kernel being used in a notebook, after opening the notebook, as explained in this site.
Open the notebook. Then navigate to Kernel -> Change Kernel and select the kernel you want to use.
While trying to execute a timeit command on the command line using the python command line interface I accidentally put .function() on the outside of the command like so:
$ python3 -m timeit '<code>'.function()
Rather than the timeit command being executed, I was prompted as such:
function>
Thinking I had entered the python repl I tried to quit with q. Yes, I'm aware quit() is the correct way to do this. Having returned to the command line, I noticed the error and corrected it like so:
$ python3 -m timeit `<code>.function()`
I expected this code to execute correctly, but instead I received the following error:
python3:7: command not found: q
After discussing it with some colleagues, it was suggested that I check which python was being used:
$ which python3
python3 () {
q
}
This was not what I was expecting! Normally the result would be /usr/local/bin/python3. Through some trial and error I was able to determine that the minimal case to reproduce this is:
$ python3 ()
function> q
$
Now that the context is out of the way, I have two questions about the behaviour I witnessed:
1. What exactly does python3 () do?
2. How do I return execution to its original state in the same terminal window? I'm aware I can open a new terminal window and the original state exists in that window.
The syntax foo () is used in POSIX-compliant shells (such as bash, dash, and zsh) to define a function. Your entire snippet defines a function called python3 and executes the command q when it's ran. You can bypass shell functions and aliases using the command command: command -p python3 myfile.py
To remove the function from the current shell process, you can use unset -f python3. If it keeps coming back after starting new shells, then it's likely defined in one of you shell initialization files.
I am attempting to use MyPy with modules that use ruamel.yaml and Mypy cannot find ruamel.yaml even though Python has no problem finding it. I am puzzled because I can't find a module called YAML.py or class called YAML either, even though these statements work in Python:
from ruamel.yaml import YAML
yaml = YAML()
x = yaml.load()
What do I need to do to get MyPy to recognize ruamel.yaml?
A workaround is to run without the incremental logic of mypy:
python -m mypy --no-incremental myfile.py
Background
There is a known issue in mypy, see here.
In summary:
Something is not working with the incremental logic of mypy when it is encountering ruamel.
When you run it once, all goes ok. This is the command:
python -m mypy myfile.py
Then, when you run it again, you get an error:
error: Skipping analyzing 'ruamel': found module but no type hints or library stubs [import]
Then, when you run it again, it all goes ok
etc.
You should not be looking for a file YAML.py. The YAML in
yaml = YAML()
is a class that is defined in ruamel/yaml/main.py and that gets imported into ruamel/yaml/__init__.py (both under site-packages). That is why you do:
from ruamel.yaml import YAML
(the alternative would be that there is a file yaml.py under the directory ruamel, but the loader/dumper is a bit too much to put in one file).
What might work if the above knowledge doesn't help you resolve things, is explicitly set the global flag mypy_path or the environment variable MYPYPATH. This has to include the directory in which the directory ruamel is located.
( I could not find it mentioned in the documentation, but from the source ( mypy/build.py:mypy_path() ) you can see that this is supposed to be a string that gets split on os.pathsep (which is the colon (:) on my Linux based system))
I have the same issue.
Even after setting MYPYPATH=./.venv/lib/python3.7/site-packages
A temporary 'solution' is ignoring the missing import exception
mypy --ignore-missing-imports
I cannot run a script in Eclipse that works perfectly in a terminal.
It seems that Eclipse console cannot support some functions. I am looking for a workaround to be able to debug the script using Pydev.
Is it possible to set PyDev to use for example /usr/bin/gnome-terminal instead of the Eclipse native console ?
Otherwise it there a way to define a wrapper as a python interpreter for PyDev that will launch an terminal external to Eclipse (I've tried but failed on that).
Thank you
Nga
Right now, curses-based apps really don't run well inside Eclipse/PyDev, so, you must really launch it externally. To debug you can use the remote debugger: http://pydev.org/manual_adv_remote_debugger.html
If you use Aptana Studio, there's a terminal view which should emulate a terminal better... try running python (i.e.: running your program) from inside that view. If it does work properly there, maybe I could check a way to better integrate there and launch directly in that view.
Thank you for your reply. I have finally defined kind of a wrapper as a bash script calling python in a xterm. Pydev is checking some configuration by calling eclipse/plugins/org.python.pydev_2.4.0.2012020116/PySrc/interpreterInfo.py so the script first echo the format expected by PyDev. here is the script "
#!/bin/bash
# dummy return for Eclipse Pydev - respect interpreter info format
echo "EXECUTABLE:/home/user/python_custom/python_xterm|
|/home/user/eclipse/plugins/org.python.pydev_2.4.0.2012020116/PySrc
|/usr/lib/python2.5
|/usr/lib/python2.5/plat-linux2
|/usr/lib/python2.5/lib-tk
|/usr/lib/python2.5/lib-dynload
|/usr/local/lib/python2.5/site-packages
|/usr/lib/python2.5/site-packages
|/usr/lib/python2.5/site-packages/Numeric
|/usr/lib/python2.5/site-packages/PIL
|/usr/lib/python2.5/site-packages/gst-0.10
|/var/lib/python-support/python2.5
|/usr/lib/python2.5/site-packages/gtk-2.0
|/var/lib/python-support/python2.5/gtk-2.0
|/var/lib/python-support/python2.5/HTMLgen
|/var/lib/python-support/python2.5/pyinotify
|/usr/lib/python2.5/site-packages/wx-2.6-gtk2-unicode
|/usr/lib/site-python
#
$
|__builtin__
|__main__
|_ast
|_codecs
|_sre
|_symtable
|_types
|errno
|exceptions
|gc
|imp
|marshal
|posix
|pwd
|signal
|sys
|thread
|xxsubtype
|zipimport
"
# activate scrollbar -sb with 6000 lines
# allow logging -l with filename log_$NOW
xterm -g 150x100+0+0 -sb -sl 6000 -si -hold -e "python $*"
that does the job, and I can use Pydev and its debugger