I would like to override what happens when run is pressed for certain cells in ipython notebook.
For example, I would like to be able to write an SQL query directly in a cell and define a function that processes it.
It seems it should be possible to do this as with ipython-notebook extensions. Does anyone know of a similar extension? An easy way to do this directly from ipython?
Ideally this would involve defining a custom cell type, but I would be happy to use special tags to separate the usual python code from, say, a custom SQL query cell.
I've once had the similar desire and I ended up with the following solution:
from sqlalchemy import create_engine
import pandas as pd
from IPython.core.magic import register_cell_magic
from IPython import get_ipython
con = create_engine(DB_URL)
#register_cell_magic
def sql(line, cell):
cell = cell.format(**globals())
if line.strip() != '-':
res = pd.read_sql(cell, con)
if line.strip() != '': get_ipython().user_ns[line.strip()] = res
return res
else:
con.execute(cell)
del sql
You can now write in other cells:
%sql outputvar
select * from whatever where ...
For example:
Related
I am trying to integrate interactive ipywidgets with a loop in my code that also performs other tasks (in this case, acquiring data from some hardware attached from the computer and updating live plots).
In the past, I could do this by using IPython.kernel.do_one_iteration() in my while loop: this would trigger a sync of the ipywidget changes and I would be able to retrieve them from the python widget objects. A minimal example is here:
import ipywidgets as widgets
from time import sleep
import IPython
do_one_iteration = IPython.get_ipython().kernel.do_one_iteration
w = widgets.ToggleButton()
display(w)
i=0
while True:
do_one_iteration()
print(i, w.value, end="\r")
w.decription = str(i)
sleep(0.5)
i+=1
Here, the for loop prints out the ticker integer along with the state of the widget. (In the real code, I would also acquire data, update plots, and change plot / acquisition settings dependent on the interaction with the user via the widgets.)
With ipykernel 5.3.2 and ipython 7.16.1, this worked fine: if the widget changed, calling do_one_iteration() synced the widget states to the kernel and I could retrieve it from my while loop.
After an upgrade (to 6.4.1 and 7.29.0), this no longer works. It seems that do_one_iteration() is now a coroutine: I get a warning coroutine 'Kernel.do_one_iteration' was never awaited if I use the above code.
With some help of a friend, we found a way to do this with threading an asyncio:
%gui asyncio
import asyncio
import ipywidgets as widgets
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
i=0
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
asyncio.create_task(f());
And this works (also adding a stop button, and changing to text output widget instead of printing). But to throw a spanner in the works, I need to use a library that itself uses a QT gui event loop. Some more puzzling suggests that this should be the code to make this work:
%gui qt5
import asyncio
import ipywidgets as widgets
import qasync
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
from qtpy import QtWidgets
APP = QtWidgets.QApplication.instance()
loop = qasync.QEventLoop(APP)
asyncio.set_event_loop(loop)
asyncio.create_task(f());
But with this code, the updates do not propagate, and I get the following error on the terminal running my notebook server:
[IPKernelApp] ERROR | Error in message handler
Traceback (most recent call last):
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 457, in dispatch_queue
await self.process_one()
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 440, in process_one
t, dispatch, args = await self.msg_queue.get()
RuntimeError: Task <Task pending name='Task-2'
coro=<Kernel.dispatch_queue() running at
/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py:457>
cb=[IOLoop.add_future.<locals>.<lambda>() at /Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/tornado/ioloop.py:688]>
got Future <Future pending> attached to a different loop
It seems that somehow my ipywidgets events are propagating to the wrong event loop.
And now my question is: does anybody know what is going on here?
It's hard for me to identify if this is a "bug", and if so, in which software package do things go wrong? ipykernel? Or tornado? Or ipywidgets? Or asyncio? Or maybe I'm missing something?
Any thoughts highly welcome, thanks!
Found at least a partial solution: using the nest_asyncio package allows me to now use do_one_iteration(), just by adding the following to the first code block:
import nest_asyncio
nest_asyncio.apply()
and then using await do_one_iteration() instead of calling it directly.
(see https://github.com/ipython/ipykernel/issues/825)
For my purposes, this solves my issue since I don't need asynchronous interaction with my GUI. The problem of the %gui qt5 interaction with the event loop in the asynchronous versions of the code is still a mystery though...
I am trying to read and write the Mainframe Datasets data in Python3.6. I am using Anaconda's Spyder(version 3.2.4). I am using Zosftplib inorder to import mainframe features. Below is the code snippet:
import zosftplib
Myzftp = zosftplib.Zftp("ip address-mainframe","username","password")
mf_file = open("mainframe ps file-name", 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
Mainframe PS-file name contains 1 record with data-0010021023457893.But the output I am getting is spaces in Spyder kernel.I also tried using ftplib but it didn't worked there too.I believe there's conversion required as its not a text file which I am reading.Does anyone has any suggestion on this.Please reply.Thanks
Thru FTPLIB and Zosftplib import
import zosftplib
Myzftp = zosftplib.Zftp("ip address-mainframe","username","password")
mf_file = open("mainframe ps file-name", 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
Expected result should be 0010021023457893 after file read and print.
The zosftplib package will provide you ftp access to your dataset on z/OS, meaning you can download it, but you have to open it locally. Also, you need to be aware of the encoding differences between your local machine and the z/OS environment, so you should specify the sbdataconn() argument to provide codepage translation. I was able to do what you want with code like this:
import zosftplib
Myzftp = zosftplib.Zftp('mainframe_ip',
'mainframe_userid',
'mainframe_password',
timeout=500.0,
sbdataconn='(ibm-1147,iso8859-1)')
Myzftp.download_text('mainframe_dataset_name', '/tmp/local_filename.txt')
mf_file = open('/tmp/local_filename.txt', 'r+')
ffa = mf_file.read(16);
print ("Read record is :", ffa)
mf_file.close()
I would like to save a copy of a notebook (or rename it) from within a cell of the notebook.
Preferably without too much JavaScript. Actually I guess something of this form should work
from IPython.display import display_html
display_html("script>Jupyter....???...()</script>")
Here is a solution only in Python. The notebook_path function comes from P.Toccaceli's solution on How do I get the current IPython Notebook name.
from notebook import notebookapp
import urllib
import json
import os
import ipykernel
from shutil import copy2
def notebook_path():
"""Returns the absolute path of the Notebook or None if it cannot be determined
NOTE: works only when the security is token-based or there is also no password
"""
connection_file = os.path.basename(ipykernel.get_connection_file())
kernel_id = connection_file.split('-', 1)[1].split('.')[0]
for srv in notebookapp.list_running_servers():
try:
if srv['token']=='' and not srv['password']: # No token and no password, ahem...
req = urllib.request.urlopen(srv['url']+'api/sessions')
else:
req = urllib.request.urlopen(srv['url']+'api/sessions?token='+srv['token'])
sessions = json.load(req)
for sess in sessions:
if sess['kernel']['id'] == kernel_id:
return os.path.join(srv['notebook_dir'],sess['notebook']['path'])
except:
pass # There may be stale entries in the runtime directory
return None
def copy_current_nb(new_name):
nb = notebook_path()
if nb:
new_path = os.path.join(os.path.dirname(nb), new_name+'.ipynb')
copy2(nb, new_path)
else:
print("Current notebook path cannot be determined.")
Then, simply use copy_current_nb('Save1') to create a copy named Save1.ipynb in the same directory.
I was wondering if there was a way to modify Jupyter Lab or Notebook's tab-complete functionality. For example, if I type "\alpha", and then press the tab key, I will get the UTF-8 character "α" in the cell.
Is there any way that I can do a custom "[string without spaces]" to tab-complete into some specific UTF-8 character or string?
e.g. "\implies" + tab -> "⇒"
I can see a large number of use cases for this in my programming life, so I was wondering if Jupyter (Lab or Notebook) offered the ability to modify some settings (or load in a file) that maps strings to a tabbed output. Or is there a different idea that you could use to implement this?
Thanks!
Jupyter is (based on) IPython and it's completion engine - we can make use of it.
Update: you can override latex_symbols dict to get the behavior you want:
from IPython.core.latex_symbols import latex_symbols
latex_symbols['\\implies'] = '⇒'
You should be able to build upon this example to make it work with a whole range of characters.
Initial answer/another solution: you can hook into IPython completers, though it would not work exactly as you wish: it won't replace the text, but just append a special character. Here is an example:
from IPython import get_ipython
def implies_competer(ipython, event):
return ['⇒']
ipython = get_ipython()
ipython.set_hook('complete_command', implies_competer, re_key='.*implies')
Then when you type (the space after the keyword is important):
and then press tab, you will get:
I want to put couple of cells with commands I need in almost every new notebook in every new notebook I create.
For example when I create a new notebook it should put a
%matplotlib inline
import matplotlib.pyplot as plt
in a cell by default but not execute it.
How could I set something like that up?
This will work for both terminal based IPython shell and Browser based Notebook:
Navigate to ~/.ipython/profile_default
Create a folder called startup if it’s not already there
Add a new Python file called start.py
Put your favorite imports (and custom functions may be) in this file
Launch IPython or a Jupyter Notebook and your favorite libraries will be automatically loaded every time!
Here is my sample for start.py:
Another Source
To define set of commands on default startup, you need to add the commands in the templete ipy_user_conf.py file in your ~/.ipython directory.
This module is imported during IPython startup. So you can easily do : import modules, configure extensions, change options, define magic commands, put variables and functions in the IPython namespace etc.
Here is the sample ipy_user_conf.py :
# Most of your config files and extensions will probably start
# with this import
import IPython.ipapi
ip = IPython.ipapi.get()
# You probably want to uncomment this if you did %upgrade -nolegacy
# import ipy_defaults
import os
def main():
#ip.dbg.debugmode = True
ip.dbg.debug_stack()
# uncomment if you want to get ipython -p sh behaviour
# without having to use command line switches
import ipy_profile_sh
import jobctrl
# Configure your favourite editor?
# Good idea e.g. for %edit os.path.isfile
#import ipy_editors
# Choose one of these:
#ipy_editors.scite()
#ipy_editors.scite('c:/opt/scite/scite.exe')
#ipy_editors.komodo()
#ipy_editors.idle()
# ... or many others, try 'ipy_editors??' after import to see them
# Or roll your own:
#ipy_editors.install_editor("c:/opt/jed +$line $file")
o = ip.options
# An example on how to set options
#o.autocall = 1
o.system_verbose = 0
#import_all("os sys")
#execf('~/_ipython/ns.py')
# -- prompt
# A different, more compact set of prompts from the default ones, that
# always show your current location in the filesystem:
#o.prompt_in1 = r'\C_LightBlue[\C_LightCyan\Y2\C_LightBlue]\C_Normal\n\C_Green|\#>'
#o.prompt_in2 = r'.\D: '
#o.prompt_out = r'[\#] '
# Try one of these color settings if you can't read the text easily
# autoexec is a list of IPython commands to execute on startup
#o.autoexec.append('%colors LightBG')
#o.autoexec.append('%colors NoColor')
o.autoexec.append('%colors Linux')
# some config helper functions you can use
def import_all(modules):
""" Usage: import_all("os sys") """
for m in modules.split():
ip.ex("from %s import *" % m)
def execf(fname):
""" Execute a file in user namespace """
ip.ex('execfile("%s")' % os.path.expanduser(fname))
main()
For more details, please refer the link : Customization of IPython.
I hope this is what you wanted to know.
JupyterLab
In a comment to one of the other answers, the OP pointed out the need to insert the actual code instead of having it load in the background. One way is to create a text keyboard shortcut by going to Settings -> Advanced settings editor -> JSON settings Editor and adding the following under User Preferences:
{
"shortcuts": [
{
"command": "apputils:run-first-enabled",
"selector": "body",
"keys": ["Alt I"],
"args": {
"commands": [
"console:replace-selection",
"fileeditor:replace-selection",
"notebook:replace-selection",
],
"args": {"text": "import pandas as pd\nimport altair as alt\n\n"}
}
}
]
}
This will insert the following snippet each time you press Alt + i in the notebook:
import pandas as pd
import altair as alt
# <-- Cursor placed here
More on text shortcuts in jupyterlab
IPython console
If you are interested in automatically importing commonly used libraries in the IPython console only so that they are there for interactive use, but not in the notebook to avoid issues with sharing notebooks lacking some imports, you can launch IPython like so (and set up an alias to not have to type this each time):
ipython -c "import pandas as pd; import numpy as np" -i
(This was what I was looking for when I originally found this question)