I'm trying to test a jupyter notebook with nbval.
py.test --nbval ~/MyNotebook.ipynb
Howver, the notebook has a getpass() function in it. This blocks the test:
========================================================================================= FAILURES =========================================================================================
__________________________________________________________________________________ Untitled.ipynb::Cell 1 __________________________________________________________________________________
Notebook cell execution failed
Cell 1: Cell execution caused an exception
Input:
getpass.getpass()
Traceback:
---------------------------------------------------------------------------
StdinNotImplementedError Traceback (most recent call last)
<ipython-input-2-b4e90adc512e> in <module>
----> 1 getpass.getpass()
/opt/miniconda/lib/python3.8/site-packages/ipykernel/kernelbase.py in getpass(self, prompt, stream)
834 """
835 if not self._allow_stdin:
--> 836 raise StdinNotImplementedError(
837 "getpass was called, but this frontend does not support input requests."
838 )
StdinNotImplementedError: getpass was called, but this frontend does not support input requests.
I've tried to override the function, however it still prompts for input:
How can I override the getpass() function or provide Stdin so that it doesn't block for user input?
What you tried would seem work, here it is in the Python interpreter
>>> import getpass
>>> getpass.getpass = lambda: "abc"
>>> getpass.getpass()
'abc'
However, on Jupyter, IPython overrides getpass between blocks.
The solution is either to monkeypatch at the start of every block, or monkeypatch IPython directly:
The IPython version of getpass is defined here and here, I hacked accordingly.
import IPython
from ipykernel.kernelapp import IPythonKernel
import builtins
# Modified version of original function
def _forward_input_hacked(self, allow_stdin=False):
"""Forward raw_input and getpass to the current frontend.
via input_request (hacked)
"""
self._allow_stdin = allow_stdin
self._sys_raw_input = builtins.input
builtins.input = self.raw_input
self._save_getpass = getpass.getpass
getpass.getpass = lambda: "xyz"
#getpass.getpass = self.getpass
ipyk = IPython.Application.instance().kernel
ipyk._forward_input = _forward_input_hacked.__get__(ipyk, IPythonKernel)
import getpass
This should persist across blocks.
Note: a previous version of my answer did not answer the question properly, this monkeypatch should do the trick.
Related
I am trying to integrate interactive ipywidgets with a loop in my code that also performs other tasks (in this case, acquiring data from some hardware attached from the computer and updating live plots).
In the past, I could do this by using IPython.kernel.do_one_iteration() in my while loop: this would trigger a sync of the ipywidget changes and I would be able to retrieve them from the python widget objects. A minimal example is here:
import ipywidgets as widgets
from time import sleep
import IPython
do_one_iteration = IPython.get_ipython().kernel.do_one_iteration
w = widgets.ToggleButton()
display(w)
i=0
while True:
do_one_iteration()
print(i, w.value, end="\r")
w.decription = str(i)
sleep(0.5)
i+=1
Here, the for loop prints out the ticker integer along with the state of the widget. (In the real code, I would also acquire data, update plots, and change plot / acquisition settings dependent on the interaction with the user via the widgets.)
With ipykernel 5.3.2 and ipython 7.16.1, this worked fine: if the widget changed, calling do_one_iteration() synced the widget states to the kernel and I could retrieve it from my while loop.
After an upgrade (to 6.4.1 and 7.29.0), this no longer works. It seems that do_one_iteration() is now a coroutine: I get a warning coroutine 'Kernel.do_one_iteration' was never awaited if I use the above code.
With some help of a friend, we found a way to do this with threading an asyncio:
%gui asyncio
import asyncio
import ipywidgets as widgets
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
i=0
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
asyncio.create_task(f());
And this works (also adding a stop button, and changing to text output widget instead of printing). But to throw a spanner in the works, I need to use a library that itself uses a QT gui event loop. Some more puzzling suggests that this should be the code to make this work:
%gui qt5
import asyncio
import ipywidgets as widgets
import qasync
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
from qtpy import QtWidgets
APP = QtWidgets.QApplication.instance()
loop = qasync.QEventLoop(APP)
asyncio.set_event_loop(loop)
asyncio.create_task(f());
But with this code, the updates do not propagate, and I get the following error on the terminal running my notebook server:
[IPKernelApp] ERROR | Error in message handler
Traceback (most recent call last):
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 457, in dispatch_queue
await self.process_one()
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 440, in process_one
t, dispatch, args = await self.msg_queue.get()
RuntimeError: Task <Task pending name='Task-2'
coro=<Kernel.dispatch_queue() running at
/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py:457>
cb=[IOLoop.add_future.<locals>.<lambda>() at /Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/tornado/ioloop.py:688]>
got Future <Future pending> attached to a different loop
It seems that somehow my ipywidgets events are propagating to the wrong event loop.
And now my question is: does anybody know what is going on here?
It's hard for me to identify if this is a "bug", and if so, in which software package do things go wrong? ipykernel? Or tornado? Or ipywidgets? Or asyncio? Or maybe I'm missing something?
Any thoughts highly welcome, thanks!
Found at least a partial solution: using the nest_asyncio package allows me to now use do_one_iteration(), just by adding the following to the first code block:
import nest_asyncio
nest_asyncio.apply()
and then using await do_one_iteration() instead of calling it directly.
(see https://github.com/ipython/ipykernel/issues/825)
For my purposes, this solves my issue since I don't need asynchronous interaction with my GUI. The problem of the %gui qt5 interaction with the event loop in the asynchronous versions of the code is still a mystery though...
In my pytest script, I need to customize the pytest-HTML report for capturing the stdout at the same time writing into the console as I have user input in my automated test.
test_TripTick.py
import os
import sys
import pytest
from Process import RunProcess
from recordtype import recordtype
from pip._vendor.distlib.compat import raw_input
#pytest.fixture(scope="module")
def Process(request):
# print('\nProcess setup - module fixture')
fileDirectory = os.path.abspath(os.path.dirname(__file__))
configFilePath = os.path.join(fileDirectory, 'ATR220_Config.json')
process = RunProcess.RunProcess()
process.SetConfigVariables(configFilePath)
process.GetComPort(["list=PID_0180"])
def fin():
sys.exit()
request.addfinalizer(fin)
return process
def test_WipeOutReader(Process):
assert Process.WipeOutTheReader() == True
def test_LoadingKeysIntoMemoryMap(Process):
assert Process.LoadingKeysIntoMemoryMap() == True
def test_LoadingFW(Process): # only use bar
assert Process.LoadingFW() == True
def test_LoadingSBL(Process):
assert Process.LoadingSBL() == True
def test_CCIDReadForPaymentCards(Process):
assert Process.CCIDReadWrite('Payment Card') == True
Currently, if I run the following command from the windows command line, I get output on the console, but no captured output on the HTML report.
pytest C:\Projects\TripTickAT\test_TripTick.py -s --html=Report.html --verbose
Also, I would like to know the programmatic way of customizing the HTML report where I can update the file name, ordering test based on time of the execution and capturing the std-out.
I have tried additional flags to the pytest command. --capture sys and
-rP for Passed tests. --capture sys and -rF for failed tests
And I can see the console log as well in the html document after I click show All details button as shown in the output. I have captured only partial screen for the purpose of showing to you. But you can scroll down the output to see all the console logs. The grey area in the image below is the console output. I am not sure of the command line level flag that works regardless of failed or passed tests. But here is a temporary solution. This will print logs on command line console as well as html logs console
`
I would like to save a copy of a notebook (or rename it) from within a cell of the notebook.
Preferably without too much JavaScript. Actually I guess something of this form should work
from IPython.display import display_html
display_html("script>Jupyter....???...()</script>")
Here is a solution only in Python. The notebook_path function comes from P.Toccaceli's solution on How do I get the current IPython Notebook name.
from notebook import notebookapp
import urllib
import json
import os
import ipykernel
from shutil import copy2
def notebook_path():
"""Returns the absolute path of the Notebook or None if it cannot be determined
NOTE: works only when the security is token-based or there is also no password
"""
connection_file = os.path.basename(ipykernel.get_connection_file())
kernel_id = connection_file.split('-', 1)[1].split('.')[0]
for srv in notebookapp.list_running_servers():
try:
if srv['token']=='' and not srv['password']: # No token and no password, ahem...
req = urllib.request.urlopen(srv['url']+'api/sessions')
else:
req = urllib.request.urlopen(srv['url']+'api/sessions?token='+srv['token'])
sessions = json.load(req)
for sess in sessions:
if sess['kernel']['id'] == kernel_id:
return os.path.join(srv['notebook_dir'],sess['notebook']['path'])
except:
pass # There may be stale entries in the runtime directory
return None
def copy_current_nb(new_name):
nb = notebook_path()
if nb:
new_path = os.path.join(os.path.dirname(nb), new_name+'.ipynb')
copy2(nb, new_path)
else:
print("Current notebook path cannot be determined.")
Then, simply use copy_current_nb('Save1') to create a copy named Save1.ipynb in the same directory.
FULL CODE:
import sys
import time
counter = 0
while True:
sys.stdout.write(str(counter))
time.sleep(1)
sys.stdout.truncate()
counter += 1
PART THAT MATTERS
sys.stdout.truncate()
QUESTION(S)
Why does sys.stdout.truncate() return an error? How can I truncate sys.stdout if sys.stdout.truncate() will not work?
OPERATING SYSTEM AND MORE INFO
Operating System: Windows
Operating System Version: Windows 10
Programming Language: Python
Programming Language Version: Python 3.6
Other Details: Run from command line
sys.stdout is a a file object which corresponds to the interpreter’s standard output and does not have a truncate() method:
https://docs.python.org/3/library/sys.html#sys.stdout
It seems like you want to create status bar style output.
This should work in Python3:
import time
counter = 0
while True:
print("{}".format(counter), end="\r")
time.sleep(1)
counter += 1
See How to overwrite the previous print to stdout in python? for more info
I want to put couple of cells with commands I need in almost every new notebook in every new notebook I create.
For example when I create a new notebook it should put a
%matplotlib inline
import matplotlib.pyplot as plt
in a cell by default but not execute it.
How could I set something like that up?
This will work for both terminal based IPython shell and Browser based Notebook:
Navigate to ~/.ipython/profile_default
Create a folder called startup if it’s not already there
Add a new Python file called start.py
Put your favorite imports (and custom functions may be) in this file
Launch IPython or a Jupyter Notebook and your favorite libraries will be automatically loaded every time!
Here is my sample for start.py:
Another Source
To define set of commands on default startup, you need to add the commands in the templete ipy_user_conf.py file in your ~/.ipython directory.
This module is imported during IPython startup. So you can easily do : import modules, configure extensions, change options, define magic commands, put variables and functions in the IPython namespace etc.
Here is the sample ipy_user_conf.py :
# Most of your config files and extensions will probably start
# with this import
import IPython.ipapi
ip = IPython.ipapi.get()
# You probably want to uncomment this if you did %upgrade -nolegacy
# import ipy_defaults
import os
def main():
#ip.dbg.debugmode = True
ip.dbg.debug_stack()
# uncomment if you want to get ipython -p sh behaviour
# without having to use command line switches
import ipy_profile_sh
import jobctrl
# Configure your favourite editor?
# Good idea e.g. for %edit os.path.isfile
#import ipy_editors
# Choose one of these:
#ipy_editors.scite()
#ipy_editors.scite('c:/opt/scite/scite.exe')
#ipy_editors.komodo()
#ipy_editors.idle()
# ... or many others, try 'ipy_editors??' after import to see them
# Or roll your own:
#ipy_editors.install_editor("c:/opt/jed +$line $file")
o = ip.options
# An example on how to set options
#o.autocall = 1
o.system_verbose = 0
#import_all("os sys")
#execf('~/_ipython/ns.py')
# -- prompt
# A different, more compact set of prompts from the default ones, that
# always show your current location in the filesystem:
#o.prompt_in1 = r'\C_LightBlue[\C_LightCyan\Y2\C_LightBlue]\C_Normal\n\C_Green|\#>'
#o.prompt_in2 = r'.\D: '
#o.prompt_out = r'[\#] '
# Try one of these color settings if you can't read the text easily
# autoexec is a list of IPython commands to execute on startup
#o.autoexec.append('%colors LightBG')
#o.autoexec.append('%colors NoColor')
o.autoexec.append('%colors Linux')
# some config helper functions you can use
def import_all(modules):
""" Usage: import_all("os sys") """
for m in modules.split():
ip.ex("from %s import *" % m)
def execf(fname):
""" Execute a file in user namespace """
ip.ex('execfile("%s")' % os.path.expanduser(fname))
main()
For more details, please refer the link : Customization of IPython.
I hope this is what you wanted to know.
JupyterLab
In a comment to one of the other answers, the OP pointed out the need to insert the actual code instead of having it load in the background. One way is to create a text keyboard shortcut by going to Settings -> Advanced settings editor -> JSON settings Editor and adding the following under User Preferences:
{
"shortcuts": [
{
"command": "apputils:run-first-enabled",
"selector": "body",
"keys": ["Alt I"],
"args": {
"commands": [
"console:replace-selection",
"fileeditor:replace-selection",
"notebook:replace-selection",
],
"args": {"text": "import pandas as pd\nimport altair as alt\n\n"}
}
}
]
}
This will insert the following snippet each time you press Alt + i in the notebook:
import pandas as pd
import altair as alt
# <-- Cursor placed here
More on text shortcuts in jupyterlab
IPython console
If you are interested in automatically importing commonly used libraries in the IPython console only so that they are there for interactive use, but not in the notebook to avoid issues with sharing notebooks lacking some imports, you can launch IPython like so (and set up an alias to not have to type this each time):
ipython -c "import pandas as pd; import numpy as np" -i
(This was what I was looking for when I originally found this question)