bokeh development workflow with interaction - bokeh

I'm developing a visualization using
% bokeh serve --show myapp.py
The problem is that when I change myapp.py I have to kill the above command and restart it. Is there a better workflow for this kind of development?
Thanks!

Not yet. As of Bokeh 0.11.1 (and soon to be 0.12 too), this is a planned, but still open, feature request There are only a few folks working a huge pile of work for Bokeh. New contributors can help accelerate new features and fixes. If you're able to help, please reach out to us on the project gitter channel.

I was unable to understand enough of the bokeh internals to make this work nicer, but here is a hacky script that does what I want anyway.
# bokeh_watcher.py
#
# Watches specific files in directory and restarts bokeh server upon change.
#
# % python bokeh_watcher filename.py
#
# Note that you stil have to navigate your browser to localhost:5006/filename
# to see your Bokeh visualization and you might have to refresh the browser.
import sys
import time
import logging
from watchdog.observers import Observer
from watchdog.events import RegexMatchingEventHandler
from bokeh.command.bootstrap import main
import multiprocessing
import os
JOBS = []
FILE = []
def spawn_bokeh(args):
main(args)
class BokehHandler(RegexMatchingEventHandler):
'''
kills and restarts bokeh server upon filechange.
'''
def on_modified(self, event):
super(BokehHandler, self).on_modified(event)
what = 'directory' if event.is_directory else 'file'
logging.info("Modified %s: %s"% (what, event.src_path))
p=JOBS.pop()
p.terminate()
time.sleep(1) # time to die
logging.info('terminated')
logging.info('initiating restart')
p = multiprocessing.Process(target=spawn_bokeh,
args=(self.args,))
p.start()
JOBS.append(p)
if __name__ == "__main__":
here = os.path.realpath(__file__)
fullpathname=os.path.dirname(here)+os.sep+sys.argv[1]
# local logger
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
filemod_handler = BokehHandler(['.*%s'%(sys.argv[1])])
filemod_handler.args = ['','serve',fullpathname, '--log-level','info']
# fire up bokeh server
p = multiprocessing.Process(target=spawn_bokeh,args=(filemod_handler.args,))
p.start()
# store object in global for later
JOBS.append(p)
observer = Observer()
observer.schedule(filemod_handler, '.', recursive=False)
observer.start()
try:
while True:
time.sleep(3)
except KeyboardInterrupt:
observer.stop()
observer.join()

Related

PyCharm loadUi findChild - unresolved attribute reference [duplicate]

Let's say I have a ui file created in Qt Designer that I want to load dynamically to then manipulate the widgets, such as:
example.py:
from PyQt5 import QtWidgets, uic
class MyWidget(QtWidgets.QWidget):
def __init__(self, parent=None):
super(MyWidget, self).__init__(parent)
uic.loadUi('example.ui', self)
# No code completion here for self.myPushButton:
self.myPushButton.clicked.connect(self.handleButtonClick)
self.show()
Is there a standard / convenient way of enabling code completion for the widgets loaded this way in PyCharm (2017.1.4)?
At the moment I am using this (written in the constructor after the ui file is loaded):
self.myPushButton = self.myPushButton # type: QtWidgets.QPushButton
# Code completion for myPushButton works at this point
I also thought of this, but it does not seem to do the trick:
assert isinstance(self.myPushButton, QtWidgets.QPushButton)
# PyCharm does not even recognise myPushButton as an attribute of self at this point
Finally, I also thought of using python stubs, such as:
example.pyi:
class MyWidget(QtWidgets.QWidget):
def __init__(self):
self.myPushButton: QtWidgets.QPushButton = ...
However, myPushButton is properly recognised in code outside example.py but not in code inside example.py itself, which is kind of the opposite of what I wanted.
I am also considering taking my first approach but with all those lines put in a private method that will never get called, such as:
example.py:
from PyQt5 import QtWidgets, uic
class MyWidget(QtWidgets.QWidget):
def __init__(self, parent=None):
super(MyWidget, self).__init__(parent)
uic.loadUi('example.ui', self)
# Code completion now works here for self.myPushButton:
self.myPushButton.clicked.connect(self.handleButtonClick)
self.show()
def __my_private_method_never_called():
self.myPushButton = self.myPushButton # type: QtWidgets.QPushButton
# Or even this (it should have the same effect if this
# function is never called, plus it is less verbose):
self.myPushButton = QtWidgets.QPushButton()
# If I want to make sure that this is never called
# could raise an error at some point:
raise YouShouldNotHaveCalledThisError()
This seems to work fine, and it also allows me to group all my type hinting code together, isolated from the rest. I could even make some script to write all those lines for me by parsing the ui files. I am just wondering if people reading my code would find this approach very unorthodox, even if I comment clearly why am I writing a technically useless private function.
If anybody is interested, I made the script I mentioned to parse the .ui files and generate stub code ready to be copied to my class:
ui_stub_generator.py:
from __future__ import print_function
import os
import sys
import xml.etree.ElementTree
def generate_stubs(file):
root = xml.etree.ElementTree.parse(file).getroot()
print('Stub for file: ' + os.path.basename(file))
print()
print(' def __stubs(self):')
print(' """ This just enables code completion. It should never be called """')
for widget in root.findall('.//widget'):
name = widget.get('name')
if len(name) > 3 and name[:2] == 'ui' and name[2].isupper():
cls = widget.get('class')
print(' self.{} = QtWidgets.{}()'.format(
name, cls
))
print(' raise AssertionError("This should never be called")')
print()
def main():
for file in sys.argv[1:]:
generate_stubs(file)
if __name__ == '__main__':
main()
This only parses widgets whose names start with 'ui' followed by an uppercase letter, such as 'uiMyWidget', which is the naming convention that I typically follow in the Qt Designer. By doing this, the widgets with names automatically generated by the Qt Designer are ignored (if I cared about these, I would have given them a proper name). It should be straightforward to update this for any other naming conventions, or other type of objects, such as actions.
For convenience, I have set this up as an external tool in PyCharm as well; see screenshot here (change the paths as appropriate). That way, I only have to right-click my ui file in the project window, then External Tools -> Stub Generator for Qt UI Files, and I get the following output in the Run window ready to be copied:
C:\ProgramData\Anaconda3\python.exe D:\MyProject\bin\ui_stub_generator.py D:\MyProject\my_ui_file.ui
Stub for file: my_ui_file.ui
def __stubs(self):
""" This just enables code completion. It should never be called """
self.uiNameLabel = QtWidgets.QLabel()
self.uiOpenButton = QtWidgets.QPushButton()
self.uiSplitter = QtWidgets.QSplitter()
self.uiMyCombo = QtWidgets.QComboBox()
self.uiDeleteButton = QtWidgets.QPushButton()
raise AssertionError("This should never be called")
Process finished with exit code 0

problem with event loops, gui qt5, and ipywidgets in a jupyter notebook

I am trying to integrate interactive ipywidgets with a loop in my code that also performs other tasks (in this case, acquiring data from some hardware attached from the computer and updating live plots).
In the past, I could do this by using IPython.kernel.do_one_iteration() in my while loop: this would trigger a sync of the ipywidget changes and I would be able to retrieve them from the python widget objects. A minimal example is here:
import ipywidgets as widgets
from time import sleep
import IPython
do_one_iteration = IPython.get_ipython().kernel.do_one_iteration
w = widgets.ToggleButton()
display(w)
i=0
while True:
do_one_iteration()
print(i, w.value, end="\r")
w.decription = str(i)
sleep(0.5)
i+=1
Here, the for loop prints out the ticker integer along with the state of the widget. (In the real code, I would also acquire data, update plots, and change plot / acquisition settings dependent on the interaction with the user via the widgets.)
With ipykernel 5.3.2 and ipython 7.16.1, this worked fine: if the widget changed, calling do_one_iteration() synced the widget states to the kernel and I could retrieve it from my while loop.
After an upgrade (to 6.4.1 and 7.29.0), this no longer works. It seems that do_one_iteration() is now a coroutine: I get a warning coroutine 'Kernel.do_one_iteration' was never awaited if I use the above code.
With some help of a friend, we found a way to do this with threading an asyncio:
%gui asyncio
import asyncio
import ipywidgets as widgets
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
i=0
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
asyncio.create_task(f());
And this works (also adding a stop button, and changing to text output widget instead of printing). But to throw a spanner in the works, I need to use a library that itself uses a QT gui event loop. Some more puzzling suggests that this should be the code to make this work:
%gui qt5
import asyncio
import ipywidgets as widgets
import qasync
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
from qtpy import QtWidgets
APP = QtWidgets.QApplication.instance()
loop = qasync.QEventLoop(APP)
asyncio.set_event_loop(loop)
asyncio.create_task(f());
But with this code, the updates do not propagate, and I get the following error on the terminal running my notebook server:
[IPKernelApp] ERROR | Error in message handler
Traceback (most recent call last):
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 457, in dispatch_queue
await self.process_one()
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 440, in process_one
t, dispatch, args = await self.msg_queue.get()
RuntimeError: Task <Task pending name='Task-2'
coro=<Kernel.dispatch_queue() running at
/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py:457>
cb=[IOLoop.add_future.<locals>.<lambda>() at /Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/tornado/ioloop.py:688]>
got Future <Future pending> attached to a different loop
It seems that somehow my ipywidgets events are propagating to the wrong event loop.
And now my question is: does anybody know what is going on here?
It's hard for me to identify if this is a "bug", and if so, in which software package do things go wrong? ipykernel? Or tornado? Or ipywidgets? Or asyncio? Or maybe I'm missing something?
Any thoughts highly welcome, thanks!
Found at least a partial solution: using the nest_asyncio package allows me to now use do_one_iteration(), just by adding the following to the first code block:
import nest_asyncio
nest_asyncio.apply()
and then using await do_one_iteration() instead of calling it directly.
(see https://github.com/ipython/ipykernel/issues/825)
For my purposes, this solves my issue since I don't need asynchronous interaction with my GUI. The problem of the %gui qt5 interaction with the event loop in the asynchronous versions of the code is still a mystery though...

How to capture the log details on pytest-html as well as writing in to Console?

In my pytest script, I need to customize the pytest-HTML report for capturing the stdout at the same time writing into the console as I have user input in my automated test.
test_TripTick.py
import os
import sys
import pytest
from Process import RunProcess
from recordtype import recordtype
from pip._vendor.distlib.compat import raw_input
#pytest.fixture(scope="module")
def Process(request):
# print('\nProcess setup - module fixture')
fileDirectory = os.path.abspath(os.path.dirname(__file__))
configFilePath = os.path.join(fileDirectory, 'ATR220_Config.json')
process = RunProcess.RunProcess()
process.SetConfigVariables(configFilePath)
process.GetComPort(["list=PID_0180"])
def fin():
sys.exit()
request.addfinalizer(fin)
return process
def test_WipeOutReader(Process):
assert Process.WipeOutTheReader() == True
def test_LoadingKeysIntoMemoryMap(Process):
assert Process.LoadingKeysIntoMemoryMap() == True
def test_LoadingFW(Process): # only use bar
assert Process.LoadingFW() == True
def test_LoadingSBL(Process):
assert Process.LoadingSBL() == True
def test_CCIDReadForPaymentCards(Process):
assert Process.CCIDReadWrite('Payment Card') == True
Currently, if I run the following command from the windows command line, I get output on the console, but no captured output on the HTML report.
pytest C:\Projects\TripTickAT\test_TripTick.py -s --html=Report.html --verbose
Also, I would like to know the programmatic way of customizing the HTML report where I can update the file name, ordering test based on time of the execution and capturing the std-out.
I have tried additional flags to the pytest command. --capture sys and
-rP for Passed tests. --capture sys and -rF for failed tests
And I can see the console log as well in the html document after I click show All details button as shown in the output. I have captured only partial screen for the purpose of showing to you. But you can scroll down the output to see all the console logs. The grey area in the image below is the console output. I am not sure of the command line level flag that works regardless of failed or passed tests. But here is a temporary solution. This will print logs on command line console as well as html logs console
`

Save a copy of a notebook from within the notebook itself

I would like to save a copy of a notebook (or rename it) from within a cell of the notebook.
Preferably without too much JavaScript. Actually I guess something of this form should work
from IPython.display import display_html
display_html("script>Jupyter....???...()</script>")
Here is a solution only in Python. The notebook_path function comes from P.Toccaceli's solution on How do I get the current IPython Notebook name.
from notebook import notebookapp
import urllib
import json
import os
import ipykernel
from shutil import copy2
def notebook_path():
"""Returns the absolute path of the Notebook or None if it cannot be determined
NOTE: works only when the security is token-based or there is also no password
"""
connection_file = os.path.basename(ipykernel.get_connection_file())
kernel_id = connection_file.split('-', 1)[1].split('.')[0]
for srv in notebookapp.list_running_servers():
try:
if srv['token']=='' and not srv['password']: # No token and no password, ahem...
req = urllib.request.urlopen(srv['url']+'api/sessions')
else:
req = urllib.request.urlopen(srv['url']+'api/sessions?token='+srv['token'])
sessions = json.load(req)
for sess in sessions:
if sess['kernel']['id'] == kernel_id:
return os.path.join(srv['notebook_dir'],sess['notebook']['path'])
except:
pass # There may be stale entries in the runtime directory
return None
def copy_current_nb(new_name):
nb = notebook_path()
if nb:
new_path = os.path.join(os.path.dirname(nb), new_name+'.ipynb')
copy2(nb, new_path)
else:
print("Current notebook path cannot be determined.")
Then, simply use copy_current_nb('Save1') to create a copy named Save1.ipynb in the same directory.

Jupyter (Lab): Browsing the remote file system inside a notebook

I have several Jupyter notebooks which perform analysis on datasets. Right now, a dataset is specified by its filename. Every time the user wants to perform analysis on a new dataset, she/he has to edit the appropriate line in the notebook and modify dataset path string. The datasets can be located in different directories. The notebooks can also be located in different directories. In each notebook I would like to provide a widget that allows the user to browse the remote file system and pick the dataset he/she wants to analyse.
Are there any open source projects that support the above functionality? I am looking for something that is still active/supported and has some basic documentation. I did quick search on Google and surprisingly I didn't find anything.
Then I realised that JupyterLab, the evolution of Jupyter, has something very similar to what I want. It already has a very capable file browser but it is a bit "isolated" from everything else.
Is it possible somehow to get the relative (to the currently opened notebook) path of the selected file in the JupyterLab file browser?
Thank you.
Here's code for a server-side file browsing widget. Only tested in regular Jypter notebook - not Jupyter Lab. Also, must use a fairly recent version. Hope this helps.
import sys
import os
import ipywidgets as ui
from IPython.display import display
class PathSelector():
def __init__(self,start_dir,select_file=True):
self.file = None
self.select_file = select_file
self.cwd = start_dir
self.select = ui.SelectMultiple(options=['init'],value=(),rows=10,description='')
self.accord = ui.Accordion(children=[self.select])
self.accord.selected_index = None # Start closed (showing path only)
self.refresh(self.cwd)
self.select.observe(self.on_update,'value')
def on_update(self,change):
if len(change['new']) > 0:
self.refresh(change['new'][0])
def refresh(self,item):
path = os.path.abspath(os.path.join(self.cwd,item))
if os.path.isfile(path):
if self.select_file:
self.accord.set_title(0,path)
self.file = path
self.accord.selected_index = None
else:
self.select.value = ()
else: # os.path.isdir(path)
self.file = None
self.cwd = path
# Build list of files and dirs
keys = ['[..]'];
for item in os.listdir(path):
if item[0] == '.':
continue
elif os.path.isdir(os.path.join(path,item)):
keys.append('['+item+']');
else:
keys.append(item);
# Sort and create list of output values
keys.sort(key=str.lower)
vals = []
for k in keys:
if k[0] == '[':
vals.append(k[1:-1]) # strip off brackets
else:
vals.append(k)
# Update widget
self.accord.set_title(0,path)
self.select.options = list(zip(keys,vals))
with self.select.hold_trait_notifications():
self.select.value = ()
f = PathSelector('/some/data')
display(f.accord)

Resources