I am trying to make a progress bar using tqdm for an asyncio function. I've tried following the guidance at:
tqdm for asyncio
and
asyncio with tqdm
and
tqdm and coroutines
Here's my code, which is running in a jupyter notebook:
import pandas as pd
from pandas_datareader import data as pdr
import asyncio
# import tqdm
from tqdm.asyncio import tqdm
async def get_prices(index, row):
try:
prices = pdr.get_data_yahoo(row['Symbol'], row['Start'], row['End'])
except Exception as e:
print('Error',e,row['Symbol'],row['Date'])
return
prices['Symbol'] = row['Symbol']
prices['Key'] = index
return prices
async def get_stock_data(df):
# data = await asyncio.gather(*[get_prices(index, row) for index, row in df.iterrows()])
# data = [await f for f in tqdm(asyncio.as_completed([get_prices(index, row) for index, row in df.iterrows()]), total=len(df)]
flist = [get_prices(index, row) for index, row in df.iterrows()]
data = [await f for f in tqdm.as_completed(flist, total=len(df))]
return data
stocks = ['IBM', 'AAPL', 'C', 'ACTG', 'ACVA', 'ACWI', 'ACWX', 'ACXP', 'ADAG', 'ADAL', 'IBET', 'IBEX', 'IBKR', 'IBOC', 'IBRX', 'IBTB', 'IBTD', 'IBTE', 'IBTF', 'IBTG', 'IBTH', 'IBTI', 'IBTJ', 'IBTK', 'IBTL', 'IBTM', 'IBTX', 'ICAD', 'ICCC', 'ICCH', 'ICCM', 'ICFI', 'ICHR', 'ICLK', 'ICLN', 'ICLR', 'ICMB', 'ICPT', 'ICUI', 'ICVX', 'IDAI', 'IDBA', 'IDCC', 'IDEX', 'IDLB', 'IDN', 'IDRA', 'IDXX', 'IDYA', 'IEA', 'IEAWW', 'IEF', 'IEI', 'IEP', 'IESC', 'IEUS', 'IFBD', 'IFGL', 'IFRX', 'IFV', 'IGAC', 'IGACU', 'IGACW', 'IGF', 'IGIB', 'IGIC', 'IGICW', 'IGMS', 'IGNY', 'IGNYU', 'IGNYW', 'IGOV', 'IGSB', 'IGTA', 'IGTAR', 'IGTAU', 'IGTAW', 'IHRT', 'IHYF', 'III', 'IIII', 'IIIIU', 'IIIIW', 'IIIV', 'IINN', 'IINNW', 'IIVI', 'IIVIP', 'IJT', 'IKNA', 'IKT', 'ILAG', 'ILMN', 'ILPT', 'IMAB', 'IMAC', 'IMACW', 'IMAQ', 'IMAQR', 'IMAQU', 'IMAQW', 'IMBI', 'IMBIL', 'IMCC', 'IMCR', 'IMCV', 'IMGN', 'IMGO', 'IMKTA', 'IMMP', 'IMMR', 'IMMX', 'IMNM', 'IMOS', 'IMPL', 'IMPP', 'IMPPP', 'IMRA', 'IMRN', 'IMRX', 'IMTE', 'IMTX', 'IMTXW', 'IMUX', 'IMV', 'IMVT', 'IMXI', 'INAB', 'INBK', 'INBKZ', 'INBX', 'INCR', 'INCY', 'INDB', 'INDI', 'INDIW', 'INDP', 'INDT', 'INDY', 'INFI', 'INFN', 'INGN', 'INKA', 'INKAU', 'INKAW', 'INKT', 'INM', 'INMB', 'INMD', 'INNV', 'INO', 'INOD', 'INPX', 'INSE', 'INSG', 'INSM', 'INTA', 'INTC', 'INTE', 'INTEU', 'INTEW', 'INTG', 'INTR', 'INTU', 'INTZ', 'INVA', 'INVE', 'INVO', 'INVZ', 'INVZW', 'INZY', 'IOAC', 'IOACU', 'IOACW', 'IOBT', 'IONM', 'IONR', 'IONS', 'IOSP', 'IOVA', 'IPA', 'IPAR', 'IPAX', 'IPAXU', 'IPAXW', 'IPDN', 'IPGP', 'IPHA', 'IPKW', 'IPSC', 'IPVI', 'IPVIU', 'IPVIW', 'IPW', 'IPWR', 'IPX', 'IQ', 'IQMD', 'IQMDU', 'IQMDW', 'IRAA', 'IRAAU', 'IRAAW', 'IRBT', 'IRDM', 'IREN', 'IRIX', 'IRMD', 'IROQ', 'IRTC', 'IRWD', 'ISAA', 'ISDX', 'ISEE']
df_stocks = pd.DataFrame(stocks, columns=['Symbol'])
df_stocks['Start'] = '9/1/2022'
df_stocks['End'] = '9/11/2022'
data = pd.concat([d for d in await get_stock_data(df_stocks)])
data.dropna(inplace=True)
data.to_csv('../output/stockprices.csv', sep='\t')
data
The code works, but the progress bar does not. I get this output which does not change:
0%| | 0/214 [00:00<?, ?it/s]
I have also tried from tqdm.autonotebook import tqdm, but that gives the same result.
I am sure I'm doing something boneheaded, but am unable to solve this on my own.
TL;DR
Problem is that Async Function get_prices() doesn't have any Awaitable, hence it's not async but synchronous. You will definitely want to understand what Coroutine is before reading the rest of this.
Since that library pandas-datareader does not define any asynchronous function - you might be better offloading the pdr.get_data_xxx to thread or give up concurrency and use synchronous tqdm.
Explanation
Problem-wise, what you wrote could be simplified as following:
import asyncio
import random
import time
from tqdm.asyncio import tqdm
async def fake_async_task():
time.sleep(0.5 + random.random()) # <- notice there is no await & awaitable!
async def main():
tasks = [fake_async_task() for _ in range(10)]
_ = [await task_ for task_ in tqdm.as_completed(tasks, total=len(tasks))]
# asyncio.run(main()) --> when not in Jupyter
await main() # --> in Jupyter
If you run this, you'll notice this also seemingly jumps the progress from 0% to 100% just before it ends, and total execution takes much longer than 0.5 + α seconds.
0%| | 0/10 [00:00<?, ?it/s]
Reason behind it is quite complicated; Throwing a fake async function - aka async func without await - is outside of the documented usages, thus requires looking inside the library codes.
To overly simplify the flows - at cost of accuracy & proper terminology:
We feed 10 fake_async_task() coroutines to tqdm.asyncio.tqdm.as_completed.
tqdm.asyncio.tqdm.as_completed is merely a wrapper for asyncio.as_completed, so tqdm pass all given Awaitable to it then wait for any results.
asyncio.as_completed schedule execution of all given Awaitable, and then schedule Awaitable named _wait_for_one for getting results.
First scheduled fake_async_task() starts running, until next await keyword it encounters.
But we haven't put any await keywords, so coroutine ends up running start to end without stopping.
Same thing happens for other 9 scheduled fake_async_task(), and _wait_for_one is still patiently waiting for it's turn.
when it's finally _wait_for_one's turn, all tasks are already done, so yielding result happens faster than human eyes can see, so is the progress bar's progress changes.
That's why total execution time was addictive, it never really archived any concurrency during execution.
Running functions like fake_async_task() is simply neither tqdm authors nor asyncio had in mind. Others would usually write codes like this instead:
import asyncio
import random
from tqdm.asyncio import tqdm
async def task():
await asyncio.sleep(0.5 + random.random()) # <- await + something that's awaitable
# adding random val to prevent it finishing altogether,
# or progress bar will seemingly jump from 0 to 100 again.
async def main():
tasks = [task() for _ in range(10)]
_ = [await task_ for task_ in tqdm.as_completed(tasks, total=len(tasks))]
# asyncio.run(main())
await main()
Which now then print out progress - more like, slow enough so we can see - just as we wanted. Also, total execution time was 0.5 + α seconds, achieving proper concurrency.
30%|███ | 3/10 [00:00<00:01, 3.98it/s]
Alternatives
But if function you want to use happened to not have async variant, yet it is gladly not CPU intensive, but I/O intensive, then you can offload it to another thread to achieve concurrency while using asynchronous APIs.
import asyncio
import random
import time
from tqdm.asyncio import tqdm
def io_intensive_sync_task():
time.sleep(0.5 + random.random())
async def main():
tasks = [asyncio.to_thread(io_intensive_sync_task) for _ in range(10)]
_ = [await task_ for task_ in tqdm.as_completed(tasks, total=len(tasks))]
# can skip the total param as tqdm internally use len if not provided
# asyncio.run(main())
await main()
Which will run just as the previous example does.
I have encountered a theoretical question. I'm using pyqt5, but this is probably are very generalistic and framework independent question.
I have a QMainwindow sitting around waiting for the user to do stuff. The user can show / hide dialogues (subclasses of QDockwidgets) as he chooses using the QMenu and the associated shortcuts (it's a checkable QAction for each individual dialogue).
I have been struggling with showing / hiding the dialogues efficiently. Currently, I'm just initiating them all at start up, hiding those that I don't want to show up in the beginning. This makes triggering the dialogues easy, since I can just dialogue.show() /dialogue.hide() depending on the dialogues current visibility.
But I cannot believe that this is best practice and very efficient.
I have tried (I currently do not have my pyqt environment set up on this computer, so I had to strip down my actual code without being able to test if this runs):
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
class InfoPanel(QDockWidget):
def __init__(self, title='Tool Box'):
QDockWidget.__init__(self, title)
self.setFeatures(QDockWidget.DockWidgetFloatable | QDockWidget.DockWidgetMovable | QDockWidget.DockWidgetClosable)
self.setAllowedAreas(Qt.LeftDockWidgetArea | Qt.RightDockWidgetArea)
frame = QFrame()
layout = QGridLayout()
self.canvas = QGraphicsView()
self.canvas.setBackgroundBrush(QtGui.QBrush(QtGui.QColor(40, 40, 40)))
layout.addWidget(self.canvas)
frame.setLayout(layout)
self.setWidget(frame)
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.showpanelAct = QAction("&Show Panel", self, enabled=True,checkable=True, shortcut="F10")
self.showpanelAct.triggered.connect(lambda: self.showPanel(0))
self.viewMenu = QMenu("&View", self)
self.viewMenu.addAction(self.showpanelAct)
self.setDockOptions(QMainWindow.AnimatedDocks)
def showPanel(self,i:int = 0): # this is not so smart - should construct and deconstuct to save memory!?
if i == 0: #infopanel
dialogueExists = True
try: self.infoPanel
#except NameError: #does not catch the error
except:
dialogueExists = False
if dialogueExists:
print('destroy')
self.infoPanel.destroy()
else:
print('create')
self.infoPanel = InfoPanel() #init
self.infoPanel.show()
if __name__ == '__main__':
import sys
app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()
Which works the first time, but after that, it only seems to trigger the destruction of the dialogue (which, surprisingly, does not crash anything it just keeps on going).
Why is that and is there a standard way to approach the showing hiding of dialogues?
I took the exposed MCVE of OP and tried to make it running in my cygwin64 on Windows 10.
At first I had to apply little fixes. (OP stated that he was not able to test it at the time of publishing.)
First, I inserted a “hut” at first line for convenient start in bash:
#!/usr/bin/python3
Second, the self.viewMenu didn't appear. Hence, I inserted a line after
self.viewMenu = QMenu("&View", self)
self.viewMenu.addAction(self.showpanelAct)
to add the viewMenu to main menu bar:
self.menuBar().addMenu(self.viewMenu)
which fixed it.
Third, when clicking the menu item I got:
Traceback (most recent call last):
File "./testQDockPanelShowHide.py", line 27, in <lambda>
self.showpanelAct.triggered.connect(lambda: self.showPanel(0))
File "./testQDockPanelShowHide.py", line 45, in showPanel
self.infoPanel = InfoPanel() #init
File "./testQDockPanelShowHide.py", line 17, in __init__
self.canvas.setBackgroundBrush(QtGui.QBrush(QtGui.QColor(40, 40, 40)))
NameError: name 'QtGui' is not defined
Aborted (core dumped)
I must admit that my Python knowledge is very limited. (I'm the guy who writes the Python bindings in C++ for the colleagues. So, my colleagues are the actual experts. At most, I play a little bit in Python when I test whether new implemented bindings do what's expected.) However, I modified
self.canvas.setBackgroundBrush(QtGui.QBrush(QtGui.QColor(40, 40, 40)))
to:
self.canvas.setBackgroundBrush(QBrush(QColor(40, 40, 40)))
which fixed this issue.
After this, I got the behavior described by OP and did a closer look where I (and OP) suspected the error:
def showPanel(self,i:int = 0): # this is not so smart - should construct and deconstuct to save memory!?
if i == 0: #infopanel
dialogueExists = True
try: self.infoPanel
#except NameError: #does not catch the error
except:
dialogueExists = False
if dialogueExists:
print('destroy')
self.infoPanel.destroy()
else:
print('create')
self.infoPanel = InfoPanel() #init
self.infoPanel.show()
I strongly believe that try: self.infoPanel doesn't do what OP thinks it would.
It tries to access self.infoPanel which isn't existing until the first call of this method. (Please, be aware, the member variable self.infoPanel isn't existing.) So, the except: branch is executed and sets dialogueExists = False which a few lines later causes self.infoPanel = InfoPanel() #init. Now, the member variable self.infoPanel is existing, and the try: self.infoPanel will never fail again until destruction of this MainWindow.
Out of curiosity, I had a look at QWidget.destroy() (to be sure not to tell something wrong):
QWidget.destroy (self, bool destroyWindow = True, bool destroySubWindows = True)
Frees up window system resources. Destroys the widget window if destroyWindow is true.
destroy() calls itself recursively for all the child widgets, passing destroySubWindows for the destroyWindow parameter. To have more control over destruction of subwidgets, destroy subwidgets selectively first.
This function is usually called from the QWidget destructor.
It definitely doesn't destroy the member variable self.infoPanel.
After having understood this, a fix was easy and obvious:
def showPanel(self,i:int = 0): # this is not so smart - should construct and deconstuct to save memory!?
if i == 0: #infopanel
try: self.infoPanel
#except NameError: #does not catch the error
except:
print('create')
self.infoPanel = InfoPanel() #init
if self.infoPanel.isVisible():
self.infoPanel.hide()
else:
self.infoPanel.show()
Btw. I replaced destroy() by hide() which makes a re-creation of the InfoPanel() obsolete.
I tested this by toggling the menu item multiple times – it works as expected now (at least, it looks like).
The complete sample finally:
#!/usr/bin/python3
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
class InfoPanel(QDockWidget):
def __init__(self, title='Tool Box'):
QDockWidget.__init__(self, title)
self.setFeatures(QDockWidget.DockWidgetFloatable | QDockWidget.DockWidgetMovable | QDockWidget.DockWidgetClosable)
self.setAllowedAreas(Qt.LeftDockWidgetArea | Qt.RightDockWidgetArea)
frame = QFrame()
layout = QGridLayout()
self.canvas = QGraphicsView()
# self.canvas.setBackgroundBrush(QtGui.QBrush(QtGui.QColor(40, 40, 40)))
self.canvas.setBackgroundBrush(QBrush(QColor(40, 40, 40)))
layout.addWidget(self.canvas)
frame.setLayout(layout)
self.setWidget(frame)
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.showpanelAct = QAction("&Show Panel", self, enabled=True,checkable=True, shortcut="F10")
self.showpanelAct.triggered.connect(lambda: self.showPanel(0))
self.viewMenu = QMenu("&View", self)
self.viewMenu.addAction(self.showpanelAct)
self.menuBar().addMenu(self.viewMenu)
self.setDockOptions(QMainWindow.AnimatedDocks)
def showPanel(self,i:int = 0): # this is not so smart - should construct and deconstuct to save memory!?
if i == 0: #infopanel
try: self.infoPanel
#except NameError: #does not catch the error
except:
print('create')
self.infoPanel = InfoPanel() #init
if self.infoPanel.isVisible():
self.infoPanel.hide()
else:
self.infoPanel.show()
if __name__ == '__main__':
import sys
app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()
After taking a break from coding the solution to my problem was very obvious. Going back to my original code (which did not produce the expected output of creating / destroying the dialogue self.infoPanel on demand):
dialogueExists = True
try: self.infoPanel
#except NameError: #does not catch the error
except:
dialogueExists = False
if dialogueExists:
print('destroy')
self.infoPanel.destroy()
else:
print('create')
self.infoPanel = InfoPanel() #init
self.infoPanel.show()
My main problem was that I confused two separate things. Qt destroyed the widget contained in the object self.infoPanel when I called self.infoPanel.destroy(). But that doesn't mean the object self.infoPanel does not exist (that's exactly what I use try: ... for, to see if the object exists). The simple and obvious way to create and destroy dialogues on demand obviously involves deleting the object from the environment (del self.infoPanel).
The working code is:
dialogueExists = True
try:
self.infoPanel.destroy() #not sure this is needed, but I guess it doesn't hurt
del self.infoPanel #this is the deletion of the actual object
except:
dialogueExists = False
if not dialogueExists :
self.infoPanel = InfoPanel()
Cheers and many thanks for the helpful advice on deciding whether to show / hide dialogues or to create / destroy them!
I've got a Qt4 application (using the PyQt bindings) which contains a QListWidget, initialized like so:
class MyList(QtGui.QListWidget):
def __init__(self):
QtGui.QListWidget.__init__(self)
self.setDragDropMode(self.InternalMove)
I can add items, and this allows me to drag and drop to reorder the list. But how do I get notification when the list gets reordered by the user? I tried adding a dropMimeData(self, index, data, action) method to the class, but it never gets called.
I have an easier way. :)
You can actually access the listwidget's internal model with myList->model() - and from there there are lots of signals available.
If you only care about drag&drop, connect to layoutChanged.
If you have move buttons (which usually are implemented with remove+add) connect to rowsInserted too.
If you want to know what moved, rowsMoved might be better than layoutChanged.
I just had to deal with this and it's a pain in the ass but here's what to do:
You have to install an eventFilter on your ListWidget subclass and then watch for the ChildRemoved event. This event covers moves as well as removal, so it should work for re-arranging items with drag and drop inside a list.
I write my Qt in C++, but here's a pythonification version:
class MyList(QtGui.QListWidget):
def __init__(self):
QtGui.QListWidget.__init__(self)
self.setDragDropMode(self.InternalMove)
self.installEventFilter(self)
def eventFilter(self, sender, event):
if (event.type() == QEvent.ChildRemoved):
self.on_order_changed()
return False # don't actually interrupt anything
def on_order_changed(self):
# do magic things with our new-found knowledge
If you have some other class that contains this list, you may want to move the event filter method there. Hope this helps, I know I had to fight with this for a day before figuring this out.
I found Trey Stout's answer did work however I was obviously getting events when the list order had not actually changed. I turned to Chani's answer which does work as required but with no code it took me a little work to implement in python.
I thought I would share the code snippet to help out future visitors:
class MyList(QListWidget):
def __init__(self):
QListWidget.__init__(self)
self.setDragDropMode(self.InternalMove)
list_model = self.model()
list_model.layoutChanged.connect(self.on_layout_changed)
def on_layout_changed(self):
print "Layout Changed"
This is tested in PySide but see no reason it wouldn't work in PyQt.
I know this is old, but I was able to get my code to work using Trey's answer and wanted to share my python solution. This is for a QListWidget inside a QDialog, not one that is sub-classed.
class NotesDialog(QtGui.QDialog):
def __init__(self, notes_list, notes_dir):
QtGui.QDialog.__init__(self)
self.ui=Ui_NotesDialog()
# the notesList QListWidget is created here (from Qt Designer)
self.ui.setupUi(self)
# install an event filter to catch internal QListWidget drop events
self.ui.notesList.installEventFilter(self)
def eventFilter(self, sender, event):
# this is the function that processes internal drop in notesList
if event.type() == QtCore.QEvent.ChildRemoved:
self.update_views() # do something
return False # don't actually interrupt anything
Not a solution, but some ideas:
You should probably check what is returned by supportedDropActions method. It might be that you need to overwrite that method, to include Qt::MoveAction or Qt::CopyAction.
You have QListView::indexesMoved signal, but I am not sure whether it will be emitted if you're using QListWidget. It worths checking.
The QListWidget.model() approach seemed the most elegant of the proposed solutions but did not work for me in PyQt5. I don't know why, but perhaps something changed in the move to Qt5. The eventFilter approach did work, but there is another alternative that is worth considering: over-riding the QDropEvent and checking if event.source is self. See the code below which is an MVCE with all of the proposed solutions coded in for checking in PyQt5:
import sys
from PyQt5 import QtGui, QtWidgets, QtCore
class MyList(QtWidgets.QListWidget):
itemMoved = QtCore.pyqtSignal()
def __init__(self):
super(MyList, self).__init__()
self.setDragDropMode(self.InternalMove)
list_model = self.model()
# list_model.layoutChanged.connect(self.onLayoutChanged) # doesn't work
# self.installEventFilter(self) # works
self.itemMoved.connect(self.onLayoutChanged) # works
def onLayoutChanged(self):
print("Layout Changed")
def eventFilter(self, sender, event):
"""
Parameters
----------
sender : object
event : QtCore.QEvent
"""
if event.type() == QtCore.QEvent.ChildRemoved:
self.onLayoutChanged()
return False
def dropEvent(self, QDropEvent):
"""
Parameters
----------
QDropEvent : QtGui.QDropEvent
"""
mime = QDropEvent.mimeData() # type: QtCore.QMimeData
source = QDropEvent.source()
if source is self:
super(MyList, self).dropEvent(QDropEvent)
self.itemMoved.emit()
app = QtWidgets.QApplication([])
form = MyList()
for text in ("one", "two", "three"):
item = QtWidgets.QListWidgetItem(text)
item.setFlags(item.flags() | QtCore.Qt.ItemIsUserCheckable)
item.setFlags(item.flags() | QtCore.Qt.ItemIsEditable)
item.setCheckState(QtCore.Qt.Checked)
form.addItem(item)
form.show()
sys.exit(app.exec_())