KeyboardInterrupt handling is extremely slow. Where is the problem? - paramiko

I'm currently trying to write a script to read a shell via paramiko and, by far, if I prompt a single command it works great.
the problem is that my actual goal is to handle a debug flow and of course I cannot use a timeout because the script should listen without time limitations.
So the other option is to let the user stop the flow with ctrl-c. I added a "try, except" handler like
except (IOError, SystemExit):
raise
except KeyboardInterrupt:
print ("Crtl+C Pressed. Shutting down.")
and it seemed like it wasn't working: nothing happened when pressing ctrl-c. But then I realized that it was just taking forever: after a few minutes the script actually stops and the message appears.
Does anyone know how to handle this problem?
just for the reference here's my code
import paramiko
import time
import re
import socket
import os
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
address = '192.168.1.1'
username = 'admin'
password = ''
ssh.connect(address, port=22, username = username, password = password)
shell = ssh.invoke_shell(height=400)
shell.settimeout(2)
def prompt_console(command = '', channel = shell, timeout = False):
print(channel.recv(9999).decode('utf-8')) #<-- this is for removing the initial prompt from buffer
if timeout == False:
shell.settimeout(3600) #<--increasing the timeout
channel.send(command+'\n')
log = ''
while True:
try:
log += channel.recv(1).decode('utf-8') #<--every loop it loads a char into the "log" variable
if log[-9:] == '--More-- ':
shell.send(" ")
log.replace('--More-- \r \r\t\t','\r\t\t')
if len(re.findall('.*\n',log))>0: #<--it yields every line to python console
yield log
log = ''
except (IOError, SystemExit):
raise
except KeyboardInterrupt:
print ("Crtl+C Pressed. Shutting down.")
I also tried to put the try-except outside the while loop but it didn't work. I also tried many suggestions that I've found on google but they didn't work either: they all stop the script eventually but for all of them it takes 5-10 minutes.

Related

Requests(url) is having after 5 iteration

I am attempting to run a webscraping algo on indeed using beautifulSoup and loop through the different pages. However, after 2-6 iterations, the requests.get(url) hangs and stops finding the next page. I have read that it might do something with the server being blocked but that would have blocked the original requests and it also says online that Indeed allows for web scraping. I have also heard that I should set a header but I am unsure how to do that. I am running on the latest version of safari and MacOs 12.4.
A solution I came up with, thought this does not answer the question specifically, is by using a try expect statement and setting a timeout value to the request. Once the timeout value is reached, it enters the try except statement, sets a boolean value, and then continues the loop and try again. Code is inserted below.
while(i < 10):
url = get_url('software intern', '', i)
print("Parsing Page Number:" + str(i + 1))
error = False
try:
response = requests.get(url, timeout = 10)
except requests.exceptions.Timeout as err:
error = True
if error:
print("Trying to connect to webpage again")
continue
i += 1
I am leaving the question as unanswered for now however as I still don't know the root cause of this issue and this solution is just a workaround.

problem with event loops, gui qt5, and ipywidgets in a jupyter notebook

I am trying to integrate interactive ipywidgets with a loop in my code that also performs other tasks (in this case, acquiring data from some hardware attached from the computer and updating live plots).
In the past, I could do this by using IPython.kernel.do_one_iteration() in my while loop: this would trigger a sync of the ipywidget changes and I would be able to retrieve them from the python widget objects. A minimal example is here:
import ipywidgets as widgets
from time import sleep
import IPython
do_one_iteration = IPython.get_ipython().kernel.do_one_iteration
w = widgets.ToggleButton()
display(w)
i=0
while True:
do_one_iteration()
print(i, w.value, end="\r")
w.decription = str(i)
sleep(0.5)
i+=1
Here, the for loop prints out the ticker integer along with the state of the widget. (In the real code, I would also acquire data, update plots, and change plot / acquisition settings dependent on the interaction with the user via the widgets.)
With ipykernel 5.3.2 and ipython 7.16.1, this worked fine: if the widget changed, calling do_one_iteration() synced the widget states to the kernel and I could retrieve it from my while loop.
After an upgrade (to 6.4.1 and 7.29.0), this no longer works. It seems that do_one_iteration() is now a coroutine: I get a warning coroutine 'Kernel.do_one_iteration' was never awaited if I use the above code.
With some help of a friend, we found a way to do this with threading an asyncio:
%gui asyncio
import asyncio
import ipywidgets as widgets
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
i=0
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
asyncio.create_task(f());
And this works (also adding a stop button, and changing to text output widget instead of printing). But to throw a spanner in the works, I need to use a library that itself uses a QT gui event loop. Some more puzzling suggests that this should be the code to make this work:
%gui qt5
import asyncio
import ipywidgets as widgets
import qasync
button = widgets.ToggleButton()
display(button)
text = widgets.Text()
display(text)
text.value= str(button.value)
stop_button = widgets.ToggleButton()
stop_button.description = "Stop"
display(stop_button)
async def f():
while True:
i += 1
text.value = str(i) + " " + str(button.value)
await asyncio.sleep(0.2)
if stop_button.value == True:
return
from qtpy import QtWidgets
APP = QtWidgets.QApplication.instance()
loop = qasync.QEventLoop(APP)
asyncio.set_event_loop(loop)
asyncio.create_task(f());
But with this code, the updates do not propagate, and I get the following error on the terminal running my notebook server:
[IPKernelApp] ERROR | Error in message handler
Traceback (most recent call last):
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 457, in dispatch_queue
await self.process_one()
File "/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 440, in process_one
t, dispatch, args = await self.msg_queue.get()
RuntimeError: Task <Task pending name='Task-2'
coro=<Kernel.dispatch_queue() running at
/Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/ipykernel/kernelbase.py:457>
cb=[IOLoop.add_future.<locals>.<lambda>() at /Users/gsteele/anaconda3/envs/myenv2/lib/python3.9/site-packages/tornado/ioloop.py:688]>
got Future <Future pending> attached to a different loop
It seems that somehow my ipywidgets events are propagating to the wrong event loop.
And now my question is: does anybody know what is going on here?
It's hard for me to identify if this is a "bug", and if so, in which software package do things go wrong? ipykernel? Or tornado? Or ipywidgets? Or asyncio? Or maybe I'm missing something?
Any thoughts highly welcome, thanks!
Found at least a partial solution: using the nest_asyncio package allows me to now use do_one_iteration(), just by adding the following to the first code block:
import nest_asyncio
nest_asyncio.apply()
and then using await do_one_iteration() instead of calling it directly.
(see https://github.com/ipython/ipykernel/issues/825)
For my purposes, this solves my issue since I don't need asynchronous interaction with my GUI. The problem of the %gui qt5 interaction with the event loop in the asynchronous versions of the code is still a mystery though...

Python script run in background not writing to file?

I've created my first ever python script and it works fine in the foreground, but when I run it in the background it creates, but fails to write anything to the file. I run the script with command : python -u testit.py &
Please help me understand why this happens.
#!/usr/bin/env python
import time
import datetime
dt = datetime.datetime.now()
dtLog = dt.strftime("ThermoLogs/TempLOG%Y%m%d")
f = open(dtLog,"a")
while True:
dt = datetime.datetime.now()
print('{:%Y-%m-%d %H:%M:%S}'.format(dt))
f.write('{:%Y-%m-%d %H:%M:%S}'.format(dt))
f.write('\n')
time.sleep(5)
The output will arrive in bursts (it is being buffered). With the sleep set to 0.01 I am getting bursts about every 2 seconds. So for a delay of 5 you will get them much less frequent. You may also note that it outputs all pending outputs when you terminate it.
To make the output arrive now call f.flush().
I don't see any difference between foreground and background. However it should not buffer stdout when going to a terminal.

How to capture the log details on pytest-html as well as writing in to Console?

In my pytest script, I need to customize the pytest-HTML report for capturing the stdout at the same time writing into the console as I have user input in my automated test.
test_TripTick.py
import os
import sys
import pytest
from Process import RunProcess
from recordtype import recordtype
from pip._vendor.distlib.compat import raw_input
#pytest.fixture(scope="module")
def Process(request):
# print('\nProcess setup - module fixture')
fileDirectory = os.path.abspath(os.path.dirname(__file__))
configFilePath = os.path.join(fileDirectory, 'ATR220_Config.json')
process = RunProcess.RunProcess()
process.SetConfigVariables(configFilePath)
process.GetComPort(["list=PID_0180"])
def fin():
sys.exit()
request.addfinalizer(fin)
return process
def test_WipeOutReader(Process):
assert Process.WipeOutTheReader() == True
def test_LoadingKeysIntoMemoryMap(Process):
assert Process.LoadingKeysIntoMemoryMap() == True
def test_LoadingFW(Process): # only use bar
assert Process.LoadingFW() == True
def test_LoadingSBL(Process):
assert Process.LoadingSBL() == True
def test_CCIDReadForPaymentCards(Process):
assert Process.CCIDReadWrite('Payment Card') == True
Currently, if I run the following command from the windows command line, I get output on the console, but no captured output on the HTML report.
pytest C:\Projects\TripTickAT\test_TripTick.py -s --html=Report.html --verbose
Also, I would like to know the programmatic way of customizing the HTML report where I can update the file name, ordering test based on time of the execution and capturing the std-out.
I have tried additional flags to the pytest command. --capture sys and
-rP for Passed tests. --capture sys and -rF for failed tests
And I can see the console log as well in the html document after I click show All details button as shown in the output. I have captured only partial screen for the purpose of showing to you. But you can scroll down the output to see all the console logs. The grey area in the image below is the console output. I am not sure of the command line level flag that works regardless of failed or passed tests. But here is a temporary solution. This will print logs on command line console as well as html logs console
`

Telnetlib doesn't seem to be passing .write commands

I'm a python noob here just trying to learn a new skill so please be gentle :)
After executing the following script it appears that my script creates a telnet session and I can see the cisco devices banner but is not passing the username/password when prompted.
I've tried changing tn.read_until() and tn.read_very_eager() neither of which is triggering the script to move on to write the desired input. I've also tried forcing the username and password to be input as a byte as well as adding + '\n' to my write functions. I've also used sleep to pass a little extra time to wait for the banner to finish printing out.
tldr: when executing the script I see the username prompt, but can't get further than that.
any assistance here is welcomed.
'''
from time import sleep
import telnetlib
from telnetlib import Telnet
from getpass import getpass
Username = input('please provide your username ')
password = getpass()
# f is the .txt document that lists the IP's we'll be using.
f = open('devicess.txt')
for line in f:
print ('Configuring Device ' + (line))
device = (line)
#open connection to variable
tn = telnetlib.Telnet(device)
#For those devices in the above list, connect and run the below commands
for device in f:
tn.expect('Username: ')
tn.write(Username)
tn.expect('Password: ')
tn.write(password)
tn.write('show ver')
print('connection established to ' + device)
tn.write('logout')
output = tn.read_all()
print(output)
print('script complete')
'''

Resources