I am working on script where I will be dealing with huge amount of data to process via python.
I have written a script using asyncio in python3.8 on windows box which is working perfectly fine but when I execute the same script on unix on python3.8 its completing the execution but not terminating the program at the end. Seems like its not release resources/lock.
When I debug further, found that on windows the asyncio uses ProactorEventLoop whereas on Unix it uses _UnixSelectorEventLoop, But not sure if this affect by any means.
I cant share the full script but it follows below structure:
import asyncio
async def myCoroutine():
print("My Coroutine")
try:
loop = asyncio.get_event_loop()
loop.run_until_complete(myCoroutine())
print("Execution Completed")
finally:
print("Closing the loop")
loop.close()
print("loop Closed")
Output:
Execution Completed
Closing the loop
loop closed
But program is not terminating.
Is anyone faced the similar issue before? Any inputs?
Thanks in Advance!!
Related
I have a flask app hosted on gunicorn webserver and trying to parallelize long running I/O bound task "somemethod" as shown below. However often times (not always) it throws error "Event loop is closed". What could be the reason for the error happening randomly ?
Removing loop.close() does the fix the errors, but I am not sure if there would be memory leak in the python worker process.
async def somemethod():
""" Do some work """
app.route('/hello', method=['POST'])
def sayhello():
loop = asyncio.new_event_loop()
try:
asyncio.set_event_loop(loop)
future = asyncio.ensure_future(somemethod)
loop.run_until_complete(future)
finally:
loop.close()
I am currently doing several simulations in R that each take quite a long time to execute and the time it takes for each to finish varies from case to case. To use the time in between more efficiently, I wondered if it would be possible to set up something (like a e-mail notification system or similar) that would notify me as soon a a chunk of simulation is completed.
Does somebody here have any experience with setting up something similar or does someone know a resource that could teach me to implement a notification system via R?
I recently saw an R package for this kind of thing: pushoverr. However didn't use it myself - so not tested how it works. But seems like it might be useful in your case.
I assume you run the time consuming simulations on a server, correct? If these run own you own PC, your PC will be slow as hell anyway and I would not see something beneficial in sending a mail to myself.
For long calculations: Run them on a virtual machine, I use the following workflow for my own calculations.
Write your R script. Important: Write a .txt file when the calculation file in the end. The shell script will search in a loop for the file to exist.
Copy that code an save it as Python script. I tried one day to get MailR running a Linux and it did not work. This code worked on the first try.
#!/usr/bin/env python3
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders
email_user = 'youownmail#gmail.com'
email_password = 'password'
email_send = 'theothersmail.com'
subject = 'yourreport'
msg = MIMEMultipart()
msg['From'] = email_user
msg['To'] = email_send
msg['Subject'] = subject
body = 'Calculation is done'
msg.attach(MIMEText(body,'plain'))
part = MIMEBase('application','octet-stream')
part.set_payload((attachment).read())
encoders.encode_base64(part)
msg.attach(part)
text = msg.as_string()
server = smtplib.SMTP('smtp.gmail.com',587)
server.starttls()
server.login(email_user,email_password)
server.sendmail(email_user,email_send,text)
server.quit()
Make sure you are allowed to run the script.
sudo chmod 777 /path/script.R sudo chmod 777 /path/script.py
Run both your script.R and script.py inside a script.sh file. It looks the the following:
R < /path/script.R --no-save
while [ ! -f /tmp/finished.txt ]
do
sleep 2
done
python path/script.py
This may sound a bit overwhelming if you are not familiar with these technologies, but think this is a pretty much automated workflow, which relieves your own resources and can be used "in production". (I use this workflow to send me my own stock reports).
I am using asyncio for the first time and I am getting these two errors:
1) RuntimeError: set_wakeup_fd only works in main thread
2) RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
Usage scenario:
I have to execute two terminal commands, one followed by another. To ensure the second command gets executed only after the first, I'm using asyncio to set the loops and running them.
I am trying to execute a function in this python file using a flask api ultimately.
Has someone come across these errors and can guide me to resolve it?
Any help would be great!
Code:
async def segmentVide(command):
process = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE)
stdout, stderr = await process.communicate()
asyncio.get_child_watcher()
return stdout.decode().strip()
def segmentOutput(folderCreated):
#command = MP4Box terminal command
loop = asyncio.new_event_loop()
asyncio.get_child_watcher().attach_loop(loop)
asyncio.set_event_loop(loop)
coro = loop.run_in_executor(None, segmentVide, command)
loop.run_until_complete(coro)
loop.close()
print('completed')
There is another function executing the other terminal command but the asyncio usage part is the same in it too.
While using py.test, I have some tests that run fine with SQLite but hang silently when I switch to Postgresql. How would I go about debugging something like that? Is there a "verbose" mode I can run my tests in, or set a breakpoint ? More generally, what is the standard plan of attack when pytest stalls silently? I've tried using the pytest-timeout, and ran the test with $ py.test --timeout=300, but the tests still hang with no activity on the screen whatsoever
I ran into the same SQLite/Postgres problem with Flask and SQLAlchemy, similar to Gordon Fierce. However, my solution was different. Postgres is strict about table locks and connections, so explicitly closing the session connection on teardown solved the problem for me.
My working code:
#pytest.yield_fixture(scope='function')
def db(app):
# app is an instance of a flask app, _db a SQLAlchemy DB
_db.app = app
with app.app_context():
_db.create_all()
yield _db
# Explicitly close DB connection
_db.session.close()
_db.drop_all()
Reference: SQLAlchemy
To answer the question "How would I go about debugging something like that?"
Run with py.test -m trace --trace to get trace of python calls.
One option (useful for any stuck unix binary) is to attach to process using strace -p <PID>. See what system call it might be stuck on or loop of system calls. e.g. stuck calling gettimeofday
For more verbose py.test output install pytest-sugar. pip install pytest-sugar And run test with pytest.py --verbose . . .
https://pypi.python.org/pypi/pytest-sugar
I had a similar problem with pytest and Postgresql while testing a Flask app that used SQLAlchemy. It seems pytest has a hard time running a teardown using its request.addfinalizer method with Postgresql.
Previously I had:
#pytest.fixture
def db(app, request):
def teardown():
_db.drop_all()
_db.app = app
_db.create_all()
request.addfinalizer(teardown)
return _db
( _db is an instance of SQLAlchemy I import from extensions.py )
But if I drop the database every time the database fixture is called:
#pytest.fixture
def db(app, request):
_db.app = app
_db.drop_all()
_db.create_all()
return _db
Then pytest won't hang after your first test.
Not knowing what is breaking in the code, the best way is to isolate the test that is failing and set a breakpoint in it to have a look. Note: I use pudb instead of pdb, because it's really the best way to debug python if you are not using an IDE.
For example, you can the following in your test file:
import pudb
...
def test_create_product(session):
pudb.set_trace()
# Create the Product instance
# Create a Price instance
# Add the Product instance to the session.
...
Then run it with
py.test -s --capture=no test_my_stuff.py
Now you'll be able to see exactly where the script locks up, and examine the stack and the database at this particular moment of execution. Otherwise it's like looking for a needle in a haystack.
I just ran into this problem for quite some time (though I wasn't using SQLite). The test suite ran fine locally, but failed in CircleCI (Docker).
My problem was ultimately that:
An object's underlying implementation used threading
The object's __del__ normally would end the threads
My test suite wasn't calling __del__ as it should have
I figured I'd add how I figured this out. Other answers suggest these:
Found usage of pytest-timeout didn't help, the test hung after completion
Invoked via pytest --timeout 5
Versions: pytest==6.2.2, pytest-timeout==1.4.2
Running pytest -m trace --trace or pytest --verbose yielded no useful information either
I ended up having to comment literally everything out, including:
All conftest.py code and test code
Slowly uncommented/re-commented regions and identified the root cause
Ultimate solution: using a factory fixture to add a finalizer to call __del__
In my case the Flask application did not check if __name__ == '__main__': so it executed app.start() when that was not my intention.
You can read many more details here.
In my case diff worked very slow on comparing 4 MB data when assert failed.
with open(path, 'rb') as f:
assert f.read() == data
Fixed by:
with open(path, 'rb') as f:
eq = f.read() == data
assert eq
I wrote this simple prototype client to send commands to a server I'm developing. It works perfectly running in GHCi, but the compiled version buffers everything typed in until I type in "quit" and the program exits. At that point all the input text gets sent.
What am I doing wrong? And why is it different when compiled?
Update: it does work as expected if compiled with ghc Main.hs. The problem happens when compiled with Leksah via Package -> Build. Anyone know how to get the command line Leksah is using?
System info: OSX 10.6, GHC 7.0.3, network 2.3.0.2
module Main (
main
) where
import System.IO
import Network
main = do
hServer <- connectTo "localhost" (PortNumber 7000)
hSetBuffering hServer NoBuffering
loop hServer
hClose hServer
where loop :: Handle -> IO ()
loop hServer = do
s <- getLine
hPutStrLn hServer s
case s of "quit" -> return ()
otherwise -> loop hServer
Leksah uses "cabal build", older versions "runhaskell Setup build".
Hmm, it seems Leksah wasn't actually building the app when I thought it was. I must have been running old code without the hSetBuffering call. A clean & rebuild has sorted it out. Apologies and thanks to everyone for your time and help.
edit: Found it - in case anyone else gets confused by this, when package->build is clicked, Leksah does not generate a compiled app if it's in debug/ghci mode.