Mocking datetime.now() midway through a unittest.TestCase in Python 2.7 - python-unittest

I'm trying to test a bit of debouncing logic - these are local unittests I run for a Google App Engine webapp, using the 2.7 runtime environment. All my other tests are going well but this one has me stumped!
def testThat_emailDebouncingWorks(self):
# Do something, it triggers an email.
doSomething()
self.assertEqual(emails_sent, 1)
# Do something again, the new email is debounced.
doSomething()
self.assertEqual(emails_sent, 1)
# After an hour, the emails should start working again...
mockWaitingAnHour()
doSomething()
self.assertEqual(emails_sent, 2)
# ... and so should the debouncing.
doSomething()
self.assertEqual(emails_sent, 2)
The file under test logs the time an email was sent using datetime.now(), then reruns datetime.now() on all future attempts and returns early if under an hour has elapsed.
There are two things going wrong:
I think the unittest library only added mock support in 3.X, and I'm not keen on updating my whole app.
Even if I was using 3.X, all the examples I see are about faking a datetime response for your entire test case (using a mock decorator above the test def). Whereas I want to change that behaviour midway through my test, not for the entire case.
Any tips? Thanks in advance!

Okay, I got to the bottom of it and wanted to document the answers for anyone who finds this on Google ;)
1. Enable mocking on AppEngine for Python 2.7
You need to follow the instructions for copying a third party library (in our case, "mock") from the official docs. It's worth noting that on Ubuntu, the suggested command:
pip install -t lib/ mock
Will fail. You'll get an error like this:
DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base
This is to do with a weird conflict with Ubuntu which seems to have gone unfixed for years, and you'll see a lot of people suggesting a virtualenv workaround. I added the --system flag instead:
pip install --system -t lib/ mock
and it worked fine. Remember to follow the rest of the instructions with appengine_config, and you should be set. "import mock" is a good way to check.
2. Mocking the datetime.now() call
My module under test uses:
from datetime import datetime
In my test module, import some stuff:
from mock import patch, Mock
import my_module #Also known as my_module.py
import datetime
Then the actual test case:
#patch.object(my_module, 'datetime', Mock(wraps=datetime.datetime))
def testThat_myModule_debouncesEmails(self):
fake_time = datetime.datetime.now()
# This is the first time the thing happened. It should send an email.
doSomething()
self.assertEqual(1, emails_sent)
# Five minutes later, the thing happens again. Should be debounced.
fake_time += datetime.timedelta(minutes=5)
my_module.datetime.now.return_value = fake_time
doSomething()
self.assertEqual(1, emails_sent)
# Another 56 minutes pass, the thing happens again. An hour has elapsed, so don't debounce.
fake_time += datetime.timedelta(minutes=56)
my_module.datetime.now.return_value = fake_time
doSomething()
self.assertEqual(2, emails_sent)
# Give it another 15 minutes to check the debouncing kicks back in.
fake_time += datetime.timedelta(minutes=15)
my_module.datetime.now.return_value = fake_time
doSomething()
self.assertEqual(2, emails_sent)
Hope this helps someone!

Related

AIRFLOW DAG for uploading files fails when files not there

I have an airflow DAG that works perfectly when files are present, but error->fails when the source files are not there.
Randomly, I recieve files from a given source, that my DAG picks up and processes. While I need to run the DAG daily, files are not necessarily there daily. Could be monday, wednesday, or even sunday evening.
I'm not worried about days with no new files. i worry about days when new files come and it breaks.
How do I tell the DAG that when no file exist then gracefully exit with success?
My DAG below (please ignore schedule setting. I'm still in development mode):
import airflow
from airflow import models
from airflow.operators.gcs_to_bq import GoogleCloudStorageToBigQueryOperator
from airflow.operators.gcs_to_gcs import GoogleCloudStorageToGoogleCloudStorageOperator
args = {
'owner': 'Airflow',
'start_date': airflow.utils.dates.days_ago(2),
'email': ['email#gmail.com'],
'email_on_failure': True,
'schedule_interval': 'None',
}
dag = models.DAG(
dag_id='Source1_Ingestion',
default_args=args
)
# [START load ATTOM File to STAGING]
load_File_to_Source1_RAW = GoogleCloudStorageToBigQueryOperator(
task_id='Source1_GCS_to_GBQ_Raw',
bucket='Source1_files',
source_objects=['To_Process/*.txt'],
destination_project_dataset_table='Source1.Source1_RAW',
schema_fields=[
{'name': 'datarow', 'type': 'STRING', 'mode': 'NULLABLE'},
],
field_delimiter='ยง',
write_disposition='WRITE_TRUNCATE',
google_cloud_storage_conn_id='GCP_EDW_Staging',
bigquery_conn_id='GCP_EDW_Staging',
dag=dag)
# [END howto_operator_gcs_to_bq]
# [START move files to Archive]
archive_attom_files = GoogleCloudStorageToGoogleCloudStorageOperator(
task_id='Archive_Source1_Files',
source_bucket='Source1_files',
source_object='To_Process/*.txt',
destination_bucket='Source1_files',
destination_object='Archive/',
move_object=True,
google_cloud_storage_conn_id='GCP_EDW_Staging',
dag=dag
)
# [END move files to archive]
load_File_to_Source1_RAW.set_downstream(archive_Source1_files)
One way to approach this would be to add a Sensor Operator to the workflow.
Nehil Jain describes sensors nicely:
Sensors are a special kind of airflow operator that will keep running until a certain criterion is met. For example, you know a file will arrive at your S3 bucket during certain time period, but the exact time when the file arrives is inconsistent.
For your use case, it looks like there's a Google Cloud Sensor, which "checks for the existence of a file in Google Cloud Storage." The reason you'd incorporate a sensor is that you're decoupling the operation "determine if a file exists" from the operation "get the file (and do something with it)".
By default, sensors have two methods (source):
poke: the code to run at poke_interval times, which tests to see if the condition is true
execute: use the poke method to test for a condition on a schedule defined by the poke_interval; fails out when the timeout argument is reached
In a common file-detection sensor, the operator receives instructions to check a source for a file on a schedule (e.g. check every 5 minutes for up to 3 hours to see if the file exists). If the sensor succeeds in meeting its test condition, it succeeds and allows the DAG to continue downstream to the next operator(s). If it fails to find the file, it times out and the sensor operator is marked failed.
With just a sensor operator, you've already succeeded in separating the error cases - the DAG fails at the GoogleCloudStorageObjectSensor instead of the GoogleCloudStorageToBigQueryOperator when the file doesn't exist, and fails at the GoogleCloudStorageToBigQueryOperator when something is wrong with the transfer logic. Importantly for your use case, Airflow supports a soft_fail argument, which "mark[s] the task as SKIPPED on failure"
For this next part, I'll caveat this next part by explicitly stating that I'm not intimately familiar with the GoogleCloudStorage operators. If the operator doesn't allow wildcarding in the sensor, you may need to rewire the sensor's poke method to allow for more complex, pattern based file detection. This is where Airflow's plug-in architecture can really shine, allowing you to modify and extend existing operators to meet your exact needs.
The example I'll give you here is that the SFTPSensor only supports poking for a specific file out of the box. I needed wildcard based poking, so I wrote a plugin that modifies the SFTPSensor to support regular expressions in file identification. In my case, it was just modifying the poke to switch from polling for the existence of a single file to polling a list of files and then passing it through a regular expression to filter the list.
At a cursory glance, it looks like the way that the GoogleCloudStorageSensor pokes for an object is with the hook.exists method. I can't speak to whether a wildcard would work there, but if it doesn't, it looks like there's a hook.list method which would allow you to implement a similar workflow to what I did for the SFTPRegexSensor.
I've included some of the source code for the SFTPRegexSensor Plugin's poke method, modified for how I think it'd work with GCS in case it's helpful:
def poke(self, context):
# create a hook (removed some of the SSH/SFTP intricacies for simplicity)
# get list of file(s) matching regex
files = hook.list(self.bucket, self.prefix) # you need to define operator paramters for the choices that are dynamic in the operator's poke (e.g. which bucket, what the file prefix is); swapped in the GCS args
regex = re.compile(self.remote_filename)
files = list(filter(regex.search, files))
if not files:
return False
return True

How to fake DateTime in a phpunit test?

ssI write a unittest using Symfony's KernelTestCase and have to test for a functionality, which happens only at a certain tie of the day (extra early or extra late).
So when I let my test run a noon of course nothing happens. How can I fake my system time to pretend that it has a different time and my test case is triggered.
I tried around with Symfony's ClockMock class but it does not work.
https://symfony.com/doc/current/components/phpunit_bridge.html#clock-mocking
This is my test code:
use Symfony\Bridge\PhpUnit\ClockMock;
use \DateTime;
/**
* testUserAchievedEarlyBirdTrophy
* #group time-sensitive
*/
public function testUserAchievedEarlyBirdTrophy()
{
ClockMock::withClockMock(strtotime('2018-11-05 01:00:00'));
echo (new DateTime())->format('Y-m-d H:m:s');
$user8Id = $this->user8->getId();
$progressSaveRequest = new ProgressSaveRequest($user8Id, $this->content_1_1->getId());
$this->progressService->saveProgress($progressSaveRequest);
$this->assertTrue($this->loggerCreator->hasDebugThatContains(
'Early Bird'
));
}
the echo give me today's date: 2019-02-01 16:02:06
I also had the feeling that ClockMock is rather used to skip time for e.g. testing caching instead of sleep().
What am I doing wrong?
The listener configuration is in place in my phpunit.xml
Calling bin/simple-phpunit causes a lot of installation happening.
Can't I use the normal phpunit?
Are there any other options to fake the time of the day?
Following the link you included in your post, the first paragraph says:
The ClockMock class provided by this bridge allows you to mock the
PHP's built-in time functions time(), microtime(), sleep() and
usleep(). Additionally the function date() is mocked so it uses the
mocked time if no timestamp is specified. Other functions with an
optional timestamp parameter that defaults to time() will still use
the system time instead of the mocked time.
(Emphasis added.)
This means that your call of
echo (new DateTime())->format('Y-m-d H:m:s');
is expected to give the system time, not the mocked time.
Change it to
echo date('Y-m-d H:m:s');
in order to match the requirements of ClockMock and get the mocked time.
Note: I've never used ClockMock myself, but just looking at the documentation this should be a first step to see if this resolves your problem.

Alexa Echo Dot - ASK skill problems

I'm tying to make a simple test custom Alexa Skill, but I'm stuck and I'm not sure what the problem is. Maybe someone more experienced know what I'm missing?
Invocation Name
home system
Intent Schema
{
"intents": [
{
"intent": "AMAZON.HelpIntent",
"slots": []
},
{
"intent": "TestIntent",
"slots": [
{"name": "test", "type": "AMAZON.NUMBER"}
]
}
]
}
Sample Utterances
TestIntent set state {test}
TestIntent add state
I have written my own little python server on my own self hosted server, I already have a working news flash skill on the same system. I have spend plenty of time looking at the documentation, reading tutorials and I looks like I have done what I'm supposed to do.
The result I get is this:
A LaunchRequest works, both in the Service Simulator and on the Echo. It triggers a HTTP POST with the expected JSON, and I get the expected voice reply.
But the IntentRequest only works from the Service Simulator, it never works on the Echo. I say for example "alexa home system set state eight", no requests are made to my server, the echo just makes a sound and that's all.
I have no idea how to debug this, the skill is a US skill and my Echo is in US mode. I have tried to set the endpoint in both Europe and North America. Tried different trigger words, different slots, no slots ... and I have of course checked under Settings -> History to make sure that the device understood me correctly.
Any idea what to try next? How to debug this?
I found the problem, it was a classic PEBCAK (Problem Exists Between Chair And Keyboard) problem.
I had missed that I had to be much more precise how to invoke an intent (a single sentence that contains both the trigger word and intent in one go). A example of valid and working examples are:
Alexa, ask home system to set state nine
Alexa, set state twelve using home system
Alexa, tell home system set state one
I realised this when I used the alternative 2-step invoking, and realized that it worked. It had to be the way I invoked the skill, not the backend:
Alexa, open home system
(Alexa responds, and listens for the command)
Set state to eight
(Intent triggered, Alexa responds)
The first request above is the LaunchRequest
The LaunchRequest responds with shouldEndSession: false, if not the session will end. That's maps to question(...) in my code.
There are plenty of more ways to trigger the skills, a full list see this page: https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/supported-phrases-to-begin-a-conversation (scroll down to the tables)
Finally thank you u-gen for the feedback, bst was a interesting project (never tried it), guess it can be really useful if you uses a hosted solution like lambda. But thanks to the docs I found flask-ask, a project that simplified my code.
Finally, the python part of my test project if someone else like to try it out.
#!/usr/bin/env python
from flask import Flask, render_template
from flask_ask import Ask
from flask_ask import statement, question, convert_errors
app = Flask(__name__)
ask = Ask(app, '/ask/')
#app.route('/')
def hello_world():
return 'Hello, World!'
#ask.launch
def launched():
return question('Welcome to Foo')
#ask.intent('TestIntent')
def hello():
return statement('Hello, world')
#ask.session_ended
def session_ended():
return "", 200
if __name__ == '__main__':
app.run(debug=True, host="0.0.0.0", threaded=True)

How to correctly unit test (using nose) a sqlalchemy Model by creating a new database

I currently have an application using flask-sqlalchemy. My model is connected to a postgresql database, and now I would like to write unit tests (using nose). I was told to use SQLite to create a new database for testing, and after a lot of searching (and looking at the texting section on the flask-sqlalchemy website) I'm still confused as how to do it. Each class in my model.py looks something like the following:
db = SQLAlchemy(app)
class Prod(db.Model):
__tablename__ = 'prod'
id = db.Column(db.Integer, primary_key=True)
desc = db.Column(db.String)
def __init__(self, id, desc):
self.id = id
self.desc = desc
My config.py:
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres://name:pass#server/db
and I would like to test my insert functions in a new file by setting up and tearing down a new database for each test. If anyone can give me some example code that would be great. Thanks!
I can't answer your specific question, but will provide some general advise:
You will find that setting up and tearing down the complete database for each test will be too slow. Imagine in the future when you might have hundreds of tests or even thousands.
The approach we take is:
For testing purposes we have a database populated with test data. We have a script which creates a fresh database and populates it with this test data.
We run this script prior to running our test suite. All tests can assume this data exists.
Each test may create additional records if necessary, but it is their responsibility to undo any changes they make (delete new records, undo changes) - in order words to leave the database in the same state as it was before the test began. This prevents tests from interfering with each other.
In a project I manage we have a test suite of 1070 tests which runs in about 5 minutes using this approach.
What if we had taken your approach? Let's assume that 50% of these tests actually exercise the database (and need a fresh reload). That's 1070 * .50 * 20 seconds for the reload / 3600 = 2.97 hours. Oops - that's far too slow to be useful.
Even at a much smaller scale though, you'll be much happier if your test suite runs in 1 minute instead of 20 minutes.

Qt or PyQt - check when file is used by another process. Wait until finish copy

Good morning,
What is the best strategy for check when a big file o big directory has finished to copy?
I want wait until a file has finish fully to copy. Is there a code example in q
I'm working on mac os x.
thanks
Update
I use QFileSystemWatcher. the problem is that I receive file or directory change notification when o copy it is in progress. So user copy a big folder (inside many files), the operating system copy process start, it take 5 minuts, but in same times my application receive file changed notification. This is a problem because when i receive a change notification my application start for doing some operations on that files, but the copy is already in progress!!!!
There is only one reliable way to do this: Change the copy process to write to temporary files and then rename them after the copy is finished.
That way, you can ignore new files which end with .tmp and rename is an atomic operation.
If you can't change the copy process, all you can do is add a timer to wait for, say, half an hour to make sure the copy is really finished.
A more fine grained (and more risky) approach is to add a loop that check the file size and stops when the file size doesn't change for a certain time but that's also hard to get right.
Worse, this doesn't prevent you from reading partial files (when the copy process was terminated in the middle).
I think that the QFileSystemWatcher is the right start for you to get to the point of monitoring for changes, but as you have found, these changes are ANY changes. From this point, I think it should be easy enough for you to just check the modification time of the file.
Here is a simple example of a Watcher class that will let you specify a file to monitor and see if it has been modified after a given time. It can run a callback or emit a signal that anyone can watch:
import os.path
import time
from PyQt4 import QtCore
class Watcher(QtCore.QObject):
fileNotModified = QtCore.pyqtSignal(str)
MOD_TIME_DIFF = 5 #seconds
def __init__(self, aFile, callback=None, checkEvery=5):
super(Watcher, self).__init__()
self.file = aFile
self.callback = callback
self._timer = QtCore.QTimer(self)
self._timer.setInterval(checkEvery*1000)
self._timer.timeout.connect(self._checkFile)
def _checkFile(self):
diff = time.time() - os.path.getmtime(self.file)
if diff > self.MOD_TIME_DIFF:
self._timer.stop()
self.fileNotModified.emit(self.file)
if self.callback:
self.callback()
def start(self):
self._timer.start()
def stop(self):
self._timer.stop()
An example of using it:
def callbackNotify():
print "Callback!"
def signalNotify(f):
print "Signal: %s was modified!" % f
# You could directly give it a callback
watcher = Watcher("/path/to/file.file", callback=callbackNotify)
# Or could use a signal
watcher.fileNotModified.connect(signalNotify)
# tell the watcher timer to start checking
watcher.start()
## after the file hasnt been modified in 5 seconds ##
# Signal: /path/to/file.file was modified!
# Callback!
Try using QtConcurrent framework.
In particular, check out QFuture and QFutureWatcher. You can execute asynchronous copy operations inside a QFuture object and monitor its progress through signals and slots with a watcher.
bool copyFunction() {
// copy operations here, return true on success
}
MyHandlerClass myObject;
QFutureWatcher<bool> watcher;
connect(&watcher, SIGNAL(finished()), &myObject, SLOT(handleFinished()));
QFuture<bool> future = QtConcurrent::run(copyFunction);
Since you have no control on the external application, my suggestion is that you lock the files while you work on them. In this way other programs will not be able to access them while locked.
Alternatively, if you have access to the other program's source, you should implement some form of inter process communication,via sockets, messages or whatever method you prefer.

Resources