We have a legacy codebase which we've just started adding unit test coverage. This means that while we have a massive codebase (1.5million lines, at least 600 classes, plus support, template files), we only have one or two unit tests in the entire application. (Sad but true).
My problem is that arcanist unit testing (either with arc unit or part of arc diff) takes over four minutes to run. PHPUnit, on the other hand takes only seconds.
Unit tests are supposed to be short, especially if it's only on the most recent diff. How do I speed these up, or, barring that, see what arcanist is trying to do (so I can optimize myself)?
My best guess is that this has to do with the fact that arcanist needs to look for a test file, while PHPUnit does not, but in that case I don't know how to make arcanist default to the right directory.
It turns out that this is actually caused not by the testing but by the coverage. By default Arcanist tries to provide code coverage reports:
PASS 14ms★ Test\Unicorn\Feeds\Somber\DubStep::testLoadBadType
PASS 5ms★ Test\Unicorn\Feeds\Somber\DubStep::testLoadBadJSON
PASS 8ms★ Test\Unicorn\Feeds\Somber\DubStep::testLoadBadSeasonWeek
PASS 148ms Test\Unicorn\Feeds\Somber\DubStep::testEmptyData
PASS 200ms Test\Unicorn\Feeds\Somber\DubStep::testCompleteLoad
PASS 5ms★ Test\Fantasy\Feeds\Somber\DubStep::testGetProjectionStats
PASS 5ms★ Test\Fantasy\Feeds\Somber\DubStep::testGetNumGamesPlayed
PASS 12ms★ Test\Unicorn\Feeds\Somber\DubStep::testIsStatBonus
PASS 140ms Test\Fantasy\Feeds\Somber\DubStep::testIsPlayerUpdatedBaseConditions
PASS 7ms★ Test\Fantasy\Feeds\Somber\DubStep::testCalcNormalizedWeekProjection
COVERAGE REPORT
0% tests/feeds/lib/Feeds/Somber/Extension/usfoak-qub-te-hk3.json
0% tests/feeds/lib/Feeds/Somber/DubStep.php
93% feeds/lib/Feeds/Somber/PlayerProjectedStats.php
The coverage it runs a complete coverage report against the entire repository, and it doesn't cache the result. This means each file in each arc diff has a coverage report run. In a small project that's not a big deal, but in a large project that makes a major difference. The best way to avoid this is use phpunit coverage report directly and use --no-coverage flag in all of your arc diffs.
Related
I am using python's unittest framework. Since each test case execution will take the same amount of time [obtain the product log to determine whether it is successful or not], I want to extract the time-consuming operation to determine the success/failure of the test and put it in tearDownClass In this method, the product logs are obtained uniformly, and the success of each use case is compared one by one, which can save a lot of test execution time,
The core requirement is: in tearDownClass or elsewhere, finally modify the execution results of each test case one by one, changing success to failure
(Disclaimer: I'm quite new to Tosca but not to testing in general)
DESCRIPTION
I'm automating a very dynamic page in Tosca, with content that is added (or not) as you progress through a form.
I do not have a testsheet at hand and do not have time to create one so I cannot use a template and the 'conditions' (I'm using TC-Parameters and they do not seem to apply to the 'Condition' column).
I wanted to use libraries as much as possible because most of the steps are the same, and there are a LOT of possible outcomes (I have more than 100 TCs to automate) so I'm trying to have my steps as 'generic' as possible, so that if the interface is changed in the future I'll be ale to maintain most of it 'centrally'.
PROBLEM
I've added four 'ifs' in strategic points. The problem is that an unsuccessful 'if' seems to hang for 10s, no matter what I use inside: 'verify' takes 10s & 'waiton' also takes 10s (although for the latter I modified the settings to 5s so I don't understand why).
I'm actually not interested in 'verify' waiting at all. I know that the content has either to be there or not at the precise moment where I have my condition. I'd be happy with a 1s delay, that'd be more than enough time for the app to display the content.
The TCs duration varies between 1m and 1m40s (4*10s if my 4 'if' are negative). It'd be great if I could speed it up, especially because most of the 'ifs' will NOT trigger. Any ideas?
You could try checking some settings regarding how long Tosca waits for Verify/Waiton:
Settings > Engine > Waiton > Maximum Wait Duration
Settings > TBox > Synchronization > Synchronization Time out
However, I've also found using buffered values to be more efficient time-wise for some of my testing scenarios.
I solved it by adding TCPs and buffering them, in order to check them as conditions (not sure why Tosca requires the intermediate buffering step but that does the trick). Now I can configure whether my tests should expect a given follow-up item or not. It's a lot of extra configs but at least my tests are nicely modular.
My JSR352 batch job needs to read from a database, and then depending on the result flows to one of two pathways, each of which involves some more if/else scenarios. I wonder what the pros and cons between writing a single step with a large batchlet and several steps consisting of smaller batchlets would be. This job does not involves chunk steps with chunk size larger than 1, as it needs to persists the read result immediately in case there is any before proceeding to other logic. The job will be run using Control-M, I wonder if using multiple smaller steps provides more control points.
From that description, I'd suggest these
Benefits of more, fine-grained steps
1. Restart
After a job failure, the default behavior on restart is to begin executing at the step where the previous job execution failed. So breaking the job up into more steps allows you to avoid writing the logic to resume where you left off and avoid re-processing, and may save execution time in the process.
2. Reuse
By encapsulating a discrete function as its own batchlet, you can potentially compose other steps in other jobs (or even later in this job) implemented with this same batchlet.
3. Extract logic into XML
By moving the transition logic into the transition elements, and extracting the conditional flow (e.g. <next on="RC1" to="step3"/>, etc.)
into the job definition XML (JSL), you can introduce changes at a standard control point, without having to go into the Java source and find the right place.
Final Thoughts
You'll have to decide if those benefits are worth it for your case.
One more thought
I wouldn't automatically rule out the chunk step just because you are using a 1-item chunk, if you can still find benefits from the checkpointing or even possibly the skip/retry. (But that's probably a separate question.)
I have 2 questions on the case recorder.
1- I am not sure how to restart an optimizaiton from where the recorder left off. I can read in the case reader sql file etc but can not see how this can be fed into the problem() to restart.
2- this question is maybe due to my lack of knowledge in python but how can one access to the iteration number from within an openmdao component (one way is to read the sql file that is constantly being updated but there should be a more efficient way.)
You can re-load a case back via the load_case method on the problem.
See the docs for it here.
Im not completely sure what you mean by access the iteration count, but if you just want to know the number of times your components are called you can add a counter to them yourself.
There is not a programatic API for accessing the iteration count in OpenMDAO as of version 2.3
I am using Visual Studio TS Load Test for running WebTest (one client/controls hitting one server). How can I configure goal based load pattern to achieve a constant test / second?
I tried to use the counter 'Tests/Sec' under 'LoadTest:Test' but it does not seem to do anything.
I've recently tested against the Tests / Sec, and I've confirmed it working.
For the settings on the Goal Based Load Pattern, I used:
Category: LoadTest:Test
Counter: Tests/Sec
Instance: _Total
When the load test starts, verify it doesn't show an error re: not being able to access that Performance Counter.
Tests I ran for my own needs:
Set Initial User Load quite low (10), and gave it 5 minutes to see if
it would reach the target Tests / Sec target, and stabilise. In my case, it stabilised after about 1 minute 40.
Set the Maximum User Count [Increment|Decrement] to 50. Turns out the
user load would yo-yo up and down quite heavily, as it would keep
trying to play catch-up. (As the tests took 10-20 seconds each)
Set the Initial User Load quite close to the 'answer' from test 1,
and watched it make small but constant adjustments to the user
volume.
Note: When watching the stats, watch the value under "Last". I believe the "Average" is averaged over a relatively long period, and may appear out of step with the target.