I've just managed to get mutation testing working for the first time. My usual testing framework is Codeception but as of writing, it is not compatible with mutation testing (although I believe work is being done on it and it's not far off). I'm using PHPUnit and Infection, neither of which seem easy to work out how to use.
My test suite generated ten mutants. Nine were killed and one escaped. However, I don't know what part of the code or the tests needs to be improved to kill the final mutant.
How do you get information about what code allowed the mutant to escape?
I found in this blog what I couldn't find in Infection's documentation: the results are saved in infection.log.
The log file looks like this:
Escaped mutants:
================
1) <full-path-to-source-file>.php:7 [M] ProtectedVisibility
--- Original
+++ New
## ##
use stdClass;
trait HiddenValue
{
- protected function hidden_value($name = null, $value = null)
+ private function hidden_value($name = null, $value = null)
{
static $data = [];
$keys = array_map(function ($item) {
Timed Out mutants:
==================
Not Covered mutants:
====================
It says that the mutation changed the protected visibility to private and that no tests failed as a result. If this is important, I can now either change the code or write another test to cover this case.
Now that I've found this, I've searched on the Infection website for infection.log and found --show-mutations or -s which will output escaped mutants to the console while running.
Related
I have been using Postman so much and built a lot of useful things in it, but now need to implement one more thing.
Briefly:
Need to create test cases to test a correct records counting for every institution.
Now I handled it like that
Structure:
---------Collection
-------Folder
---Districts(folder)
-some folders with tests
---Colleges(folder)
-some folders with tests
---Schools(folder)
-Principal(Folder with test in Schools folder)
where these requests:
POST: Create a list
where in Tests:
var jList = JSON.parse(responseBody);
postman.setEnvironmentVariable("list_id", jList.data.id);
POST: Add some filters to list
GET: lists/{{list_id}} whete in "Test" code:
```
var allLists = JSON.parse(responseBody);
pm.test("test count", function () {
const value = allLists.data.count;
pm.expect(typeof value === 'number').to.eql(true);
pm.expect(value > 0 && value < 999999).to.eql(true);
}); // I check if the number(made in step№2 in a range we need
```
DELETE: delete this list
I guess I do 'overhead work' adding POST(create a list) and -DELETE endpoints in every folder, can I somehow bring it out to environment or variable and execute it before every POST: Add some filters to list and -DELETE endpoint after that
Something like general BeforeEach and AfterEach
Perhaps can I even bring this(below) out in one place for every separate test?
pm.test("test count", function () {
const value = allLists.data.count;
pm.expect(typeof value === 'number').to.eql(true);
pm.expect(value > 0 && value < 999999).to.eql(true);
});
Here is an example how it looks like if it not difficult, give me a piece of advice! Thanks
I want to write a script which compare two files in java and see there difference in html page ( side by side ), can someone help me out how to write ( where to start). I am pulling my hair out for this....
I want to use this script in beanshell postprocessor so that I can compare the standard output files with result files easily
I don't think you should be asking people for writing code for you here, consider hiring a freelancer instead.
Alternatively you can use the following approach:
Add JSR223 Assertion as a child of the request which you would like to fail if files won't be equal
Put the following code into "Script" area:
def file1 = new File('/path/to/file1')
def file2 = new File('/path/to/file2')
def file1Lines = file1.readLines('UTF-8')
def file2Lines = file2.readLines('UTF-8')
if (file1Lines.size() != file2Lines.size()) {
AssertionResult.setFailure(true)
AssertionResult.setFailureMessage('Files size is different, omitting line-by-line compare')
} else {
def differences = new StringBuilder()
file1Lines.eachWithIndex {
String file1Line, int number ->
String file2Line = file2Lines.get(number)
if (!file1Line.equals(file2Line)) {
differences.append('Difference # ').append(number).append('. Expected: ')
.append(file1Line).append('. Actual: ' + file2Line)
differences.append(System.getProperty('line.separator'))
}
}
if (differences.toString().length() > 0) {
AssertionResult.setFailure(true)
AssertionResult.setFailureMessage(differences.toString())
}
}
If there will be differences in files content you will see them listed one by one in the JSR223 Assertion
See Scripting JMeter Assertions in Groovy - A Tutorial for more details.
I'm currently implementing a SBT plugin for Gatling.
One of its features will be to open the last generated report in a new browser tab from SBT.
As each run can have a different "simulation ID" (basically a simple string), I'd like to offer tab completion on simulation ids.
An example :
Running the Gatling SBT plugin will produce several folders (named from simulationId + date of report generaation) in target/gatling, for example mysim-20140204234534, myothersim-20140203124534 and yetanothersim-20140204234534.
Let's call the task lastReport.
If someone start typing lastReport my, I'd like to filter out tab-completion to only suggest mysim and myothersim.
Getting the simulation ID is a breeze, but how can help the parser and filter out suggestions so that it only suggest an existing simulation ID ?
To sum up, I'd like to do what testOnly do, in a way : I only want to suggest things that make sense in my context.
Thanks in advance for your answers,
Pierre
Edit : As I got a bit stuck after my latest tries, here is the code of my inputTask, in it's current state :
package io.gatling.sbt
import sbt._
import sbt.complete.{ DefaultParsers, Parser }
import io.gatling.sbt.Utils._
object GatlingTasks {
val lastReport = inputKey[Unit]("Open last report in browser")
val allSimulationIds = taskKey[Set[String]]("List of simulation ids found in reports folder")
val allReports = taskKey[List[Report]]("List of all reports by simulation id and timestamp")
def findAllReports(reportsFolder: File): List[Report] = {
val allDirectories = (reportsFolder ** DirectoryFilter.&&(new PatternFilter(reportFolderRegex.pattern))).get
allDirectories.map(file => (file, reportFolderRegex.findFirstMatchIn(file.getPath).get)).map {
case (file, regexMatch) => Report(file, regexMatch.group(1), regexMatch.group(2))
}.toList
}
def findAllSimulationIds(allReports: Seq[Report]): Set[String] = allReports.map(_.simulationId).distinct.toSet
def openLastReport(allReports: List[Report], allSimulationIds: Set[String]): Unit = {
def simulationIdParser(allSimulationIds: Set[String]): Parser[Option[String]] =
DefaultParsers.ID.examples(allSimulationIds, check = true).?
def filterReportsIfSimulationIdSelected(allReports: List[Report], simulationId: Option[String]): List[Report] =
simulationId match {
case Some(id) => allReports.filter(_.simulationId == id)
case None => allReports
}
Def.inputTaskDyn {
val selectedSimulationId = simulationIdParser(allSimulationIds).parsed
val filteredReports = filterReportsIfSimulationIdSelected(allReports, selectedSimulationId)
val reportsSortedByDate = filteredReports.sorted.map(_.path)
Def.task(reportsSortedByDate.headOption.foreach(file => openInBrowser((file / "index.html").toURI)))
}
}
}
Of course, openReport is called using the results of allReports and allSimulationIds tasks.
I think I'm close to a functioning input task but I'm still missing something...
Def.inputTaskDyn returns a value of type InputTask[T] and doesn't perform any side effects. The result needs to be bound to an InputKey, like lastReport. The return type of openLastReport is Unit, which means that openLastReport will construct a value that will be discarded, effectively doing nothing useful. Instead, have:
def openLastReport(...): InputTask[...] = ...
lastReport := openLastReport(...).evaluated
(Or, the implementation of openLastReport can be inlined into the right hand side of :=)
You probably don't need inputTaskDyn, but just inputTask. You only need inputTaskDyn if you need to return a task. Otherwise, use inputTask and drop the Def.task.
I might be being thick here, but I can't figure out how to get a little info about why a test failed.
E.g. I have this test which is there to check that we haven't got any new methods with no test...
//TEST ALL METHODS TESTED
public function testAllMethodsTested()
{
$temp = $this->class_namespace.substr(__CLASS__, 0, -5);
$class_methods = get_class_methods($temp);
$test_methods = get_class_methods(__CLASS__);
foreach($class_methods as $class_method)
{
$this->assertEquals(true, in_array('test_'.$class_method, $test_methods));
}
}
When this fails, I get something like...
1) c_functions_core_ext_Test::testAllMethodsTested
Failed asserting that false matches expected true.
To make it easier to see where to go and fix some code, I would like to get some kind of helpful debug output in console and report.html (like you get for acceptance tests) along the lines of...
1) some_class_Test::testAllMethodsTested
Checking test exists for some_class::someMethod
Failed asserting that false matches expected true.
Is this possible for unit tests?
Yes, I am thick. It seems the assert functions allow for specifying your own message...
//TEST ALL METHODS TESTED
public function testAllMethodsTested()
{
$target_class = $this->class_namespace.substr(__CLASS__, 0, -5);
$class_methods = get_class_methods($target_class);
$test_methods = get_class_methods(__CLASS__);
$result = in_array('test_'.$class_method, $test_methods);
//FAIL WITH A USEFUL ERROR
$this->assertTrue($result, 'There is no test for '.$target_class.'::'.$class_method);
}
When this fails, I now get something like...
1) c_functions_core_ext_Test::testAllMethodsTested
There is no test for Namespace\target_class::someMethodName
Failed asserting that false matches expected true.
Upon compiling, I am getting the following error:
C:\UDK\UDK-2010-03\Development\Src\FixIt\Classes\ZInteraction.uc(41) : Error, Unrecognized member 'OpenMenu' in class 'GameUISceneClient'
Line 41 is the following:
GetSceneClient().OpenMenu("ZInterface.ZNLGWindow");
But when I search for OpenMenu, I find that it is indeed defined in GameUISceneClient.uc of the UDK:
Line 1507: exec function OpenMenu( string MenuPath, optional int PlayerIndex=INDEX_NONE )
It looks like I have everything correct. So what's wrong? Why can't it find the OpenMenu function?
From the wiki page on Legacy:Exec Function:
Exec Functions are functions that a player or user can execute by typing its name in the console. Basically, they provide a way to define new console commands in UnrealScript code.
Okay, so OpenMenu has been converted to a console command. Great. But still, how do I execute it in code? The page doesn't say!
More searching revealed this odd documentation page, which contains the answer:
Now then, there is also a function
within class Console called 'bool
ConsoleCommand(coerce string s)'. to
call your exec'd function,
'myFunction' from code, you type:
* bool isFunctionThere; //optional
isFunctionThere = ConsoleCommand("myFunction myArgument");
So, I replaced my line with the following:
GetSceneClient().ConsoleCommand("OpenMenu ZInterface.ZNLGWindow");
Now this causes another error which I covered in my other question+answer a few minutes ago. But that's it!
Not sure if this is your intent, but if you are trying to create a UIScene based on an Archetype that has been created in the UI Editor, you want to do something like this:
UIScene openedScene;
UIScene mySceneArchetype;
mySceneArchetype = UIScene'Package.Scene';
GameSceneClient = class'UIRoot'.static.GetSceneClient();
//Open the Scene
if( GameSceneClient != none && MySceneArchetype != none )
{
GameSceneClient.OpenScene(mySceneArchetype,LocalPlayer(PlayerOwner.Player), openedScene);
}