I might be being thick here, but I can't figure out how to get a little info about why a test failed.
E.g. I have this test which is there to check that we haven't got any new methods with no test...
//TEST ALL METHODS TESTED
public function testAllMethodsTested()
{
$temp = $this->class_namespace.substr(__CLASS__, 0, -5);
$class_methods = get_class_methods($temp);
$test_methods = get_class_methods(__CLASS__);
foreach($class_methods as $class_method)
{
$this->assertEquals(true, in_array('test_'.$class_method, $test_methods));
}
}
When this fails, I get something like...
1) c_functions_core_ext_Test::testAllMethodsTested
Failed asserting that false matches expected true.
To make it easier to see where to go and fix some code, I would like to get some kind of helpful debug output in console and report.html (like you get for acceptance tests) along the lines of...
1) some_class_Test::testAllMethodsTested
Checking test exists for some_class::someMethod
Failed asserting that false matches expected true.
Is this possible for unit tests?
Yes, I am thick. It seems the assert functions allow for specifying your own message...
//TEST ALL METHODS TESTED
public function testAllMethodsTested()
{
$target_class = $this->class_namespace.substr(__CLASS__, 0, -5);
$class_methods = get_class_methods($target_class);
$test_methods = get_class_methods(__CLASS__);
$result = in_array('test_'.$class_method, $test_methods);
//FAIL WITH A USEFUL ERROR
$this->assertTrue($result, 'There is no test for '.$target_class.'::'.$class_method);
}
When this fails, I now get something like...
1) c_functions_core_ext_Test::testAllMethodsTested
There is no test for Namespace\target_class::someMethodName
Failed asserting that false matches expected true.
Related
I've just managed to get mutation testing working for the first time. My usual testing framework is Codeception but as of writing, it is not compatible with mutation testing (although I believe work is being done on it and it's not far off). I'm using PHPUnit and Infection, neither of which seem easy to work out how to use.
My test suite generated ten mutants. Nine were killed and one escaped. However, I don't know what part of the code or the tests needs to be improved to kill the final mutant.
How do you get information about what code allowed the mutant to escape?
I found in this blog what I couldn't find in Infection's documentation: the results are saved in infection.log.
The log file looks like this:
Escaped mutants:
================
1) <full-path-to-source-file>.php:7 [M] ProtectedVisibility
--- Original
+++ New
## ##
use stdClass;
trait HiddenValue
{
- protected function hidden_value($name = null, $value = null)
+ private function hidden_value($name = null, $value = null)
{
static $data = [];
$keys = array_map(function ($item) {
Timed Out mutants:
==================
Not Covered mutants:
====================
It says that the mutation changed the protected visibility to private and that no tests failed as a result. If this is important, I can now either change the code or write another test to cover this case.
Now that I've found this, I've searched on the Infection website for infection.log and found --show-mutations or -s which will output escaped mutants to the console while running.
I have written the following spec. Surprisingly it passes even though the result doesn't match
code snippet
val controller = new UserController(mockUserRepository,mockControllerComponents,mockSilhouette)
//val request = FakeRequest[AnyContentAsJson]("POST", "/ws/users/signup").withJsonBody(Json.parse("""{"bad": "field"}"""))//FakeRequest(POST,"/ws/users/signup").withJsonBody(Json.parse("""{"bad":"field"}"""));
val request = FakeRequest("POST","ws/users/signup")
println("sending request",request)
//val result = controller.someMethod()
val result = controller.signupUser(request)
Await.result(result,Duration("10 secs"))
result.map(response => {
println("response from controller:"+response)
response mustBe play.api.mvc.Results.BadRequest
})
console prints
(sending request,POST ws/users/signup)
print in controller. (received request,POST ws/users/signup)
Controller returns 200OK but I am matching it with BadRequest. Still the test passes!
response from controller:Result(200, Map())
I suspect that I am not matching the result correctly. I am doing response mustBe play.api.mvc.Results.BadRequest. I know that response is Results but BadRequest is Status. But I don'w know how else to match and also why the test doesn't fail. I also tried following and can see the the values are different but even then the test passes.
println("response from controller:"+response.header.status+" compared with "+play.api.mvc.Results.BadRequest.header.status)
response.header.status mustBe play.api.mvc.Results.BadRequest.header.status
console print - response from controller:200 compared with 400
Importing members of Helpers object like so
import play.api.test.Helpers._
gives access to various utility methods to extract results from Future[Result] in tests. For example, Helpers.status can be used to extract status code like so:
val result: Future[Result] = controller.signupUser(request)
status(result) mustBe BAD_REQUEST
Another option is to use whenReady from ScalaFutures:
val result: Future[Result] = controller.signupUser(request)
whenReady(result) { _.header.status mustBe BAD_REQUEST }
Another option is to extend AsyncWordSpec and then you can map over the Future like so:
val result: Future[Result] = controller.signupUser(request)
result.map(_.header.status mustBe BAD_REQUEST)
I have a small testing program that uses testthat
library(testthat)
source("src/MyFile.r")
results <- test_dir("tests", reporter="summary")
So, I am running this via Rscript. The problem is that the exit code is always 0 even when there were test failures. So, I want to call stop if there are any failures. But I can't seem to get the right bit of code to do that. Is there a method or field in results that I should be looking at to determine if there were any errors?
You could also tweak jamesatha's answer to retrieve the number of test failures.
failed.tests <- sapply(results, function(r) {
!is(r$result[[1]], "expectation_success")
})
You then allows you to fail, as before:
if (any(failed.tests)) {
stop("There were test failures")
}
or do something more bespoke like
if (any(failed.tests)) {
failed.test.count <- length(which(failed.tests))
stop(paste(failed.test.count,"failed tests is",failed.test.count,"too many!")
}
Currently, my solution is iterating through the results like this:
for (i in 1:length(results)) {
if (!is(results[[i]]$result[[1]], "expectation_success")) {
stop("There were test failures")
}
}
You can simply pass stop_on_failure=TRUE in as a parameter to test_dir. If you get any test failures it will raise an error and exit non zero.
e.g:
results <- test_dir("mypath", stop_on_failure=TRUE)
This is documented here
So this is a strange one. My code does a bunch a things that are hard to explain (but if necessary I´ll try to explain), but the following works:
var res = data.delete_if (function(key, value) { return key == "a"; })
but the following crashes:
data.delete_if (function(key, value) { return key == "a"; })
So, the fact that I do not save the result of the delete_if function crashes the browser with the following stack trace:
Error: test: B environment should proxy a Ruby hash. (MDArraySolTest): Java::JavaLang::IllegalStateException: Channel stream was closed before response has been received.
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
Any ideas of why this happens? Any solutions? I can provide more information if needed.
EDIT1:
Doing some more tests I found out that the error occurs only if the call to data.delete_if is the last statement on the script. If I add for example: console.log(""); after the call, everything works fine.
Thanks
can anyone tell me why this one line after the last break; stays uncovered and therefore the method get a coverage fo 0%!
private function isADateParameter($fieldName, $fieldValue){
if(!is_array($fieldValue)){
$this->addPayloadError('filter',320, $fieldName. 'is a date field and needs an array filter like {"gt":"2015-05-01"}');
return false;
}
foreach ($fieldValue as $criteria => $date){
$ar_date = explode('-', $date);
switch(false){
case in_array($criteria, ['gt', 'lt', 'gte', 'lte']):
$this->addPayloadError('filter',321, $fieldName. ' supports only constraints gt, lt, gte or lte');
break;
case preg_match("/^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])$/",$date):
$this->addPayloadError('filter',322, $fieldName. ' supports only date in format YYYY-MM-DD');
break;
case checkdate($ar_date[1], $ar_date[2], $ar_date[0]):
$this->addPayloadError('filter',323, $date. ' is not a real date');
break;
} //uncovered red line in html report
}
if(isset($this->payloadErrors['filter']) && count($this->payloadErrors['filter']) > 0){
return false;
}
return true;
}
See this bug report, specifially the close note:
This is not a bug. In theory "$value" could be something else than 1, 2 or 3 and in that case the switch() would not leave due to one of the values. Xdebug can not make assumptions on what "doSomething()" returns an has no way to know that all that ever is returned than 1, 2 or 3. The last } of each function and method includes an implicit "return NULL;" - which PHP sees as executable code. Because there is a possible code path where the } is not hit, it is marked as "not executed".
In your case, you have three case statements, that I imagine you've written tests for, but there's a third, implied path that you may or may not be testing for - what to do if none of the three cases match?
tl;dr: Add a default case to the switch, and make sure your tests exercise it. Or don't and ignore that coverage line.