How do you make unit tests for the HTML output of your PHP functions/scripts, specifically to check that the output is HTML5 valid?
Currently a can test functionality in PHPUnit and presentation with online copy/paste validators. But it would be much nicer if this could be integrated into the PHPUnit testing.
Is there a standard way to go about such things, or is it mainly a matter of using regular unit tests on functions which create the inserted content, and then making sure it looks correct in the browser/W3C Validator?
Similar question for older version of PHPUnit that no longer applies:
Unit tests for HTML Output?
What you looking for is behavior testing. Take a look at Behat
The Twine project (http://twineproject.sourceforge.net/doc/phphtml.html) replaces the copy/paste manual process. It might be useful; it still sends the HTML to the w3C site each time, which is not ideal for unit tests. (The W3C says all their stuff is open source and so you might be able to download it and run it locally... I couldn't find the download link though!)
An alternative approach is to use DomDocument::validate() However it requires the DTD to be referenced inside, and as this answer https://stackoverflow.com/a/15245834/841830 explains, HTML5 has no DTD.
(I'm assuming you meant that you have functions that return HTML5 strings, and you want to unit test those functions: if you want to test the whole output of a web app, e.g. as run through Apache and seen in a browser, I would use CasperJS or Selenium. But this is high-level functional testing, notably slower to run than unit tests, so I recommend to unit test what can be unit tested: and I still cannot find an offline HTML5 validator for Casper/Phantom/Slimer nor for Selenium!)
Related
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
Background I: *.phpt in phpunit
I recently read an article about *.phpt support in PhpUnit:
https://www.moxio.com/blog/32/start-testing-with-phpt-tests-in-phpunit
The big advantages here:
Supported out of the box with phpunit, no need for custom code to discover and deal with those files.
Common format that everybody seems to agree upon.
One can directly specify a specific *.phpt as a cli parameter.
A typical *.phpt file would look like this:
--TEST--
Basic arithmetic - addition
--FILE--
<?php
var_dump(42 + 1);
?>
--EXPECT--
int(43)
Background II: Self-updating fixtures
Earlier than that, I read about unit tests that can update (= overwrite) their own fixture files if a cli variable is present:
https://tomasvotruba.com/blog/2020/07/20/how-to-update-hundreds-of-test-fixtures-with-single-phpunit-run/
The following cli command would then overwrite the "expected" part in fixtures for failing tests:
$ UPDATE_TESTS=1 ./vendor/bin/phpunit
One would then review the git diff, and either accept the changes as desired functionality change, or fix the behavior.
The format of fixture files proposed in this article is very similar to the *.phpt file format, but in the end the format can be completely arbitrary and project-specific.
Benefits:
Start with empty "--EXPECT--" section, and have this part generated automatically.
Automatically update tests after behavior change, e.g. after a bug was fixed, or when BC-breaking changes were introduced.
Frequently update fixtures during early development, when behavior is not really stable yet.
No more copy/paste from cli diff output.
Disadvantages:
No built-in support in phpunit. Requires custom code. This does not scale well, if I want to do it in multiple projects.
One cannot target the *.phpt file as a cli parameter.
Question
Is there a built-in way to auto-update those *.phpt test files with phpunit?
Or should I perhaps use something other than phpunit?
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
I've two files test_utils.r and test_core.r, they contain tests for various utilities and some core functions separated into different 'context'. I can control the flow of tests within each file by moving around my test_that() statements.
But am looking for a way in which I can create different workflows, say ensuring that at run time, tests from Context A_utils runs first followed by tests from Context B_Core followed by context B_Utils.
Any ideas on how this can be achieved?
BrajeshS,
I have an idea. Have you tried the skip() function in version 0.9 or better? See testthat documentation on page 7:
Description
This function allows you to skip a test if it’s not currently available.
This will produce an informative message, but will not cause the test suite
to fail.
It was introduced to skip tests if an internet connection or API not available. You could then dependent on your workflow, jump over tests.
To see example code using skip_on_cran, look at wibeasley's answer where he provides test code in Rappster's reply - https://stackoverflow.com/a/26068397/4606130
I am still getting to grips with testthat. Hope this helps you.
I've got the interesting task to build complex workflow tests for an existing web application which has never had unit or integration tests (all testing by developers / users without structure or guidelines).
Setup
The target is an ASP.NET (no MVC) web application, that has been build over several years. There is no clean MVC or any other pattern, so the HTML output is very bad (generated IDs, not much css classes, in line css styles
--> hard to test).
The application is data-centric, so there is a lot to test and the status of the database is very important for the tests.
I'm thinking of the following workflow:
1. Reset Database to test start data
2. Run Tests which create and test data by simulation user interactions
Tools
I was playing around with Selenium IDE, but it does not feel structured enough.
I'm dreaming of a toolset, where I can write tests in (literate) coffeescript / javascript, that can be executed in the browser (no need for headless testing), but all tools I can find aim to test javascript functions and not user interactions.
My test steps need to be:
page loads
test if page has loaded correctly by searching for text A or counting elements in ul B
click on button C, wait for popup D, type text in field E
click save button and check if text from field E is shown in element F
Is jasmine able to test these kind of user interactions? I can only find jasmine tests for javascript functions or the existance of HTML elements, but not for complex workflows with test steps that rely on each other.
Thanks in advance!
Selenium IDE is not the way to go for this.
Selenium is built on the WebDriver JSON Wire Protocol. Having this 'base' means it has been very easily plugged into many many different languages, all using the same kind of API.
One of them being JavaScript:
https://code.google.com/p/selenium/wiki/WebDriverJs
Disclaimer: the JS Bindings are very new!
I'm not sure I understand why you must do it in JavaScript, especially as I can see this severely limits your options.
I suggest IBM Rational Functional Tester. It's java based, making it very extensible. It's not Javascript, so be warned that test are written in java code. For most of it there's a recorder: you simply click around and it records your actions.
Regarding automated tests with it, here's my opinions .
It is a commercial product. I'm not affiliated with IBM but I work with RFT a lot.
You may want to look at RIATest for cross-platform cross-browser testing of web applications. It uses ECMAScript based script language which is very similar to JavaScript.
It works on Windows and Mac, supported browsers are Firefox, IE and Chrome. Automated testing scripts written on one platform/browser can be run against all other supported platforms/browsers.
It is certainly possible to automate the steps that you described. Dynamically generated IDs are not good but you should be able to use other properties (such as text, element type, etc.) to identify the HTML objects that you want to automate and test for.
(Disclaimer: I am a RIATest team member).