When building R packages, we use testthat to write tests. We have 2 files: a test file for the specific package (specific.R), and one that we use to make sure all packages continue to work together and the overall result is fine (overall.R). Both tests are currently run when we push to github or create a PR through Travis, which implicitly runs this line of code(R CMD check *tar.gz). check runs all the tests in the test folder, and thus both files are run.
Now, I'm a bit new to testing... but it seems that we have more or less recreated the difference b/w a unit test and an integration test.
Considering that the tests for overall.R do take a lot longer to run, we would like to restrict it so that they only run when we do a pull-request to the package (when we have introduced new functionality on a different dev branch) while the package-specific tests keep running every time we commit/push to the repo.
Is this possible in github or Travis?
You could put your overall.R script into a different directory and then specify this folder as the new tests directory for pull-request hooks, but this will then only run your integration tests and not the unit tests. See R CMD check --help
R CMD check --test-dir integration_tests package.tar.gz
Related
R CMD check automatically runs tests located in tests/ directory. However running the tests this way requires building the package first. After that R CMD check goes through various different sanity checks before finally reaching the tests at the end.
Question: Is there a way to run those tests without having to build or install the package first?
NOTE: without using testthat or other non-standard packages.
To summarise our discussion.
To my knowledge there is no standard alternative to R CMD check for unit testing provided by base R
Typically for unit testing, I source everything under R/ (and dyn.load everything under source/) and then source everything under tests/ (actually, I also use the Example sections of the help pages in the man/ directory as test cases and compare their outcome to those from previous package versions)
I assume that these are the basic testing functionalities provided by devtools and testthat. If you expect to develop multiple packages and want to stay independent from non-base-R, I'd recommed to automate the above processes with custom scripts/packages.
I'd recomment looking into http://r-pkgs.had.co.nz/tests.html.
I'm new to R and I'm having some trouble to get the testthat unit test package work via devtools::test().
I've setup a package and created a test case under the .\tests\testthat folder. My R source code files are located at .\R.
When I run:
testthat::test_dir("./tests/testthat/")
The test ran successfully.
However when I tried to run the test via
devtools::test()
Instead of running the test cases, it tried to run my source code files located under .\R.
How can I get devtools::test() to just run my test cases?
Thank you for your help.
BTW, there is little documentation about how to setup and use testthat which is very frustrating as a new R user.
test() (re)loads your package before running your tests. That's why you see your package source code being executed.
I want to use testthat for integration tests in an R package being installed into a Jupyter notebook environment. Practically, that means I don't want the tests run when the package is installed, but rather manually later, when the system is running.
I think that means I shouldn't put tests into a tests/testthat directory, but rather the R directory.
So, I still want all the expect_ functions, but how do I run a test? I'm not sure if I can specify a file or directory, because it will be after the package is installed, and I don't know what the current working directory or install directory will be.
From the testthat package . . .
If you're using testthat in a package, you should put your tests in
tests/testthat. Each test file should start with test and end in .R or
.r. To ensure R CMD check runs your tests, place the following code in
tests/testthat.R:
library(testthat) ;
library(yourpackage)
test_check("yourpackage")
It sounds like you should follow all the setup but leave the testthat.R file out of the directory. You can run the test_check("yourpackage") from the console or a script.
To speed some functions in R package, I have re-coded them in cpp functions using Rcpp and successfully embedded those cpp functions into this package. The next step is to test if the cpp functions can output the same results as the original functions in R. So writing tests is necessary.
However, I was stuck on this step. I have read some links
Testing, R package by Hadley Wickham
and CRAN:testthat, page 11.
What I have done is that I run devtools::use_testthat()to create a tests/testthat directory. Then, run use_catch(dir = getwd())to add a test file tests/testthat/test-cpp.R. At this point, I think expect_cpp_tests_pass() might work but was just stuck on it. If I have the original function called add_inflow and add_inflow_Cpp. How can I test if these two functions are equal?
The documentation for ?use_catch attempts to describe exactly how the testing infrastructure for testthat works here, so I'll just copy that as an answer:
Calling use_catch() will:
Create a file src/test-runner.cpp, which ensures that the testthat
package will understand how to run your package's unit tests,
Create an example test file src/test-example.cpp, which showcases how
you might use Catch to write a unit test, and
Add a test file tests/testthat/test-cpp.R, which ensures that testthat
will run your compiled tests during invocations of devtools::test() or
R CMD check.
C++ unit tests can be added to C++ source files within the src/
directory of your package, with a format similar to R code tested with
testthat.
When your package is compiled, unit tests alongside a harness for running these tests will be
compiled into your R package, with the C entry point
run_testthat_tests(). testthat will use that entry point to run your
unit tests when detected.
In short, if you want to write your own C++ unit tests using Catch, you can follow the example of the auto-generated test-example.cpp file. testthat will automatically run your tests, and report failures during the regular devtools::test() process.
Note that the use of Catch is specifically for writing unit tests at the C++ level. If you want to write R test code, then Catch won't be relevant for your use case.
One package you might look at as motivation is the icd package -- see this file for one example of how you might write Catch unit tests with the testthat wrappers.
A "simple" question: how can I automatically run PHPunit tests with Arcanist?
According to the documentation I should first load a custom library. As stated here I should create a .arcconfig file and load the appropriate library.
So: I've create a dir "arc_libs" in my project and in the dir "src" I used arc liberate to generate the needed files. My config is now:
{
"project.name" : "arc_libs",
"phabricator.uri" : "https://phabricator.xxx.xxx.net/",
"unit.engine" : "PhpunitTestEngine",
"load" : ["arc_libs/src"]
}
The libary DOES get loaded because I can run arc unit
[matthijs#xx xxx]$ arc unit
No tests to run.
But as you can see there are no tests to run. We keep our tests in "project_root/tests" and as far as I understand the documentation I should create a __tests__ dir in "the module" (probably my arc_libs dir ?)
However I want to run my existing PHPunit test files, not new tests I need to create. I tried using a symlink etc but I cannot get it to work. Arcanist doesn't detect my tests.
So my question: How can I automatically run my EXISTING PHPunit tests with arcanist?
(note we use arc diff that should run arc unit automatically)
The documentation you linked won't be very useful - it's aimed at Phabricator developers who want to test their libraries. There is some user-facing documentation for customising unit test tasks, but it's not great. Fortunately, it's quite easy to get Arcanist to run your project's unit tests using the included PhpunitTestEngine:
Prepare a phpunit.xml file in your project root. This should be a standard PHPUnit configuration file. You can test this by running phpunit -c phpunit.xml.
Add a phpunit_config option to your .arcconfig:
{
"phabricator.uri": "https://phabricator.xxx.xxx.net/",
"unit.engine": "PhpunitTestEngine",
"phpunit_config": "phpunit.xml"
}
Run arc unit to test it out.
Although user documentation is thin on the ground, the source code for PhpunitTestEngine has some comments and is fairly concise. If you run into problems, reading through the test engine code can help you track it down.
$ arc unit --help
unit [options] [paths]
unit [options] --rev [rev]
Supports: git, svn, hg
Run unit tests that cover specified paths. If no paths are specified,
unit tests covering all modified files will be run.
By default, arc lint and arc unit are meant to be used as part of a process of making changes, so by default it only acts on changed files. Odds are, you don't have any changed files. You probably want to specify some paths, or run arc unit --everything to run all tests.