tl;dr I want to run devtools::test() on a package and have it skip tests etc. as though it were running on CRAN, but I can't figure out how.
As I understand it, testthat::skip_on_cran() checks for the environment variable NOT_CRAN, which should be set to a value of "true" if the tests are not being run on CRAN (to back this up, the underlying test function testthat:::on_cran() is equal to
!identical(Sys.getenv("NOT_CRAN"), "true")
I am trying to use skip_on_cran() to skip some tests. I want to confirm that these tests will actually be skipped on CRAN. I have a line
cat("ON CRAN:", testthat:::on_cran(), "\n")
in my test file so that I can see what R/testthat thinks is going on.
The environment variables get set the way I want (i.e., the output includes ON CRAN: FALSE)/the tests get skipped properly) if I use
source([testfile], echo = TRUE)
(i.e., without doing anything special to set or unset the NOT_CRAN environment variable beforehand) or
withr::with_envvar(c(NOT_CRAN = "false"),
devtools::test_active_file("tests/testthat/test-bootMer.R"))
(if I run test_active_file() without wrapping it, I get ON CRAN: FALSE).
However, I don't see a way to run all the tests (via devtools::test()) in a similar way. In other words, I can't figure out how to run devtools::test() in "ON CRAN" mode. test() doesn't have an explicit argument for this (it has ... which is "additional arguments passed to wrapped functions", but I can't see anything relevant digging downward), and using withr::with_envvar() doesn't seem to help. devtools::check() does have an explicit env_vars argument, but I would like to be able to run the tests without going through the whole package check procedure ...
I'm sorry this isn't fully reproducible; if requested I can try to build a minimal package that will display the behaviour ...
Not sure if you still have this question but I put this line at the top of my first test file and it works in R 4.2.1.
Sys.setenv(NOT_CRAN='skip')
Related
R's testthat package has a number of functions for running tests: https://testthat.r-lib.org/reference/index.html#run-tests. However, the most coarse level you can filter tests seems to be at a file level, since there is a test_file() function that doesn't have any filtering arguments, and test_dir() has a filter argument but it is only used to filter by filename.
However I frequently want to only run a single test, because it's new, or because I know it's relevant to a change I just made.
Is there a way, in the R console or in RStudio to run a single testthat test? If not, is there some other recommended solution to this problem such as putting each test in it's own file (this seems pretty painful though)?
Let's say we have a symbol array of packages packages::Vector{Symbol} = [...] and we want to create a sys image using PackageCompiler.jl. We could simply use
using PackageCompiler
create_sysimage(packages; incremental = false, sysimage_path = "custom_sys.dll"
but without a precompile_execution_file, this isn't going to be worth it.
Note: sysimage_path = "custom_sys.so" on Linux and "custom_sys.dylib" on macOS...
For the precompile_execution_file, I thought running the test for each package might do it so I did something like this:
precompilation.jl
packages = [...]
#assert typeof(packages) == Vector{Symbol}
import Pkg
m = Module()
try Pkg.test.(Base.require.(m, packages)) catch ; end
The try catch is for when some tests give an error and we don't want it to fail.
Then, executing the following in a shell,
using PackageCompiler
packages = [...]
Pkg.add.(String.(packages))
Pkg.update()
Pkg.build.(String.(packages))
create_sysimage(packages; incremental = false,
sysimage_path = "custom_sys.dll",
precompile_execution_file = "precompilation.jl")
produced a sys image dynamic library which loaded without a problem. When I did using Makie, there was no delay so that part's fine but when I did some plotting with Makie, there still was the first time plot delay so I am guessing the precompilation script didn't do what I thought it would do.
Also when using tab to get suggestions in the repl, it would freeze the first time but I am guessing this is an expected side effect.
There are a few problems with your precompilation.jl script, that make tests throw errors which you don't see because of the try...catch.
But, although running the tests for each package might be a good idea to exercise precompilation, there are deeper reasons why I don't think it can work so simply:
Pkg.test spawns a new process in which tests actually run. I don't think that PackageCompiler can see what happens in this separate process.
To circumvent that, you might want to simply include() every package's test/runtests.jl file. But this is likely to fail too, because of missing test-specific dependencies.
So I would say that, for this to work reliably and systematically for all packages, you'd have to re-implement (or re-use, if you can) some of the internal logic of Pkg.test in order to add all test-specific dependencies to the current environment.
That being said, some packages have ready-to-use precompilation scripts helping to do just this. This is the case for Makie, which suggests in its documentation to use the following file to build system images:
joinpath(pkgdir(Makie), "test", "test_for_precompile.jl")
How can I run testthat in 'auto' mode such that when I'm editing files in my R folder, only specific tests are re-run?
I have a lot of tests and some are slower than others. I need to be able to run specific tests or else I'll be waiting for up to 15 minutes for my test suite to complete. (I'd like to make the test suite faster, but that's not a realistic option right now.)
Ideally, I'd like to specify a grep expression to select the tests I want. In the JavaScript world, MochaJs and Jest both support grepping to select tests by name or by file.
Alternatively, I'd be OK with being able to specify a file directly - as long as I can do it with "auto test" support.
Here's what I've found so far with testthat:
testthat::auto_test_package runs everything at first, but only re-runs a specific test file if you edit that test file. However, if you edit any code in the R folder, it re-runs all tests.
testthat::auto_test accepts a path to a directory of test-files to test. However, testthat doesn't seem to support putting tests into different subdirectories if you want to use devtools::test or testthat::auto_test_package. Am I missing something?
testthat::test_file can run the tests from one file, but it doesn't support "auto" re-running the tests with changes.
testthat::test_dir has a filter argument, but it only filters files, not tests; it also doesn't support "auto" re-running tests
Versions:
R: 3.6.2 (2019-12-12)
testthat: 2.3.1
Addendum
I created a simple repo to demo the problem: https://github.com/generalui/select_testthat_tests
If you open that, run:
renv::restore()
testthat::auto_test_package()
It takes forever because one of the tests is slow. If I'm working on other tests, I want to skip the slow tests and only run the tests I've selected. Grepping for tests is a standard feature of test tools, so I'm sure R must have a way. testthat::test_dir has a filter option to filter files, but how do you filter on test-names, and how do you filter with auto_test_package? I just can't find it.
How do you do something like this in R:
testthat::auto_test_package(filter = 'double_it')
And have it run:
"double_it(2) == 4"
"double_it(3) == 6"
BUT NOT
"work_hard returns 'wow'"
Thanks!
It there an easy way to skip the execution of some tests in a package if the package is tested by CRAN? The background is, I like to have a lot of tests and in sum they are time consuming (not good for CRAN).
I know there is testthat::skip_on_cran() but I do not want to use package testthat to avoid another dependency. I am looking for an el-cheapo way to mimic testthat::skip_on_cran.
Ideally, I would like to have a testfile in directory pkg/tests that calls the tests (testfiles) and distuingishes if we are on cran or not:
if (!on_cran) {
## these tests are run only locally/when not on CRAN
# run test 1 (e.g. source a file with a test)
# run test 2
}
# run test 3 - always
Yes! You can handle this programmatically and automatically. Let me detail two ways I've set up:
Implicitly via version numbers: This is the approach taken by Rcpp for many years now, and it is entirely generic and not dependent on any other package. Our tests start from a file in tests/ and then hand over to RUnit, but that last part is an implementation detail.
In the main file tests/doRUnit.R we do this:
## force tests to be executed if in dev release which we define as
## having a sub-release, eg 0.9.15.5 is one whereas 0.9.16 is not
if (length(strsplit(packageDescription("Rcpp")$Version, "\\.")[[1]]) > 3) {
Sys.setenv("RunAllRcppTests"="yes")
}
In essence, we test if the version is of the form a.b.c.d -- and if so conclude that it is a development version. This implies "run all tests". Whereas a release version of the form a.b.c would go to CRAN, and not run these tests as they would exceed their time limit.
In each of the actual unit test files, we can then decide if we want to honour the variable and skip the test if set, or execute anyway:
.runThisTest <- Sys.getenv("RunAllRcppTests") == "yes"
if (.runThisTest) {
## code here that contains the tests
}
This mechanism is fully automatic, and does not depend on the user. (In the actual package version there is another if () test wrapped in there which allows us to suppress the tests, but that is a detail we do not need here).
I still like this approach a lot.
Explicitly via resource files Another package a few of us work on (a lot lately) requires a particular backend to be available. So in the Rblpapi package we tests for presence of a file which my coauthors and I each have below our $HOME directory in order to set up credentials and connection details. If the file is missing --- as e.g. on Travis CI, or CRAN, or for other users, the tests are skipped.
We chose to use the resource file as R file; it sources it if found and thereby sets values for options(). That way we can control directly whether to launch tests or not.
## We need to source an extra parameter file to support a Bloomberg connection
## For live sessions, we use ~/.Rprofile but that file is not read by R CMD check
## The file basically just calls options() and sets options as needed for blpHost,
## blpPort, blpAutoConnect (to ensure blpConnect() is called on package load) and,
## as tested for below, blpUnitTests.
connectionParameterFile <- "~/.R/rblpapiOptions.R"
if (file.exists(connectionParameterFile)) source(connectionParameterFile)
## if an option is set, we run tests. otherwise we don't.
## recall that we DO need a working Bloomberg connection...
if (getOption("blpUnitTests", FALSE)) {
## ... more stuff here which sets things up
}
Similarly to the first use case we can now set more variables which are later tested.
Explicitly via Travis CI Another option we use in rfoaas is to set the environment variable governing this in the Travis CI file:
env:
global:
- RunFOAASTests=yes
which the tests script then picks up:
## Use the Travis / GitHub integrations as we set this
## environment variable to "yes" in .travis.yml
##
## Set this variable manually if you want to run the tests
##
if (Sys.getenv("RunFOAASTests=yes") == "yes") runTests <- TRUE
In that case I also set the toggle based on my userid as I am pretty much the sole contributor to that project:
## Also run the tests when building on Dirk's box, even whem
## the environment variable is not set
if (isTRUE(unname(Sys.info()["user"])=="edd")) runTests <- TRUE
Explicitly via another variable You can of course also rely on another variable you use across all your packages. I find that to be a bad idea. If you set this in your shell, work on package A and set it to suppress tests but then switch to package B --- you will likely forget to unset the variable and then fail to test. I like this approach the least and do not use it.
Use an environment variable, like testthat does:
skip_on_cran <- function() {
if (identical(Sys.getenv("NOT_CRAN"), "true")) {
return(invisible(TRUE))
}
skip("On CRAN")
}
I am creating an R package, and found it useful to break parts of the logic in one file into internal helper functions, which I define in the same file. I have sort of a special case where my function decides which helper function to use via match.fun(). Since they won't be useful to other functions or people, I don't want to put these in separate files, and I don't want to export them.
All my testthat cases pass using test_dir(). When I don't export these functions, my testthat cases fail during R CMD check.
"object 'helperfunction1' of mode 'function' was not found", quote(get(as.character(FUN),
mode = "function", envir = envir)))
After looking at this post, I am able to get things to work if I explicitly export or add export entries to NAMESPACE, but again I don't want to export these.
Is there a better way to do this and doesn't require me to export? (I'll admit that the source of the issue may be match.fun() and am open to other ways of calling functions at runtime.)
From memory it wasn't clear in the documentation last time I read it (it may have changed), but it will work correctly (without having to export) so long as everything is in the right directories:
You should have a file:
tests/run-all.R
That looks like:
library(testthat)
library(myPackage)
test_package("myPackage")
Then your individual test files should be in the directory inst/tests
These will be run when you do R CMD check, otherwise you can call test_package("myPackage") in R manually.