testthat fails within devtools::check but works in devtools::test - r

Question

Is there any way to reproduce the environment which is used by devtools::check?
I have the problem that my tests work with devtools::test() but fail within devtools::check(). My problem is now, how to find the problem. The report of check just prints the last few lines of the error log and I can't find the complete report for the testing.
checking tests ... ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
...
I know that check uses a different environment compared to test but I don't know how I should debug these problems since they are not reproducible at all. Specially these test where running a few month ago, so not sure where to look for the problem.
EDIT
actually I tried to locate my problem and I found a solution. But to post my solution to it, I have to add more details.
So my test always failed since I was testing a markdown script if it is running without errors and afterwards I was checking if some of the environmental variables are set correctly. These where results which I calculate with the script as well as standard settings which I set. So I wanted to get a warning if I forgot to change some of my settings after developing...
Anyway, since it is a markdown script, I had to extract the code and I was using comments from this post knitr: run all chunks in an Rmarkdown document using knitr::purl to get the code and sys.source to execute it.
runAllChunks <- function(rmd, envir=globalenv()){
# as found here https://stackoverflow.com/questions/24753969
tempR <- tempfile(tmpdir = '.', fileext = ".R")
on.exit(unlink(tempR))
knitr::purl(rmd, output=tempR, quiet=TRUE)
sys.source(tempR, envir=envir)
}
For some reason, this produces an error since maybe a few weeks (not sure which new packages I installed lately...). But since there is a new comment, that I can just use knitr::knit which also executes the code, this worked as expected and now my test no longer complains.
So in the end, I don't know where the problem exactly was, but this is now working.

Answer 1

I recently had a similar issue with my tests breaking (succeeding with devtools::test() but failing with devtools::check()). I don't know if this solution necessarily fixes the problem above, but it should help to track down similar problems.
In my case, the problem ultimately came down to using a function that needed a package listed in Suggests rather than in Imports/Depends. In particular, my function called httr::content(), which broke when I tried to pass it the as = "parsed" argument. It turns out that as = "parsed" uses a suggested package, readr to read a csv, and I needed to add it to my dependencies for devtools::check() to work.

Answer 2

This is a known issue with testthat. The workaround is to add the following as the 1st line in tests/testthat.R:
Sys.setenv(R_TESTS="")

Answer 3

In case it helps someone else, this is what worked for me
Re-install all relevant packages. E.g. install.packages("testthat", "dplyr", "lubridate", "stringr") (I included all packages my package uses)
Close RStudio and reopen
Then all tests passed

Answer 4

I spent much too long looking into this error, so hoping that I can help someone out in the future. I would like to add to this that I was getting this error while using ggplot2::autoplot() in my function and it required that I added #import ggfortify to the Roxygen skeleton part of my function.

Answer 5

I ran into the same issue with my tests failing under devtools::check() while not failing under testthat::test()
And none of the above applied to my problem, so i decided to post my issue plus solution here as well. But first some NOTEs from my experience:
devtools::check() does - so it seems - deeper error checking then your own written tests.
Now to my code-setup. I had a function that was build to retrieve values from two different files. Those files contained named profiles with a set of values per profile. But the profiles were named differently, depending on the files:
Example files:
Content of file_one:
[default]
value_A = "foo"
value_B = "bar"
value_C = "baz"
[peter]
value_A = "oof"
value_B = "rab"
value_C = "zab"
content of file_two:
[default]
value_X = "fuzzly"
value_Z = "puzzly"
[profile peter]
value_X = "fuzzly"
value_Z = "puzzly"
As you can see, does the naming in file two follow another naming convention, when it comes to the named profiles. The profiles are written in "[]" and the default-profile is always '[default]' in both files. But as soon as it comes to named profiles, its just '[name]' in one file and then '[profile name]' in the other one.
Now i've build the function like that (simplyfied):
get_value <- function(file_content, what, profile) {
file_content <- readr::read_lines(file)
all_profiles_at <- grep("\\[.*\\]", file_content)
profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
profile_at <- grep(profile_regex, file_content)
profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
profile_content <- file_content[profile_at:profile_ends_at]
whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}
With this code my tests ran smoothly and even check() found no issues.
While the whole code evolved i figured, that i should read the files content beforehand and give only the alread read_in content to the function to avoid duplication in my code. So i changed the function like so:
get_value <- function(file, what, profile) {
is_file_two <- is_file_two(file_content)
all_profiles_at <- grep("\\[.*\\]", file_content)
profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
profile_at <- grep(profile_regex, file_content)
profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
profile_content <- file_content[profile_at:profile_ends_at]
whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}
As you might notice i only changed the first line of the funciton body and left the if-condition unchanged - my mistake!
But my tests didn't throw an error, as the if-condition still worked. Even though the 'file_content == "file_two"' part now generated a logical vector and if() ... else ... normally throws a warning, when the logical has length > 1. The special construct with the && doesn't throw such an error as it returns a length(1) logical:
# with warning
if(c(FALSE, FALSE, FALSE)) "Done!" else "Not done!"
# no warning:
if(c(FALSE, FALSE, FALSE) && TRUE) "Done!" else "Not done!"
Thats why my tests with testthat::test() sill worked.
But devtools::check() saw this flaw in my code and the tests failed!
And that part of the FAILURE_REPORT showed me my errors:
[...]
where 41: test_check("my_package_name")
--- value of length: 18 type: logical ---
[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE
--- function from context ---
[...]
Conclusion:
testthat::test() is great! Is checks whether or not your code still runs. But devtools::check() goes far deeper - and when your tests pass with testthat::test() but fail with devtools::check() then you've probaly got some deeper bugs and flaws in your code you MUST attend to!

Answer 6

So as I shortly mentioned above, I changed some of my code to no longer user knitr::purl but using knitr::knit and this solved my problem.
expect_error(f <- runAllChunks('010_main_lfq_analysis.Rmd'), NA)
expect_error(f <- knitr::knit('010_main_lfq_analysis.Rmd', output='jnk.R', quiet=TRUE, envir=globalenv()), NA)

Answer 7

This could also happen in the following scenario: You have a library already loaded in R and you are referring to the function in that library without namespace binding. For example, suppose you use the nnzero() function from the Matrix in a test file and happen to also have had the Matrix package already loaded with library(Matrix). Then devtools::test() will pass but devtools::check() fails. Using Matrix::nnzero() should fix the problem.