I'm writing unit tests for a package and there are some tests where I don't want the tests to throw errors if they fail but to instead give warnings.
This isn't my real code, but let's say I want to test something like:
add_x_y <- function(x, y) x + y
expect_equal( add_x_y(2, 2), 3 )
The output is an error:
Error: add_x_y(2, 2) not equal to 3.
1/1 mismatches
[1] 4 - 3 == 1
Is there a variant or alternative function that would throw a warning rather than an error for this check?
In the absence of an approach specific to testthat you could use general error handling to output a warning in place of an error.
expect_equal_or_warn <- function(...) tryCatch(expect_equal(...),
error = function(e) warning(e))
expect_equal_or_warn(add_x_y(2,2), 3)
Warning message:
add_x_y(2, 2) not equal to 3.
1/1 mismatches
[1] 4 - 3 == 1
Related
I have 3 rds files I use in my testthat tests. the files are in /tests/testthat/testdata/. When I run the tests with devtools::test(), the files can be found. When I use devtools::check(), the files won't be found.
I have seen this issue discussed here and here and many other places but all the suggested solutions won't work for me.
I tried adding the Sys.setenv(R_TESTS="") line at the top of my "testthat.R" file; I tried using here::here() from the package "here" instead of the , system.file() function, etc.
What can I do?
Here is my test:
test_that("the test_arguments() returns errors as it should", {
x <- readRDS(system.file("tests", "testthat", "testdata", "objectA.rds", package = "package"))
y <- readRDS(system.file("tests", "testthat", "testdata", "objectB.rds", package = "package"))
z <- readRDS(here::here("tests", "testthat", "testdata", "objectC.rds", package = "package"))
expect_error(test_arguments(z, y), "X has to be an object of class SpatialPolygonsDataFrame")
expect_error(test_arguments(x, z), "y has to be an object of class RasterLayer")
})
And here is the result of the devtools::check():
-- Error (test_user_input.R:3:3): the arguments are of the right class ---------------------
Error: cannot open the connection
Backtrace:
x
1. \-base::readRDS(here::here("tests", "testthat", "testdata", "objectA.rds")) test_user_input.R:3:2
2. \-base::gzfile(file, "rb")
[ FAIL 1 | WARN 1 | SKIP 0 | PASS 0 ]
Error: Test failures
Execution halted
1 error x | 1 warning x | 1 note x
I want to cat() some information to the console in the case a test fails (I'm getting confident this won't happen but I can't prove it wont) so I can investigate the issue.
Now I have code that is approximately like this:
testthat::test_that('Maybe fails', {
seed <- as.integer(Sys.time())
set.seed(seed)
testthat::expect_true(maybe_fails(runif(100L)))
testthat::expect_equal(long_vector(runif(100L)), target, tol = 1e-8)
if (failed()) {
cat('seed: ', seed, '\n')
}
})
Unfortunately, failed() doesn't exist.
Return values of expect_*() don't seem useful, they just return the actual argument.
I'm considering to just check again using all.equal() but that is a pretty ugly duplication.
Instead of using cat, you could use the info argument managed by testthat and its reporters for all expect functions (argument kept for compatibility reasons):
library(testthat)
testthat::test_that("Some tests",{
testthat::expect_equal(1,2,info=paste('Test 1 failed at',Sys.time()))
testthat::expect_equal(1,1,info=paste('Test 2 failed at',sys.time()))
})
#> -- Failure (<text>:5:3): Some tests --------------------------------------------
#> 1 not equal to 2.
#> 1/1 mismatches
#> [1] 1 - 2 == -1
#> Test 1 failed at 2021-03-03 17:25:37
Lately when I run my code that uses coxph in the survival package
coxph(frml,data = data), I am now getting warning messages of the following type
1: In model.matrix.default(Terms, mf, contrasts = contrast.arg) :
partial argument match of 'contrasts' to 'contrasts.arg'
2: In seq.default(along = temp) :
partial argument match of 'along' to 'along.with'"
I'm not exactly sure why all of a sudden these partial argument match warnings started popping up, but I don't think they effect me.
However, when I get the following warning message, I want coxph(frml,data = data) = NA
3: In fitter(X, Y, strats, offset, init, control, weights = weights, :
Loglik converged before variable 2 ; beta may be infinite.
6: In coxph(frml, data = data) :
X matrix deemed to be singular; variable 1 3 4
I used tryCatch when I wasn't getting the partial argument match warning using this code where if the nested tryCatch got either a warning or error message it would return NA
coxphfit = tryCatch(tryCatch(coxph(frml,data = data), error=function(w) return(NA)), warning=function(w) return(NA))
However, now that I am getting the partial argument match warnings, I need to only return an NA if there is an error or if I get the above warning messages 3 and 4 . Any idea about how to capture these particular warning messages and return an NA in those instances?
It's actually interesting question, if you are looking for quick and dirty way of capturing warnings you could simply do:
withCallingHandlers({
warning("hello")
1 + 2
}, warning = function(w) {
w ->> w
}) -> res
In this example the object w created in parent environment would be:
>> w
<simpleWarning in withCallingHandlers({ warning("hello") 1 + 2}, warning = function(w) { w <<- w}): hello>
You could then interrogate it:
grepl(x = w$message, pattern = "hello")
# [1] TRUE
as
>> w$message
# [1] "hello"
Object res would contain your desired results:
>> res
[1] 3
It's not the super tidy way but I reckon you could always reference object w and check if the warning message has the phrase you are interested in.
Trying to log all errors and warnings with futile.logger.
Somewhat satisfied with this for dealing with errors:
library(futile.logger)
options(error = function() { flog.error(geterrmessage()) ; traceback() ; stop() })
log("a")
# Error in log("a") : argument non numérique pour une fonction mathématique
# ERROR [2016-12-01 21:12:07] Error in log("a") : argument non numérique pour une fonction mathématique
#
# No traceback available
# Erreur pendant l'emballage (wrapup) :
There is redundancy, but I can easily separate between stderr, stdout and log file, so it's not a problem. It's certainly not pretty, there is an additional "wrapup" error message somehow caused by the final stop() that I don't understand, so I'm open to suggestions.
I cannot find a similar solution for warnings. What I tried:
options(warn = 1L)
options(warning.expression = expression(flog.warn(last.warning)))
log(- 1)
# [1] NaN
But to no avail.
Follow-up question: Are there best practices that I am unknowingly ignoring?
How about:
options(warning.expression =
quote({
if(exists("last.warning",baseenv()) && !is.null(last.warning)){
txt = paste0(names(last.warning),collapse=" ")
try(suppressWarnings(flog.warn(txt)))
cat("Warning message:\n",txt,'\n',sep = "")
}
}))
In can contribute two options to log R conditions like warnings with futile.logger and catch all warnings no matter how deep the function call stack is:
Wrap your code with withCallingHandlers for a basic solution
Use my package tryCatchLog for an advanced solution (for compliance reason: I am the author)
To explain the solutions I have created a simple R script that produces warnings and errors:
# Store this using the file name "your_code_file.R"
# This could be your code...
f1 <- function(value) {
print("f1() called")
f2(value) # call another function to show what happens
print("f1() returns")
}
f2 <- function(value) {
print("f2() called")
a <- log(-1) # This throws a warning: "NaNs produced"
print(paste("log(-1) =", a))
b <- log(value) # This throws an error if you pass a string as value
print("f2() returns")
}
f1(1) # produces a warning
f1("not a number") # produces a warning and an error
Executed "as is" (without logging) this code produces this output:
[1] "f1() called"
[1] "f2() called"
[1] "log(-1) = NaN"
[1] "f2() returns"
[1] "f1() returns"
[1] "f1() called"
[1] "f2() called"
[1] "log(-1) = NaN"
Error in log(value) : non-numeric argument to mathematical function
Calls: source -> withVisible -> eval -> eval -> f1 -> f2
In addition: Warning messages:
1: In log(-1) : NaNs produced
2: In log(-1) : NaNs produced
Solution 1 (withCallingHandlers)
Create a new R file that is called by R and sources your unchanged (!) original R script:
# Store this using the file name "logging_injector_withCallingHandlers.R"
# Main function to inject logging of warnings without changing your original source code
library(futile.logger)
flog.threshold(INFO)
# Injecting the logging of errors and warnings:
tryCatch(withCallingHandlers({
source("your_code_file.R") # call your unchanged code by sourcing it!
}, error = function(e) {
call.stack <- sys.calls() # "sys.calls" within "withCallingHandlers" is like a traceback!
log.message <- e$message
flog.error(log.message) # let's ignore the call.stack for now since it blows-up the output
}, warning = function(w) {
call.stack <- sys.calls() # "sys.calls" within "withCallingHandlers" is like a traceback!
log.message <- w$message
flog.warn(log.message) # let's ignore the call.stack for now since it blows-up the output
invokeRestart("muffleWarning") # avoid warnings to "bubble up" to being printed at the end by the R runtime
})
, error = function(e) {
flog.info("Logging injector: The called user code had errors...")
})
If you execute this wrapper code the R output is:
$ Rscript logging_injector_withCallingHandlers.R
NULL
[1] "f1() called"
[1] "f2() called"
WARN [2017-06-08 22:35:53] NaNs produced
[1] "log(-1) = NaN"
[1] "f2() returns"
[1] "f1() returns"
[1] "f1() called"
[1] "f2() called"
WARN [2017-06-08 22:35:53] NaNs produced
[1] "log(-1) = NaN"
ERROR [2017-06-08 22:35:53] non-numeric argument to mathematical function
INFO [2017-06-08 22:35:53] Logging injector: The called user code had errors...
As you can see
warnings are logged now
the call stack could be output too (I have disabled this to avoid flooding this answer)
References: https://stackoverflow.com/a/19446931/4468078
Solution 2 - package tryCatchLog (I am the author)
Solution 1 has some drawbacks, mainly:
The stack trace ("traceback") does not contain file names and line numbers
The stack trace is flooded with internal function calls you don't want to see (believe me or try it with your non-trival R scripts ;-)
Instead of copying&pasting the above code snippet again and again I have developed a package that encapsulates the above withCallingHandlers logic in a function and adds additional features like
logging of errors, warnings and messages
identifying the origin of errors and warnings by logging a stack trace with a reference to the source file name and line number
support post-mortem analysis after errors by creating a dump file with all variables of the global environment (workspace) and each function called (via dump.frames) - very helpful for batch jobs that you cannot debug on the server directly to reproduce the error!
To wrap the above R script file using tryCatchLog create a wrapper file
# Store this using the file name "logging_injector_tryCatchLog.R"
# Main function to inject logging of warnings without changing your original source code
# install.packages("devtools")
# library(devtools)
# install_github("aryoda/tryCatchLog")
library(tryCatchLog)
library(futile.logger)
flog.threshold(INFO)
tryCatchLog({
source("your_code_file.R") # call your unchanged code by sourcing it!
#, dump.errors.to.file = TRUE # Saves a dump of the workspace and the call stack named dump_<YYYYMMDD_HHMMSS>.rda
})
and execute it via Rscript to get this (shortened!) result:
# $ Rscript -e "options(keep.source = TRUE); source('logging_injector_tryCatchLog.R')" > log.txt
[1] "f1() called"
[1] "f2() called"
WARN [2017-06-08 23:13:31] NaNs produced
Compact call stack:
1 source("logging_injector_tryCatchLog.R")
2 logging_injector_tryCatchLog.R#12: tryCatchLog({
3 logging_injector_tryCatchLog.R#13: source("your_code_file.R")
4 your_code_file.R#18: f1(1)
5 your_code_file.R#6: f2(value)
6 your_code_file.R#12: .signalSimpleWarning("NaNs produced", quote(log(-1)))
Full call stack:
1 source("logging_injector_tryCatchLog.R")
2 withVisible(eval(ei, envir))
...
<a lot of logging output omitted here...>
As you can see clearly at the call stack level 6 the source code file name and line number (#12) is logged as the source of the warning together with the source code snippet throwing the warning:
6 your_code_file.R#12 .signalSimpleWarning("NaNs produced", quote(log(-1)))
The way you should use futile.logger is shown in its documentation. Here us a quick example, of how it typically is used, and how to set the threshold.
# set log file to write to
flog.appender (appender.file ("mylog.log"))
# set log threshold, this ensures only logs at or above this threshold are written.
flog.threshold(WARN)
flog.info("DOES NOT LOG")
flog.warn ("Logged!")
I am implementing a zero-inflated negative binomial in R. The code is here:
> ICHP<-read.table("ichip_data_recodeA.raw",header=TRUE)
ICHPdt<-data.table(ICHP)
covfile<-read.table("sorted.covfile.to.glm.out",header=TRUE)
covfiledt<-data.table(covfile)
library(pscl)
fhandle<-file("ichip_zi_nb_model_scoretest.csv","a")
for (i in seq(7, ncol(ICHPdt), 1)) {
notna<-which(!is.na(ICHPdt[[i]]))
string<-eval(parse(text = paste("ICHPdt$", colnames(ICHPdt)[i], sep="")))
nullglmmod<-zeroinfl(formula=OverllTot0[notna] ~ EurAdmix[notna] + Sex[notna] + DisDurMonths[notna] + BMI[notna] + Group[notna] + SmokingStatus[notna], data=covfiledt, dist="negbin")
nullsum<-coef(summary(nullglmmod))
glmmod<-zeroinfl(formula=OverllTot0[notna] ~ EurAdmix[notna] + Sex[notna] + DisDurMonths[notna] + BMI[notna] + Group[notna] + SmokingStatus[notna] + ICHPdt[[i]][notna], data=covfiledt, dist="negbin")
summ <- coef(summary(glmmod))
rownames(summ$zero)[8] <- paste0("ICHPdt$", colnames(ICHPdt)[i])
rownames(summ$count)[8] <- paste0("ICHPdt$", colnames(ICHPdt)[i])
writeLines("zero", con=fhandle)
writeLines(colnames(ICHPdt)[i], fhandle)
write.table(round(summ$zero, 4), file=fhandle)
writeLines("count", con=fhandle)
writeLines(colnames(ICHPdt)[i], fhandle)
write.table(round(summ$count, 4), file=fhandle)
}
The script errors when i=9246, and issues the following:
Error in solve.default(as.matrix(fit$hessian)) :
system is computationally singular: reciprocal condition number = 1.12288e-19
Overall, I need to go through ~100,000 markers, so I should expect ~11 such errors.
I would like to help implementing options, for instance with tryCatch() for catching such an error, skipping that marker, and moving on.
I recommend reading this page for a quick primer and this page for a more complete explanation of error handling, and you should eventually read ?conditions, but in short, there are two ways to handle errors. The first is with a try-catch, as in:
AS.NUMERIC <- function(x){
# for use in the warning handler
expectedWarning <- FALSE
result = tryCatch({
# a calculation that might raise an error or warning
as.numeric(x)
}, warning = function(w) {
# the typical way to identify the type of
# warning is via it's message attribure
if(grepl('^NAs introduced by coercion',w$message)){
cat('an expected warning was issued\n')
# assign the expected value using the scoping assignment
expectedWarning <<- TRUE
}else
cat('an unexpected warning was issued\n')
# reissue the warning
warning(w)
}, error = function(e) {
cat('an error occured\n')
# similar things go here but for handling errors
}, finally = {
# stuff goes here that should happen no matter what,
# such as closing connections or resetting global
# options such as par(ask), etc.
})
# you can handle errors similarly
if(expectedWarning)
result <- 5
return(result)
}
AS.NUMERIC('5')
#> [1] 5
AS.NUMERIC('five') # raises a warning
#> an expected warning was issued
#> [1] 5
#> Warning message:
#> In doTryCatch(return(expr), name, parentenv, handler) :
#> NAs introduced by coercion
The second way is to use try(), which is less nuanced:
x = try(stop('arbitrary error'),# raise an error
silent=TRUE)
# if there is an error, x will be an object with class 'try-error'
if(inherits(x,'try-error'))
# set the default value for x here
x = 5