Access R code block in multiple instances - r

I have a code block, to perform 3 times retry of the code execution in case of a specific error. In below example if HTTP 503 error occurred during the data download from ADLS container, I want the same operation to be executed maximum of 3 times retry.
require(AzureStor)
require(stringr)
recheck <- 0
while (recheck < 3){
recheck <- recheck + 1
tryCatch({
storage_download(container, file, filename, overwrite=TRUE)
recheck <- 4
}, error = function(e){
if ( sum(str_detect(e, '503')*1) > 0 ){
print(e)
print(paste0('An infra-level failure occured. Retry sequence number is : ', recheck))
} else{
recheck <<- 4
print(e)
}
}
)
}
This code works fine for me, but similar to storage_download in the above example, I have other ADLS operations like delete_blob, upload_blob, storage_upload, list_storage_files at multiple instances in the code, I have to write above mentioned code for each of these functions. I want to make the above code as a function which can be called during each of these ADLS operations. Any thoughts or suggestions would help me greatly.

The following should do the trick:
with_retries_on_failure = function (expr, retries = 3L) {
expr = substitute(expr)
for (try in seq_len(retries)) {
tryCatch(
return(eval.parent(expr)),
error = \(e) {
if (str_detect(conditionMessage(e), '503')) stop(e)
message('An infra-level failure occurred. Retry sequence number is: ', try)
}
)
}
}
Used as follows:
with_retries_on_failure(storage_download(container, file, filename, overwrite=TRUE))
Note the return() call, which immediately returns from the surrounding function without the need to update the loop variable. Likewise, in the case of a failure we also don’t have to update the loop variable since we are using a for loop, and we use stop() to break out of the loop for any error that is not a 503 HTTP response.

Related

Terminate the exectuion with STOP() but not throwing an "Error" message in R [duplicate]

This question already has answers here:
Stop an R program without error
(12 answers)
Closed 4 years ago.
I want a function that would terminate the program when there is no data are returned and print out a message to users about there is no new data are found.
However, I don't want my function throws users an "error" message. For example,
my_update_function <- function () {
df <- data # get data
if (nrow(df) == 0) {
stop('\r no new data found')
}
else {
# do some updates
}
}
current output:
Error in my_update_function() :
no new data found
my expected output: no new data found
How can I get rid of the "Error" in my output message but the program is terminated when the condition is true.
This would hide the word "error" on the console but the program still sees it as an error. Referencing on Stop an R program without error
if (nrow(new_data) == 0) {
message(log_date, " - ","no new_data is found ")
opt <- options(show.error.messages=FALSE)
on.exit(options(opt))
stop()
}
else { # DO SOMETHING ELSE
}

Error in open.connection(con, "rb") : Timeout was reached: Resolving timed out after 10000 milliseconds

So I've got a list of "Player" objects, each with an ID, called players and I'm trying to reach a web JSON with information related to the relevant ID, using JSONlite.
The HTML stem is: 'https://fantasy.premierleague.com/drf/element-summary/'
I need to access every players respective page.
I'm trying to do so as follows:
playerDataURLStem = 'https://fantasy.premierleague.com/drf/element-summary/'
for (player in players) {
player_data_url <- paste(playerDataURLStem,player#id,sep = "")
player_data <- fromJSON(player_data_url)
# DO SOME STUFF #
}
When I run it, I'm getting the error Error in open.connection(con, "rb") : Timeout was reached: Resolving timed out after 10000 milliseconds. This error is produced at a different position in my list of players each time I run the code and when I check the webpage that is causing the error, I can't see anything erroneous about it. This leads me to believe that sometimes the pages just take longer than 10000 milliseconds to reply, but using
options(timeout = x)
for some x, doesn't seem to make it wait longer for a response.
For a minimum working example, try:
playerDataURLStem = 'https://fantasy.premierleague.com/drf/element-summary/'
ids <- c(1:540)
for (id in ids) {
player_data_url <- paste(playerDataURLStem, id, sep = "")
player_data <- fromJSON(player_data_url)
print(player_data$history$id[1])
}
options(timeout= 4000000) is working for me .try increasing value of timeout to higher number

How to tell if any tests failed from result of testthat::test_dir

I have a small testing program that uses testthat
library(testthat)
source("src/MyFile.r")
results <- test_dir("tests", reporter="summary")
So, I am running this via Rscript. The problem is that the exit code is always 0 even when there were test failures. So, I want to call stop if there are any failures. But I can't seem to get the right bit of code to do that. Is there a method or field in results that I should be looking at to determine if there were any errors?
You could also tweak jamesatha's answer to retrieve the number of test failures.
failed.tests <- sapply(results, function(r) {
!is(r$result[[1]], "expectation_success")
})
You then allows you to fail, as before:
if (any(failed.tests)) {
stop("There were test failures")
}
or do something more bespoke like
if (any(failed.tests)) {
failed.test.count <- length(which(failed.tests))
stop(paste(failed.test.count,"failed tests is",failed.test.count,"too many!")
}
Currently, my solution is iterating through the results like this:
for (i in 1:length(results)) {
if (!is(results[[i]]$result[[1]], "expectation_success")) {
stop("There were test failures")
}
}
You can simply pass stop_on_failure=TRUE in as a parameter to test_dir. If you get any test failures it will raise an error and exit non zero.
e.g:
results <- test_dir("mypath", stop_on_failure=TRUE)
This is documented here

Issue while using Try Catch in R

I am new to R. Kindly excuse me if it is a basic query. I am facing a strange situation where I am using try catch to catch errors and warnings while sourcing another script. It is working fine also. However, in one particular case, while sourcing, there is an error message followed by a warning message, but try catch in my script is considering it only as a warning message.
Please find my code as follows
result1 <- tryCatch ({
source("Path with script.R")
inc_val = 1
}, error = function(er) {
inc_val = 2
} , warning = function(er) {
inc_val = 3
},finally = {
})
So in this particular case, result1 is 3 instead of 2, even though there is an error while executing the script.
Kindly let me know where I have done the mistake.
Thanks in advance.

getURL (from RCurl package) doesn't work in a loop

I have a list of URL named URLlist and I loop over it to get the source code for each of those URL :
for (k in 1:length(URLlist)){
temp = getURL(URLlist[k])
}
Problem is for some random URL, the code get stuck and I get the error message:
Error in function (type, msg, asError = TRUE) :
transfer closed with outstanding read data remaining
But when I try the getURL function, not in the loop, with the URL which had a problem, it perfectly works.
Any help please ? thank you very much
Hard to tell for sure without more information, but it could just be the requests getting sent too quickly, in which case just pausing between requests could help :
for (k in 1:length (URLlist)) {
temp = getURL (URLlist[k])
Sys.sleep (0.2)
}
I'm assuming that your actual code does something with 'temp' before writing over it in every iteration of the loop, and whatever it does is very fast.
You could also try building in some error handling so that one problem doesn't kill the whole thing. Here's a crude example that tries twice on each URL before giving up:
for (url in URLlist) {
temp = try (getURL (url))
if (class (temp) == "try-error") {
temp = try (getURL (url))
if (class (temp) == "try-error")
temp = paste ("error accessing", url)
}
Sys.sleep(0.2)
}

Resources