I retrieve data from a server using {httr} in a for loop; sometimes requests time out, which causes an error. Since I want R to continue after errors, I wrapped this function call into try(). However, this makes it impossible for me to stop R manually using the stop button in Rstudio: {httr} throws an error and the for loop continues to the next iteration.
What can I do to ensure R stops when I hit 'stop' in Rstudio?
example code:
require(httr)
authorlist<-c("alice","bob","charlie","david","erin","frank")
datalist<-list()
for(author in authorlist){
result<-NULL
count<-0
while(is.null(result) & count<5){
try(
result<-GET(paste0("https://api.pushshift.io/reddit/search/comment?author=",author),timeout(10))
)
}
datalist[length(datalist)+1]<-result
}
I expect R to stop processing; but it merely skips to the next loop iteration (of either for-loop).
Related
I was learning about the package httr and webscraping based on an exercise from Dataquest and attempting to implement it in my own practice programs. My issue comes from trying to make a query within a function.
For example, the following code:
api_request <- function(base_url, loc){
url <- modify_url(paste(base_url),
path = paste(loc))
response <- GET(url)
return(response)
}
When I run the code, everything initially appears to run correctly. The status comes back with the code 200 and no errors or warnings show up. However, I cannot get the response to save to the global environment. I've tried this method as well as changing the return(response) to just response in the function as recommended by Dataquest, but it will not save to the global environment.
I can get this to work outside of a function, but I want to implement it inside of a function so that if any errors occur when making this query I can stop the function and not save a 404 link.
How can I get the query to return from the function so that I can reference it later on in the code?
I have a script that constantly performs a set of calculations in an endless loop. But often there are various errors that I cannot predict and the script stops working. I would like to automatically restart the script every time it stops working, and I don't care why the error occurred, I just want to restart the script. I will cite a deliberately erroneous code that I would like to repeat after an error has been issued.
while (TRUE) {
1+m
}
The try() and trycatch() funtions are designed to deal with code that might cause errors. In the case of your example code, changing it to:
while (TRUE) {
try(1+m)
}
will keep trying to do the line that produces an error. If your code inside the loop is multiple lines, you can make it into a block by wrapping it in braces, e.g. :
while (TRUE) {
try({
a <- 1+m
print(a)
})
}
I am running tests in R using the test_dir() function from the testthat package. In some of the test scripts there are functions that call readline(), which - in interactive mode - causes the testing to pause and wait for user input. The functions calling readline() are not my own and I don't have any influence on them. The user input is irrelevant for the output of those functions.
Is there a way to avoid these pauses during testing?
Approaches that come to mind, but I wouldn't know how to implement them:
disable interactive mode while R is running
use another function from the testthat package that runs scripts in non-interactive mode
somehow divert stdin to something else than the terminal(??)
wrap functions calling readline() in another script that is called in non-interactive mode from my testing script and makes the results available
Testing only from the command line using Rscript is an option, but I'd rather stay in the RStudio workflow.
======
Example Code
with_pause <- function () {
readline()
2
}
without_pause <- function () {
2
}
expect_equal(with_pause(), without_pause())
I have a similar problem. I solved it with a global option setting.
original_test_mode <- getOption('my_package.test_mode')
options('my_package.test_mode' = TRUE)
# ... some tests ...
options('my_package.test_mode' = original_test_mode)
In my scripts I have a if statement
if(getOption('my_package.test_mode', FALSE)) {
# This happens in test mode
my_value <- 5
} else {
# normal processing
my_value <- readline('please write value: ')
}
Also not the nicest way but it works for me.
Maybe one more hint. It happened to that my test script failed. The problem here is, that the global option stays TRUE and in the next round and also for executing the script in the same session, it will never prompt you to write a value. I guess I should put some stuff in a tryCatch function or so. But if you have this problem in mind, just "sometimes" options('my_package.test_mode', NULL) helps :-)
I have a script that uses the rNOMADS package to download forecast data. Currently, it uses a for loop to call the forecast download function for each three hour forecast interval in order. The issue is the download function occasionally "freezes" at random which forces me to terminate R and start the process over. When it freezes the code hangs at the download function for minutes instead of the typical <1 sec it takes to execute, and then when I try to halt execution I get a message saying "R is not resrponding to your request to interrupt processing so to stop the current operation you may need to terminate R entirely."
Is there a way set a time limit for a specific block of code to execute in each for loop iteration, and then skip that block of code and throw an error if the time limit is reached? Something like tryCatch, that I could use to raise a flag to re-do that for loop iteration?
Something like:
for (i in 1:N) {
...
setTimeLimit(XXX seconds) {
downloadFunction()
} timeLimitReached {
doOverFlag <- 1
}
}
Thanks in advance!
The function evalWithTimeout of package R.utils does this.
evalWithTimeout(Sys.sleep(10), timeout = 1)
(times are in seconds).
Note: I have not used this function a lot, I liked your question so I did some googling around and found this.
I really like R.utils for some situations, but it clobbers the traceback for the internal error message if there was one (lets' say you're running in parallel and want to wrap it in a timeout)
R base has the functionality setTimeLimit that you can wrap using {} with your expression. It returns a simple error message so it's very useful and does not remove other error handling possibilities (like withCallingHandlers which is extremely useful for parsing/storing error messages and the call stack):
test_fun <- function() {
repeat {
runif(100)
}
}
res <- {
setTimeLimit(5)
test_fun()
}
This function works as follows now:
library(R.utils)
withTimeout(Sys.sleep(10), timeout = 1)#stop execution after one second
Good advice, following on from the answers above, and regarding the fact that the loop stops: To avoid breaking the entire loop, make sure you add if (all(class(variable_name)=="try-error")) next()
I am running an optimization program I wrote in a multi-language framework. Because I rely on different languages to accomplish the task, everything must be standalone so it can be launched through a batch file. Everything has been going fine for 2-3 months, but I finally ran out of luck when one of the crucial parts of this process, executed through a standalone R script, encountered something new and gave me an error message. This error message makes everything screech to a halt despite my best efforts:
selMEM<-forward.sel(muskfreq, musk.MEM, adjR2thresh=adjR2)
Procedure stopped (adjR2thresh criteria) adjR2cum = 0.000000 with 0 variables (superior to -0.005810)
Error in forward.sel(muskfreq, musk.MEM, adjR2thresh = adjR2) :
No variables selected. Please change your parameters.
I know why I am getting this message: it is warning me that no variables are above the threshold I have programmed to retain during a forward selection. Although this didn't happen in hundreds of runs, it's not that big a deal, I just need to tell R what to do next. This is where I am lost. After an exhaustive search through several posts (such as here), it seams that try() and tryCatch() are the way to go. So I have tried the following:
selMEM<-try(forward.sel(muskfreq, musk.MEM, adjR2thresh=adjR2))
if(inherits(selMEM, "try-error")) {
max<-0
cumR2<-0
adjR2<-0
pvalue<-NA
} else {
max<-dim(selMEM)[1]
cumR2<-selMEM$R2Cum[max]
adjR2<-selMEM$AdjR2Cum[max]
pvalue<-selMEM$pval[max]
}
The code after the problematic line works perfectly if I execute it line by line in R, but when I execute it as a standalone script from the command prompt, I still get the same error message and my whole process screeches to a halt before it executes what follows.
Any suggestions on how to make this work?
Note this in the try help:
try is implemented using tryCatch; for programming, instead of
try(expr, silent = TRUE), something like tryCatch(expr, error =
function(e) e) (or other simple error handler functions) may be more
efficient and flexible.
Look to tryCatch, possibly:
selMEM <- tryCatch({
forward.sel(muskfreq, musk.MEM, adjR2thresh=adjR2)
}, error = function(e) {
message(e)
return(NULL)
})
if(is.null(selMEM)) {
max<-0
cumR2<-0
adjR2<-0
pvalue<-NA
} else {
max<-dim(selMEM)[1]
cumR2<-selMEM$R2Cum[max]
adjR2<-selMEM$AdjR2Cum[max]
pvalue<-selMEM$pval[max]
}
Have you tried setting the silent parameter to true in the Try function?
max<-0
cumR2<-0
adjR2<-0
pvalue<-NA
try({
selMEM <- forward.sel(muskfreq, musk.MEM, adjR2thresh=adjR2)
max<-dim(selMEM)[1]
cumR2<-selMEM$R2Cum[max]
adjR2<-selMEM$AdjR2Cum[max]
pvalue<-selMEM$pval[max]
}, silent=T)