i'm trying to run a API request for a number of parameters with the lapply function in R.
However, when i run this function, i get the error " Error in file(con, "r") : cannot open the connection"
Google suggests using setInternet2(TRUE) to fix this issue, however, i get the error: Error: 'setInternet2' is defunct.
See help("Defunct"
localisedDestinationNameForGivenLang <- function (LocationId) {
gaiaURL <- paste0("https://URL/",LocationId, "?geoVersion=rwg&lcid=", "1036",
"&cid=geo&apk=explorer")
print(LocationId)
localisation <- fromJSON(gaiaURL)
}
lapply(uniqueLocationId, localisedDestinationNameForGivenLang)
Can someone suggest a fix please?
Here's a sample of how you could identify which sites are throwing errors while still getting response from the ones that don't:
urls = c("http://citibikenyc.com/stations/test", "http://citibikenyc.com/stations/json")
grab_data <- function(url) {
out <- tryCatch(
{fromJSON(url)},
error=function(x) {
message(paste(url, x))
error_msg = paste(url, "threw an error")
return(error_msg)
})
return(out)
}
result <- lapply(urls, grab_data)
result will be a list that contains API response for urls that work, and error msg for those that don't.
Related
In summary, I'm finding that when running httr::POST against a plumber api, within an R.utils::withTimeout, the post is throwing an error message which can't be suppressed. The error message is:
Error in .Call(R_curl_fetch_memory, enc2utf8(url), handle, nonblocking) : reached elapsed time limit
Things I've tried:
suppressmessages/warnings
tryCatch with no error response
Using sinks
I can't think of any other way of stopping this error. The code carries on running but it's resulting in users of the api getting spammed with error messages when checking for an available port to call. Any ideas welcome.
Reproducible example (note this uses rstudio jobs pkg and a separate plumber file, but i've tried without the jobs pkg and the issue persists so it's not connected to that.) :
plumber.R:
#* quick ping
#* #post /ping
function() {
list(msg = "you got here!")
}
#* slow 10s call
#* #post /slowfxn
function() {
Sys.sleep(10)
1
}
code which calls and produces errors:
#run the plumber file in a separate job:
r <- plumber::plumb("errortest/plumber.R")
job::job({r$run(swagger = F,port = 1111)})
#this function will throw the error if no response comes within 3s:
call_with_timeout = function() {
R.utils::withTimeout(
rawToChar(httr::POST("http://127.0.0.1:1111/echo")$content)
,timeout = 3,onTimeout = "silent")
}
#now call the fxn (again in a job) which takes 10 s to run:
job::job({httr::POST(url = "http://127.0.0.1:1111/slowfxn")})
#wait a second for that job to be initiated:
Sys.sleep(1)
#while that's running, try and call the quick fxn, with a 3 second timeout:
#try wrap it in a trycatch to suppress:
tryCatch( {
call_with_timeout()
},
error = function(x) {},
TimeoutException = function(x){},
warning = function(x) {}
)
suppressMessages(suppressWarnings(call_with_timeout()))
#not even a sink can stop it!
sink("delete.txt")
call_with_timeout()
sink(NULL)
I have been using readRDS(gzcon(url("my dropbox links")))for a long time to load my saved .rds field from Dropbox without any issues. But ever wonder why readRDS("my dropbox links") does not do the same thing. I got an error like
Error in gzfile(file, "rb") : cannot open the connection
It seems like a fair, simple question, but I couldn't figure it myself. Many thanks in advance~
another alternative would be to first download the file and then read the RDS:
download.file("https:www.somesite.com/somefile.rds",
"data.rds",
method = "curl")
The readRDS - function checks if the argument is a connection and but doesn't create URLS itself:
> readRDS
function (file, refhook = NULL)
{
if (is.character(file)) {
con <- gzfile(file, "rb")
on.exit(close(con))
}
else if (inherits(file, "connection"))
con <- if (inherits(file, "url"))
gzcon(file)
else file
else stop("bad 'file' argument")
.Internal(unserializeFromConn(con, refhook))
}
<bytecode: 0x5648012c7c50>
<environment: namespace:base>
therefore the url - function is needed.
> link<-"https://www.google.com"
> inherits(link,"connection")
[1] FALSE
> link2<-url("https://www.google.com")
> inherits(link2,"connection")
[1] TRUE
I am building a function to connect to a specific password-protected ODBC data source that will be used many members of a team - it may be used in multiple environments. In the event that the connection is rejected, I would like to display the warning messages but mask the password that's displayed. If I use suppressWarnings() nothing gets captured as far as I can tell, and if I don't, then the message is displayed in the standard output with the password. Here's the function so far:
connectToData <- function(uid, pswd, dsn='myDSN') {
# Function to connect to myDSN data
#
# Args:
# uid: The user's ID for connecting to the database
# pswd: The user's password for connecting to the database.
# dsn: The DSN for the (already existing) ODBC connection to the 5G
# data. It must be set up on an individual Windows user's machine,
# and they could use any name for it. The default is 'myDSN'
#
# Returns:
# The 'RODBC' class object returned by the RODBC:odbcConnect() function.
#
# TODO: 1) See if you can specify the connection using odbcDriverConnect()
# so as to not rely on user's ODBC connections
# 2) Capture warnings from odbcConnect() and print them while
# disguising password using gsub, as I've attempted to do below.
library('RODBC')
db.conn <- odbcConnect(dsn,
uid=uid,
pwd=pswd)
if(class(db.conn) != 'RODBC') { # Error handling for connections that don't make it
print(gsub(pswd,'******',warnings())) # This doesn't work like I want it to
stop("ODBC connection could not be opened. See warnings()")
} else {
return(db.conn)
}
}
When I run it with the right username/password, I get the right result but when I run it with a bad password, I get this:
> db.conn <- connectTo5G(uid='myID',pswd='badpassword', dsn='myDSN')
[1] "RODBC::odbcDriverConnect(\"DSN=myDSN;UID=myID;PWD=******\")"
[2] "RODBC::odbcDriverConnect(\"DSN=myDSN;UID=myID;PWD=******\")"
Error in connectTo5G(uid = "myID", pswd = "badpassword", dsn = "myDSN") :
ODBC connection could not be opened. See warnings()
In addition: Warning messages:
1: In RODBC::odbcDriverConnect("DSN=myDSN;UID=myID;PWD=badpassword") :
[RODBC] ERROR: state 28000, code 1017, message [Oracle][ODBC][Ora]ORA-01017: invalid username/password; logon denied
2: In RODBC::odbcDriverConnect("DSN=myDSN;UID=myID;PWD=badpassword") :
ODBC connection failed
The print(gsub(...)) appears to work on the most recent warnings from before the function was invoked, and it also only prints the function call that produced the warning, not the text of the warning.
What I would like to do is capture everything after "In addition: Warning messages:" so that I can use gsub() on it, but avoid printing it before the gsub() gets a chance to work on it. I think I need to use withCallingHandlers() but I've looked through the documentation and examples and I cannot figure it out.
Some extra background: This is an Oracle database that locks users out after three attempts to connect so I want to use stop() in case someone writes code that calls this function multiple times. Different users in my group work in both Windows and Linux (sometimes going back and forth) so any solution needs to be flexible.
Catching error messages
I do not fully understand what you want to accomplish with ODBC but in terms of converting the error message, you can use tryCatch as #joran suggested
pswd = 'badpassword'
# Just as a reproducable example, a function which fails and outputs badpassword
failing <- function(){
badpassword == 1
}
# This would be the error handling part
tryCatch(failing(),
error = function(e) gsub(pswd, '******', e))
[1] "Error in failing(): object '******' not found\n"
e in this case is the error message and you could think of other ways to manipulate what is put to your screen, so it would not be as easy to guess passwords based on what was replaced. Note for example that 'object' would have been replaced as well if the password had been 'object' for some reason. Or even parts of words, which get replaced as well. At the very least, it would make sense to include word boundaries in the gsub command:
pswd = 'ling'
failing <- function(){
ling == 1
}
tryCatch(failing(),
error = function(e) gsub(paste0("\\b", pswd, "\\b"), '******', e))
[1] "Error in failing(): object '******' not found\n"
For other improvements you should look closely at the specific error messages.
Warnings
trycatch can also manipulate warning:
pswd = 'ling'
failing <- function(){
warning("ling")
ling == 1
}
tryCatch(failing(),
warning = function(w) gsub(paste0("\\b", pswd, "\\b"), '******', w),
error = function(e) gsub(paste0("\\b", pswd, "\\b"), '******', e))
[1] "simpleWarning in failing(): ******\n"
This will not show the error then, however.
withCallingHandlers
If you really want to catch all output from errors and warnings, you do indeed need withCallingHandlers, which works mostly in the same way, except that it does not terminate the rest of the evaluation.
pswd = 'ling'
failing <- function(pswd){
warning(pswd)
warning("asd")
stop(pswd)
}
withCallingHandlers(failing(),
warning = function(w) {
w <- gsub(paste0("\\b", pswd, "\\b"), '******', w)
warning(w)},
error = function(e){
e <- gsub(paste0("\\b", pswd, "\\b"), '******', e)
stop(e)
})
I heard about exception handling first time in python two days ago and consequently I want to apply here in R. I had a look at a number of questions post either here in stack overflow or some other online Q&As but I am still really confused in using it.
I would really appreciate if someone can answer it with this simple example so later on I can apply it to my questions.
For example I have 3 data files with file names shown below; and the first file is a 0 bytes empty file. What I can do to continue run the loop for all files and the number extracted from the empty file can be expressed as NA?
> output_names_hdf5_list[1:5]
[1] "simulation-results fL=0.1,fks=1,fno=0.1,fnc=0.1,fr=0.1,fs=0.1.hdf5"
[2] "simulation-results fL=0.1,fks=1,fno=0.1,fnc=0.1,fr=0.1,fs=1.05.hdf5"
[3] "simulation-results fL=0.1,fks=1,fno=0.1,fnc=0.1,fr=0.1,fs=2.hdf5"
for (i in 1:5){
channelflow_outlet[,i]=h5read(paste(outputdir, output_names_hdf5_list[i], sep=""),"Channel")$Qc_out[460,][2:100]
}
With try function I can manage to run the program without stuck in an error message but when I replace the argunment with channelflow_outlet[,i]= h5read(....) inside try function, it just returns error.
for (i in 1:5){
try(h5read(paste(outputdir, output_names_hdf5_list[i], sep=""),"Channel")$Qc_out[460,][2:100])
}
Without error handling, it will have a error message like this.
> h5read(paste(outputdir, output_names_hdf5_list[1], sep=""),"Channel")$Qc_out[460,][2:100]
HDF5: unable to open file
Error in h5checktypeOrOpenLoc(file, readonly = TRUE) :
Error in h5checktypeOrOpenLoc(). File 'D:/Data/Mleonard/pytopkapi.staged.makefile/RunModel/Output/3x6-729-04072014/simulation-results fL=0.1,fks=1,fno=0.1,fnc=0.1,fr=0.1,fs=0.1.hdf5' is not a valid HDF5 file.
>
I hope my code helps. For those messages in the code, you can delete them if you want. They are here purely to help you see where it shows warning or error.
setwd("D:/Dropbox/Test/"); outputdir = "D:/Dropbox/Test/"
output_names_hdf5_list=c("simulation-results fL=0.1,fks=1,fno=1.05,fnc=1.05,fr=1.05,fs=1.05.hdf5",
"simulation-results fL=0.1,fks=1,fno=1.05,fnc=2,fr=1.05,fs=1.05.hdf5",
"simulation-results fL=0.1,fks=1,fno=2,fnc=1.05,fr=0.1,fs=1.05.hdf5",
"simulation-results fL=0.1,fks=1,fno=2,fnc=1.05,fr=2,fs=2.hdf5",
"simulation-results fL=0.5,fks=1,fno=2,fnc=2,fr=0.1,fs=1.05.hdf5")
channelflow_outlet = matrix(NA, nrow=100, ncol=5)
hdf5_list_reading_tool= function(output_names_hdf5_list) {
out = tryCatch(
{
message("This is the 'try' part")
h5read(paste(outputdir, output_names_hdf5_list, sep=""),"Channel")$Qc_out[460,][2:100]
},
error=function(cond) {
message("Here's the original error message:")
message(cond)
return(rep(NA,100))
},
warning=function(cond) {
message("Here's the original warning message:")
message(cond)
return(rep(NA,100))
},
finally={
message(paste("Processed URL:", output_names_hdf5_list))
message("Some other message at the end")
}
)
return(out)
}
channelflow_outlet=sapply(output_names_hdf5_list, hdf5_list_reading_tool)
I have the following code:
urls <- c(
"xxxxx",
"http://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html",
"http://en.wikipedia.org/wiki/Xz"
)
readUrl <- function(url) {
out <- tryCatch(
readLines(con=url, warn=FALSE),
error=function(e) {
message(paste("URL does not seem to exist:", url))
message(e)
return(NA)
},
finally=message(paste("Processed URL:", url))
)
return(out)
}
y <- lapply(urls, readUrl)
When I run it, I get:
URL does not seem to exist: xxxxx
cannot open the connectionProcessed URL: xxxxx
Processed URL: http://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html
Processed URL: http://en.wikipedia.org/wiki/Xz
Warning message:
In file(con, "r") : cannot open file 'xxxxx': No such file or directory
but I expected:
URL does not seem to exist: xxxxx
cannot open the connectionProcessed URL: xxxxx
Warning message:
In file(con, "r") : cannot open file 'xxxxx': No such file or directory
Processed URL: http://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html
Processed URL: http://en.wikipedia.org/wiki/Xz
So, why do I get:
Warning message:
In file(con, "r") : cannot open file 'xxxxx': No such file or directory
The call to readLines issues warnings. You can suppress warnings with suppressWarnings(). Try this:
readUrl <- function(url) {
out <- tryCatch(
suppressWarnings(readLines(con=url, warn=FALSE)),
error=function(e) {
message(paste("URL does not seem to exist:", url))
message(e)
return(NA)
},
finally=message(paste("Processed URL:", url))
)
return(out)
}
y <- lapply(urls, readUrl)
The screen output:
URL does not seem to exist: xxxxx
cannot open the connectionProcessed URL: xxxxx
Processed URL: http://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html
Processed URL: http://en.wikipedia.org/wiki/Xz
Alternatively, you can use options(warn=1) to display warnings as they occur. Try this:
readUrl <- function(url) {
op <- options("warn")
on.exit(options(op))
options(warn=1)
out <- tryCatch(
readLines(con=url, warn=FALSE),
error=function(e) {
message(paste("URL does not seem to exist:", url))
message(e)
return(NA)
},
finally=message(paste("Processed URL:", url))
)
return(out)
}
y <- lapply(urls, readUrl)
The output:
Warning in file(con, "r") :
cannot open file 'xxxxx': No such file or directory
URL does not seem to exist: xxxxx
cannot open the connectionProcessed URL: xxxxx
Processed URL: http://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html
Processed URL: http://en.wikipedia.org/wiki/Xz
To display all warnings, simply run this:
options(warn=1)
Here's how it works:
Set warn = -1 (or any negative integer), to ignore all warnings
Set warn = 0 (the default), to display 10 or fewer warnings
Set warn = 1 (as above), to display all warnings
Set warn = 2 (or any positive integer greater than 2) to convert all warnings into errors
Further explanation available by running ?options:
integer value to set the handling of warning messages. If warn is negative all warnings are ignored. If warn is zero (the default) warnings are stored until the top–level function returns. If 10 or fewer warnings were signalled they will be printed otherwise a message saying how many were signalled. An object called last.warning is created and can be printed through the function warnings. If warn is one, warnings are printed as they occur. If warn is two (or larger, coercible to integer), all warnings are turned into errors.