How to fix Error cannot open connection on Ubuntu machine? - r

This might be a very trivial issue with R-related Ubuntu settings, but I'm pretty new to it and have no clue to solve this issue.
I'm trying to load a file from URL on a linux Ubuntu machine into R (R-Studio) with different strategies:
load(url("http://www.yaj.ch/data/test.rda"))
read.csv(url("http://www.yaj.ch/data/test.csv"))
readstata13::read.dta13("http://www.yaj.ch/data/test.dta")
data.table::fread("http://www.yaj.ch/data/test.csv")
but they all fail with similar error messages:
Error in load(url("http://www.yaj.ch/data/test.rda")) :
cannot open the connection to 'http://www.yaj.ch/data/test.rda'
In addition: Warning message:
In load(url("http://www.yaj.ch/data/test.rda")) :
cannot open URL 'http://www.yaj.ch/data/test.rda': HTTP status was '403 Forbidden'
that of fread is somewhat different:
trying URL 'http://www.yaj.ch/data/test.csv'
Error in download.file(input, tmpFile, method = method, mode = "wb", quiet =
!showProgress) :
cannot open URL 'http://www.yaj.ch/data/test.csv'
In addition: Warning message:
In download.file(input, tmpFile, method = method, mode = "wb", quiet =
!showProgress) :
cannot open URL 'http://www.yaj.ch/data/test.csv': HTTP status was '403 Forbidden'
This is what I got when I was mobile:
Error in load(url("http://www.yaj.ch/data/test.rda")) :
cannot open the connection to 'http://www.yaj.ch/data/test.rda'
Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> load
Execution halted
When I Shift-click on the link in the script, the data opens in the browser, so the connection seems to be okay.
On my Windows machine everything works fine. So I guessed the issue is related to Ubuntu, as if R has connection problems or wrong access rights. But I'm not sure, since strangely enough e.g. this one opens in Ubuntu:
load(url("http://www.rdatamining.com/data/graph.rdata?attredirects=0&d=1"))
This solution wasn't helpful to me, and Google won't tell me anything either.
Is there any help?

Related

Web Scraping with R: error related to reset of the connection with server

I have a problem with obtaining data from specific website - when trying to download raw website data with R 3.6.3 using following example code:
website_raw <- readLines("https://tge.pl/gaz-rdn?dateShow=09-02-2022")
The result I got is:
Error in file(con, "r") : cannot open the connection In addition: Warning message: In file(con, "r") : InternetOpenUrl failed: 'the connection with the server was reset'
readLines() method used to work fine on this website but from one week on it fails. I've tried also download.file() method: at the beginning the result was the same (error, connection reset) but after setting options(download.file.method = "libcurl"), website file starts to download but then it suddenly stops with information:
trying URL 'https://tge.pl/gaz-rdn?dateShow=09-02-2022'
Error in download.file("https://tge.pl/gaz-rdn?dateShow=09-02-2022", "test.html") :
cannot open URL 'https://tge.pl/gaz-rdn?dateShow=09-02-2022'
In addition: Warning message:
In download.file("https://tge.pl/gaz-rdn?dateShow=09-02-2022", "test.html") :
URL 'https://tge.pl/gaz-rdn?dateShow=09-02-2022': status was 'Failure when receiving data from the peer'
I've tried also disabling Use Internet Explorer library/proxy for HTTP in Rstudio Global Options but it didn't help. Another solution that I've tested was read_html() from rvest package - getting following error:
Error in open.connection(x, "rb") : Send failure: Connection was reset
Downloading data from other websites works fine though, with all considered methods.
Is there any way I can download data from this website with R?
Any kind of help or suggestion will be highly appreciated

R CMD check fails with ubuntu when trying to download file, but function works within R

I am writing an R package and one of its functions download and unzips a file from a link (it is not exported to the user, though):
download_f <- function(download_dir) {
utils::download.file(
url = "https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php",
destfile = file.path(download_dir, "fines.rar"),
mode = 'wb',
method = 'libcurl'
)
utils::unzip(
zipfile = file.path(download_dir, "fines.rar"),
exdir = file.path(download_dir)
)
}
This function works fine with me when I run it within some other function to compile an example in a vignette.
However, with R CMD check in github action, it fails consistently on ubuntu 16.04, release and devel. It [says][1]:
Error: Error: processing vignette 'IBAMA.Rmd' failed with diagnostics:
cannot open URL 'https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php'
--- failed re-building ‘IBAMA.Rmd’
SUMMARY: processing the following file failed:
‘IBAMA.Rmd’
Error: Error: Vignette re-building failed.
Execution halted
Error: Error in proc$get_built_file() : Build process failed
Calls: <Anonymous> ... build_package -> with_envvar -> force -> <Anonymous>
Execution halted
Error: Process completed with exit code 1.
When I run devtools::check() it never finishes running it, staying in "creating vignettes" forever. I don't know if these problems are related though because there are other vignettes on the package.
I pass the R CMD checks with mac os and windows. I've tried switching the "mode" and "method" arguments on utils::download.file, but to no avail.
Any suggestions?
[1]: https://github.com/datazoompuc/datazoom.amazonia/pull/16/checks?check_run_id=2026865974
The download fails because libcurl tries to verify the webservers certificate, but can't.
I can reproduce this on my system:
trying URL 'https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php'
Error in utils::download.file(url = "https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php", :
cannot open URL 'https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php'
In addition: Warning message:
In utils::download.file(url = "https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php", :
URL 'https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php': status was 'SSL peer certificate or SSH remote key was not OK'
The server does not allow you to download from http but redirects to https, so the only thing to do now is to tell libcurl to not check the certificate and accept what it is getting.
You can do this by specifying the argument -k to curl
download_f <- function(download_dir) {
utils::download.file(
url = "https://servicos.ibama.gov.br/ctf/publico/areasembargadas/downloadListaAreasEmbargadas.php",
destfile = file.path(download_dir, "fines.rar"),
mode = 'wb',
method = 'curl',
extra = '-k'
)
utils::unzip(
zipfile = file.path(download_dir, "fines.rar"),
exdir = file.path(download_dir)
)
}
This also produces some download progress bar, you can silence this by setting extra to -k -s
This now opens you up to a Machine In The Middle Attack. (You possibly already are attacked this way, there is no way to check without verifying the current certificate with someone you know at the other side)
So you could implement an extra check, e.g. check the sha256sum of the downloaded file and see if it matches what you expect to receive before proceeding.
myfile <- system.file("fines.rar")
hash <- sha256(file(myfile))

Error Detecting Locale when deploying shiny

When I try to deploy my shiny application, I go this error. I tried to setlocale but it didnt help. Why did I get this error, what should I do?
Uploading bundle for document: 2646593...Detecting system locale ... Error in curl::curl_fetch_memory(url, handle = handle) :
Send failure: Connection was reset
Calls: ... tryCatch -> tryCatchList -> tryCatchOne ->
In addition: Warning message:
Error detecting locale: Error in read.table(file = file, header = header, sep = sep, quote = quote, : incomplete final line found by readTableHeader on 'raw'
(Using default: en_US)
Timing stopped at: 0.09 0.17 25.96
Execution halted
It's a little difficult to tell where you're deploying to or what the function error might be (is this to shinyappa.io? RStudio Connect?)
Have you tried look into the warning message? Addressing the read.table error might help, either the formatting of the file being uploaded could be throwing an error or your application may not be able to properly locate it after deployment.
Check out this set of answers to a similar read.table error.

issue in getting data with getGEO in R

I want to download the soft file of the GPL6480 platform in R.
I've used this command :
gset<-getGEO("GPL6480")
but I faced with this problem:
curl: (6) Could not resolve host: www.ncbi.nlm.nih.gov
File stored at:
/tmp/RtmpbHgZqQ/GPL6480.soft
Error in file(fname, "r") : cannot open the connection
In addition: Warning messages:
1: In download.file(myurl, destfile, mode = mode, quiet = TRUE, method = getOption("download.file.method.GEOquery")) :
download had nonzero exit status
2: In file(fname, "r") :
cannot open file '/tmp/RtmpbHgZqQ/GPL6480.soft': No such file or directory
I googled for this problem but I can't find the answer. acctauly i don't know that what is the problem exactly.you should know that my connection to net is ok.
I'm using R version 3.2.0 on Ubuntu 14.04.2 trusty
tnx mansoor

download.file "operation timed out" error with large files

I'm using R 3.1.2 with RStudio 0.98 on Windows 7 32 bits.
I want to download some weather forecasts files of the GFS model, to be found on an open ftp server, e.g.:
ftp://nomads.ncdc.noaa.gov/GFS/Grid4/201412/20141221/gfs_4_20141221_0000_000.grb2
The internet connection is done through a proxy (.Renviron is properly configured), and I'm basically using the donwload.file function for this purpose.
url <- file.path("ftp://nomads.ncdc.noaa.gov/GFS/Grid4/201412/20141221/gfs_4_20141221_0000_000.grb2")
download.file(url, destfile="temp.grb2", mode="wb")
Where I get the following error message:
trying URL 'ftp://nomads.ncdc.noaa.gov/GFS/Grid4/201412/20141221/gfs_4_20141221_0000_000.grb2'
using Synchronous WinInet calls
Error in download.file(url, destfile = "temp.grb2", mode = "wb", :
cannot open URL 'ftp://nomads.ncdc.noaa.gov/GFS/Grid4/201412/20141221/gfs_4_20141221_0000_000.grb2'
In addition: Warning message:
In download.file(url, destfile = "temp.grb2", mode = "wb", :
InternetOpenUrl failed: 'Operation timed out'
This message appears exactly 30 seconds after running those lines, and no issues appear when downloading a smaller file, such as 'ftp://nomads.ncdc.noaa.gov/GFS/Grid4/201412/20141221/gfs_4_20141221_0000_000.inv', so I assume it's a matter of timeout configuration.
Setting:
options(timeout=240) doesn't seem to work.
With another computer, using R 3.0.2 with RStudio 0.98 on Windows 8 64 bits, and without using proxy connection, it works perfect.
Any suggestions?

Resources