Is getURL deprecated? [duplicate] - r

ETA: Per https://github.com/hiratake55/RForcecom/issues/42, it looks like the author of the rforcecom package has updated rforcecom to use httr instead of RCurl (as of today, to be uploaded to CRAN tomorrow, 7/1/16), so my particular issue will be solved at that point. However, the general case (implementing TLS 1.1 / 1.2 in RCurl) may still be worth pursuing for other packages. Or everyone may just switch to the more recent curl package instead of RCurl.
Background: I've been using the rforcecom package to communicate with Salesforce for several months. Salesforce recently disabled support for TLS v1.0 and is requiring TLS v1.1 or higher in their Sandboxes; this update will take place for Production environments in March 2017.
rforcecom uses RCurl to communicate with salesforce.com servers. Generally the curlPerform method is used, which is implemented something like this (this example is from rforcecom.login.R):
h <- basicHeaderGatherer()
t <- basicTextGatherer()
URL <- paste(loginURL, rforcecom.api.getSoapEndpoint(apiVersion), sep="")
httpHeader <- c("SOAPAction"="login","Content-Type"="text/xml")
curlPerform(url=URL, httpheader=httpHeader, postfields=soapBody, headerfunction = h$update, writefunction = t$update, ssl.verifypeer=F)
This has been working for me for a while, as I mentioned. Now that Salesforce has disabled TLS v1.0 on Sandboxes, though, it fails with the following error:
UNSUPPORTED_CLIENT: TLS 1.0 has been disabled in this organization. Please use TLS 1.1 or higher when connecting to Salesforce using https.
I've modified and sourced (without implementing it in the local copy of the package, as I'm not experienced enough to do this) a change to my local copy of the login module for RForcecom, and I've discovered through experimentation that I can specify any of the existing enumerated values for SSLVERSION successfully by adding sslversion=SSLVERSION_TLSv1, sslversion=SSLVERSION_SSLv3, etc. to the curlPerform options where it is called. However, all of these give me the same error as above. When I attempt to use one of the options that is implemented in libcurl but not in RCurl (SSLVERSION_TLSv1.1, SSLVERSION_TLSv1.2), I get the following error:
Error in merge(list(...), .opts) : object 'SSLVERSION_TLSv1.1' not found
or:
Error in merge(list(...), .opts) : object 'SSLVERSION_TLSv1.2' not found
I've verified with curlVersion() that my libcurl version is 7.40.0, which according to https://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html does support those options. However, I'm unable to get RCurl to recognize them.
At this point what I'm looking for is a way to get RCurl to use TLS v1.1 or TLS v1.2, and I would very much appreciate any assistance I can get with that. I apologize for any problems/issues with my question as this is my first time asking one myself, I've previously always been able to muddle through by reading other people's questions and answers.

RCurl is an interface to libcurl and what it supports depends on the latter. It is possible that your libcurl was built with an older version of OpenSSL which does not have support for TLS v1.1 or v.1.2. Your can determine your SSL version from R like this:
RCurl::curlVersion()$ssl_version
I think that by default (e.g. ssl option CURL_SSLVERSION_DEFAULT), during the SSL handshake, the server and client would agree on the newest version that they both support. To make it work you would have to update OpenSSL to a newer version, recompile libcurl with it and rebuild RCurl so that it registers the updates.
That said, you can enforce a particular ssl version not defined in RCurl by passing the integer value of the required option yourself. The numbers you are looking for can be deduced from the C enum defined in the curl header file on GitHub:
enum {
CURL_SSLVERSION_DEFAULT, // 0
CURL_SSLVERSION_TLSv1, /* TLS 1.x */ // 1
CURL_SSLVERSION_SSLv2, // 2
CURL_SSLVERSION_SSLv3, // 3
CURL_SSLVERSION_TLSv1_0, // 4
CURL_SSLVERSION_TLSv1_1, // 5
CURL_SSLVERSION_TLSv1_2, // 6
CURL_SSLVERSION_TLSv1_3, // 7
CURL_SSLVERSION_LAST /* never use, keep last */ // 8
};
For example:
# the data of you post request
nameValueList = list(data1 = "data1", data2 = "data2")
CURL_SSLVERSION_TLSv1_1 <- 5L
CURL_SSLVERSION_TLSv1_2 <- 6L
# TLS 1.1
opts <- RCurl::curlOptions(verbose = TRUE,
sslversion = CURL_SSLVERSION_TLSv1_1, ...)
# TLS 1.2
opts <- RCurl::curlOptions(verbose = TRUE,
sslversion = CURL_SSLVERSION_TLSv1_2, ...)
# finally, POST the data
RCurl::postForm(URL, .params = nameValueList, .opts = opts)
In general, this may not be a good practice because the authors of cURL could decide to change the values (although I believe chances are low) those options have.

Related

In R how do you authenticate with an NTLM proxy when using the curl package

The R4.2.0 update made libcurl the default for the download.file function (and removed the wininet method for some types of url schemes). The same is true for the download.packages function.
My work uses a proxy.pac file in conjunction with NTLM authentication for the proxy. As a result, the changes made to the download.file make it not work with this type of proxy unless you specifically choose the wininet option (but this only works for some url schemes).
One of the consequences of this confluence of configurations is that users have to manually change the default package manager to be an internal mirror of CRAN to use the package installer at all in RStudio.
I've managed to use the curl library to be able to download files through the proxy:
base_directory <- "<save_path>"
handle <- curl::new_handle()
url <- "url_page_to_download"
handle_opts <- list(
proxy = curl::ie_get_proxy_for_url(url), # figures out the relevant proxy string for the url (or url stub you are using)
proxyauth = 8, # value of CURLAUTH_NTLM from curl_symbols()
proxyuserpwd = ":" # specifically set to “:” to force windows to do the negotiation. This string is the same as sending no username and password
)
handle_setopt(handle,.list=handle_opts)
data_file_location <- paste0(base_directory,"state.txt")
curl_download(url, destfile=data_file_location, quiet=FALSE, handle=handle)
I would like to find a way to do this that uses the libcurl method of the download.packages and download.file functions, which involves setting some environment variables (from what I can tell), but I can't find the set of parameters to place in the environment variables to get it to work.
How can I replicate this with the built in functions?

What is the version of SSL/TLS in Tensorflow serving

For Tensorflow serving 1.7.0, What is the version of SSL/TLS being used. Is it boringssl or OpenSSL? Please let me know the version number
TensorFlow Serving uses gRPC which in turn uses/can use boringssl. As to the concrete version, are you referring to the TLS protocol versions (i.e. 1.1 to 1.3) or the actual source code revision of the boringssl implementation ?
The latter is somewhat difficult, as there are no named releases of boringssl in the traditional sense.
TensorFlow Serving 1.7.0 seems to use a grpc version that includes the following version:
https://boringssl.googlesource.com/boringssl/+archive/886e7d75368e3f4fab3f4d0d3584e4abfc557755.tar.gz
I have not found a way to map back from the SHA256 hash (886e7d75368e3f4fab3f4d0d3584e4abfc557755) to the commit ID in the boringssl/master-with-google branch.
There is some TLS1.3 code in that version of boringssl, but checking against my tensorflow_model_server with SSL enabled (note that the stock Google server does not use SSL at all), it looks like TLS1.2 is in use (which makes sense to the newness of 1.3):
openssl s_client -connect my-ssl-enabled-tensorflowserver:myport -tls1_2
yields the appropriate "let's use this" response.

R: download data securely using TLS/SSL

Official Statements
In the past the base R download.file() was unable to work with HTTPS protocols and it was necessary to use RCurl. Since R 3.3.0:
All builds have support for https: URLs in the default methods for download.file(), url() and code making use of them. Unfortunately that cannot guarantee that any particular https: URL can be accessed. ... Different access methods may allow different protocols or use private certificate bundles ...
The download.file() help still says:
Contributed package 'RCurl' provides more comprehensive facilities to download from URLs.
which (by the way includes cookies and headers management).
Based on RCurl FAQ (look for "When I try to interact with a URL via https, I get an error"), HTTPS URLs can be managed with:
getURL(url, cainfo="CA bundle")
where CA bundle is the path to a certificate authority bundle file. One such a bundle is available from the curl site itself:
https://curl.haxx.se/ca/cacert.pem
Current status
Tests are based on Windows platforms
For many HTTPS websites download.file() works as stated:
download.file(url="https://www.google.com", destfile="google.html")
download.file(url="https://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")
As regards RCurl, using the cacert.pem bundle, downloaded above, one might get an error:
library(RCurl)
getURL("https://www.google.com", cainfo = "cacert.pem")
# Error in function (type, msg, asError = TRUE) :
# SSL certificate problem: unable to get local issuer certificate
In this instance, simply removing the reference to the certificate bundle solves the problem:
getURL("https://www.google.com") # works
getURL("https://www.google.com", ssl.verifypeer=TRUE) # works
ssl.verifypeer = TRUE is used to be sure that success is not due to getURL() suppressing security. The argument is documented in RCurl FAQ.
However, in other instances, the connection fails:
getURL("https://curl.haxx.se/ca/cacert.pem")
# Error in function (type, msg, asError = TRUE) :
# error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
And similarly, using the previously downloaded bundle:
getURL("https://curl.haxx.se/ca/cacert.pem", cainfo = "cacert.pem")
# Error in function (type, msg, asError = TRUE) :
# error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
The same error happens even when suppressing the security:
getURL("https://curl.haxx.se/ca/cacert.pem", ssl.verifypeer=FALSE)
# same error as above
Questions
How to use HTTPS properly in RCurl?
As regards mere file downloads (no headers, cookies, etc.), is there any benefit in using RCurl instead of download.file()?
Is RCurl become obsolete and should we opt for curl?
Update
The issue persists as of
R version 3.4.1 (2017-06-30) under Windows 10.
openssl bundled with RCurl is a bit old currently, which does not support the TLS v1.2
Yes, curl package is OK
Or you can use httr package which is a wrapper for the curl package
> library("httr")
> GET("https://curl.haxx.se/ca/cacert.pem",config(sslversion=6,ssl_verifypeer=1))
Response [https://curl.haxx.se/ca/cacert.pem]
Date: 2017-08-16 17:07
Status: 200
Content-Type: application/x-pem-file
Size: 256 kB
<BINARY BODY>

TLS v1.1 / TLS v1.2 support in RCurl

ETA: Per https://github.com/hiratake55/RForcecom/issues/42, it looks like the author of the rforcecom package has updated rforcecom to use httr instead of RCurl (as of today, to be uploaded to CRAN tomorrow, 7/1/16), so my particular issue will be solved at that point. However, the general case (implementing TLS 1.1 / 1.2 in RCurl) may still be worth pursuing for other packages. Or everyone may just switch to the more recent curl package instead of RCurl.
Background: I've been using the rforcecom package to communicate with Salesforce for several months. Salesforce recently disabled support for TLS v1.0 and is requiring TLS v1.1 or higher in their Sandboxes; this update will take place for Production environments in March 2017.
rforcecom uses RCurl to communicate with salesforce.com servers. Generally the curlPerform method is used, which is implemented something like this (this example is from rforcecom.login.R):
h <- basicHeaderGatherer()
t <- basicTextGatherer()
URL <- paste(loginURL, rforcecom.api.getSoapEndpoint(apiVersion), sep="")
httpHeader <- c("SOAPAction"="login","Content-Type"="text/xml")
curlPerform(url=URL, httpheader=httpHeader, postfields=soapBody, headerfunction = h$update, writefunction = t$update, ssl.verifypeer=F)
This has been working for me for a while, as I mentioned. Now that Salesforce has disabled TLS v1.0 on Sandboxes, though, it fails with the following error:
UNSUPPORTED_CLIENT: TLS 1.0 has been disabled in this organization. Please use TLS 1.1 or higher when connecting to Salesforce using https.
I've modified and sourced (without implementing it in the local copy of the package, as I'm not experienced enough to do this) a change to my local copy of the login module for RForcecom, and I've discovered through experimentation that I can specify any of the existing enumerated values for SSLVERSION successfully by adding sslversion=SSLVERSION_TLSv1, sslversion=SSLVERSION_SSLv3, etc. to the curlPerform options where it is called. However, all of these give me the same error as above. When I attempt to use one of the options that is implemented in libcurl but not in RCurl (SSLVERSION_TLSv1.1, SSLVERSION_TLSv1.2), I get the following error:
Error in merge(list(...), .opts) : object 'SSLVERSION_TLSv1.1' not found
or:
Error in merge(list(...), .opts) : object 'SSLVERSION_TLSv1.2' not found
I've verified with curlVersion() that my libcurl version is 7.40.0, which according to https://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html does support those options. However, I'm unable to get RCurl to recognize them.
At this point what I'm looking for is a way to get RCurl to use TLS v1.1 or TLS v1.2, and I would very much appreciate any assistance I can get with that. I apologize for any problems/issues with my question as this is my first time asking one myself, I've previously always been able to muddle through by reading other people's questions and answers.
RCurl is an interface to libcurl and what it supports depends on the latter. It is possible that your libcurl was built with an older version of OpenSSL which does not have support for TLS v1.1 or v.1.2. Your can determine your SSL version from R like this:
RCurl::curlVersion()$ssl_version
I think that by default (e.g. ssl option CURL_SSLVERSION_DEFAULT), during the SSL handshake, the server and client would agree on the newest version that they both support. To make it work you would have to update OpenSSL to a newer version, recompile libcurl with it and rebuild RCurl so that it registers the updates.
That said, you can enforce a particular ssl version not defined in RCurl by passing the integer value of the required option yourself. The numbers you are looking for can be deduced from the C enum defined in the curl header file on GitHub:
enum {
CURL_SSLVERSION_DEFAULT, // 0
CURL_SSLVERSION_TLSv1, /* TLS 1.x */ // 1
CURL_SSLVERSION_SSLv2, // 2
CURL_SSLVERSION_SSLv3, // 3
CURL_SSLVERSION_TLSv1_0, // 4
CURL_SSLVERSION_TLSv1_1, // 5
CURL_SSLVERSION_TLSv1_2, // 6
CURL_SSLVERSION_TLSv1_3, // 7
CURL_SSLVERSION_LAST /* never use, keep last */ // 8
};
For example:
# the data of you post request
nameValueList = list(data1 = "data1", data2 = "data2")
CURL_SSLVERSION_TLSv1_1 <- 5L
CURL_SSLVERSION_TLSv1_2 <- 6L
# TLS 1.1
opts <- RCurl::curlOptions(verbose = TRUE,
sslversion = CURL_SSLVERSION_TLSv1_1, ...)
# TLS 1.2
opts <- RCurl::curlOptions(verbose = TRUE,
sslversion = CURL_SSLVERSION_TLSv1_2, ...)
# finally, POST the data
RCurl::postForm(URL, .params = nameValueList, .opts = opts)
In general, this may not be a good practice because the authors of cURL could decide to change the values (although I believe chances are low) those options have.

Sourcing R script over HTTPS

Is there some way to source an R script from the web?
e.g. source('http://github.com/project/R/file.r')
Reason: I currently have a project that I'd like to make available for use but isn't ready to be packaged yet. So it would be great to give people a single file to source from the web (that will then source all the individual function files).
On closer inspection, the problem appears to be https. How would I source this file?
https://raw.github.com/hadley/stringr/master/R/c.r
You can use the source_url in the devtools library
library(devtools)
source_url('https://raw.github.com/hadley/stringr/master/R/c.r')
This is a wrapper for the RCurl method by #ROLO
Yes you can, try running this R tutorial:
source("http://www.mayin.org/ajayshah/KB/R/tutorial.R")
(Source)
Https is only supported on Windows, when R is started with the --internet2 command line option (see FAQ):
> source("https://pastebin.com/raw.php?i=zdBYP5Ft")
> test()
[1] "passed"
Without this option, or on linux, you will get the error "unsupported URL scheme". In that case resort to the solution suggested by #ulidtko, or:
Here is a way to do it using RCurl, which also supports https:
library(RCurl)
eval( expr =
parse( text = getURL("http://www.mayin.org/ajayshah/KB/R/tutorial.R",
ssl.verifypeer=FALSE) ))
(You can remove the ssl.verifypeer if the ssl certificate is valid)
Yes, it is possible and worked for me right away.
R> source("http://pastebin.com/raw.php?i=zdBYP5Ft")
R> test()
[1] "passed"
Regarding the HTTPS part, it isn't supported by internal R code. However, R can use external utilities like wget or curl to fetch https:// URLs. One will need to write additional code to be able to source the files.
Sample code might be like this:
wget.and.source <- function(url) {
fname <- tempfile()
download.file(url, fname, method="wget")
source(fname)
unlink(fname)
}
There is a Windows-only solution too: start R with --internet2 commandline option. This will switch all the internet code in R to using IE, and consequently HTTPS will work.
Windows:
If Internet Explorer is configured to access the web using your organization's proxy, you can direct R to use these IE settings instead of the default R settings. This change can be made once by the following steps:
Save your work and close all R sessions you may have open.
Edit the following file. (Note: Your exact path will differ based on your R installation)
C:\Program Files\R\R-2.15.2\etc\Rprofile.site
Open this "Rprofile.site" file in Notepad and add the following line on a new line at the end of the file:
utils::setInternet2(TRUE)
You may now open a new R session and retry your "source" command.
Linux alikes:
Use G. Grothendieck's suggestion. At the command prompt within R type:
source(pipe(paste("wget -O -", "https://github.com/enter/your/url/here.r")))
You may get an error saying:
cannot verify certificate - - - - Self-signed certificate encountered.
At this point it is up to you to decide whether you trust the person issuing the self-signed certificate and proceed or to stop.
If you decide to proceed, you can connect insecurely as follows:
source(pipe(paste("wget -O -", "https://github.com/enter/your/url.r", "--no-check-certificate")))
For more details, see the following:
See section 2.19
CRAN R Documentation 2.19
wget documentation section 2.8 for "no-check-certificate"
Similar questions here:
Stackoverflow setInternet2 discussion
Stackoverflow Proxy configuration discussion
The methods here were giving me the following error from github:
OpenSSL: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
I used the following function to resolve it:
github.download = function(url) {
fname <- tempfile()
system(sprintf("curl -3 %s > %s", url, fname))
return(fname)
}
source(github.download('http://github.com/project/R/file.r'))
Hope that helps!
This is working for me on windows:
library(RCurl)
# load functions and scripts from github ----------------------------
fn1 <- getURL("https://raw.githubusercontent.com/SanjitNarwekar/Advanced-R-Programming/master/fn_factorial_loop.R", ssl.verifypeer = FALSE)
eval(parse(text = fn1))

Resources