I am having problems getting RCurl function getURL to access an HTTPS URL on a server that is using a self-signed certificate. I'm running R 3.0.2 on Mac OS X 10.9.2.
I have read the FAQ and the curl page on the subject. So this is where I stand:
I have saved a copy of the certificate to disk (~/cert.pem).
I have been able to use this very same file to connect to the server using python-requests and the 'verify' option, and succeeded.
curl on the command-line seems to be ignoring the --cacert option. I succeeded in accessing the website with it after I flagged the certificate as trusted using the Mac OS X 'Keychain Access' app.
RCurl stubbornly refuses to connect to the website with the following code:
getURL("https://somesite.tld", verbose=T, cainfo=normalizePath("~/cert.pem"))
This is the output I get:
* Adding handle: conn: 0x7f92771b0400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 38 (0x7f92771b0400) send_pipe: 1, recv_pipe: 0
* About to connect() to somesite.tld port 443 (#38)
* Trying 42.42.42.42...
* Connected to somesite.tld (42.42.42.42) port 443 (#38)
* SSL certificate problem: Invalid certificate chain
* Closing connection 38
When I tested both curl with the --cacert option and the RCurl code above in a Linux VM with the same cert.pem file and exact same URL, it worked perfectly.
So equal tests on Linux and Mac OS X, and only on Mac OS X do they fail. Even adding the certificate to the keychain didn't work.
The only thing that does work is using ssl.verifypeer=FALSE, but I don't want to do that for security reasons.
I'm out of ideas here. Anyone else have any suggestions on how to get this to work?
You can try:
library ("RCurl")
URL1 <- "https://data.mexbt.com/ticker/btcusd"
getURL(URL1,cainfo=system.file("CurlSSL","cacert.pem",package="RCurl"))
Coming back to this issue I just wanted to point out that if you are still using RCurl, you should be using httr (which uses curl) instead.
I have confirmed that using config(cainfo="/path/to/certificate") with httr connections will work as intended.
Related
I am developing an Asp.Net application on MacOS with F# (.NET 6.0.301). While writing code I run a dotnet watch:
dotnet watch run -v --project Server/Server.fsproj
and send a curl message to one of the api endpoints of the server
curl -k -i -d "#loginInfo.json" -H "Accept: application/json" -H "Content-Type: application/json" -v 'https://localhost:5001/services/IAdminApi/login'
* Trying 127.0.0.1:5001...
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
[...] // More handshake data
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
which returns the expected result. It worked seamessly until a few months ago, when I started to receive the following error
* Trying 127.0.0.1:5001...
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection 0
curl: (35) error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
However, when I run the server directly without the watch command:
/usr/local/share/dotnet/dotnet Server/bin/Debug/net6.0/Server.dll
everything works perfectly, and the API sends back the proper info. The server uses a self-signed certificate that is read from file.
Everything is running locally in a macOS machine. I have tried in two machines with different macOS versions, and problems started after updating to Monterey 12.6 and Ventura 13. Now both machines run updated versions (Monterey 12.6.2 and Ventura 13.1), but the problem persists.
However, dotnet watch works as expected in Windows 10. Codes are run from a terminal, without any intervention from the IDE (Rider in my case). Even though I lean towards something at the os level, also tried sending the curl command with the --tlsv1.x --tls-max 1.x options (x=0,1,2,3) with no luck. The version of curl is 7.79.1.
Any pointer to keep investigating will be greatly appreciated.
I think that that might be something related to how the dotnet watch command handles encrypted traffic. as per this page: https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-watch
"As part of dotnet watch, the browser refresh server mechanism reads this value to determine the WebSocket host environment. The value 127.0.0.1 is replaced by localhost, and the http:// and https:// schemes are replaced with ws:// and wss:// respectively."
So perhaps while the https traffic (when you run the application without dotnet watch) works fine because it uses appropriate cyphers and version of TLS, there is a bug or some omission in the implementation of the wss protocol, where TLS is fixed to version 1.
It would appear that you have two options.
run your application on local host without https
configure your operating system to allow for TLS 1.0
I'm running R on a Windows machine behind a corporate proxy. When running R 3.3.2 and version 2.3 of the curl package, this works:
curl::curl_fetch_memory("https://google.com")
Also note the following:
> curl::curl_options(filter = "sslcert")
sslcert sslcerttype
10025 10086
However, this is what happens when running R 3.4.1 and curl 2.8:
> curl::curl_fetch_memory("https://google.com")
Error in curl::curl_fetch_memory("https://google.com") :
error setting certificate verify locations:
CAfile: /mingw32/ssl/certs/ca-bundle.crt
CApath: none
Note:
> curl::curl_options(filter = "sslcert")
proxy_sslcert proxy_sslcerttype sslcert sslcerttype
10254 10255 10025 10086
I gather from the curl source code that these two new options were added in between these two versions.
Since this is the only relevant difference that I can find, my hunch is that curl is looking for the proxy_sslcert in the wrong path. But I can't figure out how to change that variable, nor what I should change it to. Also, what are the numbers shown (10254, etc)?
I don't actually need to use curl specifically, but a lot of other packages rely on it, e.g.
> httr::GET("https://google.com")
Error in curl::curl_fetch_memory(url, handle = handle) :
error setting certificate verify locations:
CAfile: /mingw32/ssl/certs/ca-bundle.crt
CApath: none
I have downloaded and installed R (3.4.1)and R studio (1.0.153) on MAC OS ( El Captain). I am facing trouble setting up the proxy. I am trying to install some packages from CRAN.
I referred to https://support.rstudio.com/hc/en-us/articles/200488488-Configuring-R-to-Use-an-HTTP-or-HTTPS-Proxy
I was able to set up proxy and used properties
http_proxy=http://MYSERVER:MYPORT
http_proxy_user=user:passwd
I enabled options(internet.info = 0) option to get a trace of internet access from R Studio.
While the proxy server set up looks good in logs, its not able to get authentication.
Here are the logs:
Connected to MYSERVER port MYPORT (#0)
> GET http://cran.rstudio.com/src/contrib/PACKAGES.gz HTTP/1.1
Host: cran.rstudio.com
User-Agent: R (3.4.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)
Accept: */*
Proxy-Connection: Keep-Alive
Pragma: no-cache
* The requested URL returned error: 407 Proxy Authentication Required
* Closing connection 0
Warning: unable to access index for repository http://cran.rstudio.com/src/contrib:
cannot open URL 'http://cran.rstudio.com/src/contrib/PACKAGES'
(Please note: I have replaced the proxy server and port )
I tried changing the
http_proxy_user=ask
As suggested in one of the work around I read, but that also didn't work.
Could anyone suggest how to set up proxy authentication properly in R studio on Mac?
I have Eclipse installed on my local using the same proxy port and authentication and it is able to download stuff fine.
I am able to hit the url http://cran.rstudio.com/src/contrib/PACKAGES.gz directly in firefox.
I was finally able to make it work. The user id and password was accepted by adding them in the same property
http_proxy=http://username:password#proxyServer:port
This created another issue, my password contained # char, which is why the proxy url was messed up. I referred to another post on stacktrace which suggested to try urlencoding the "#" in the password to %40, With this change, it finally went through the proxy.
options(internet.info = 0) helped look through the detailed log which helped solve the issue.
I'm trying to fetch the following URL via various libraries and command-line tools, but running into a SSL handshake error.
> curl https://www.joelosteen.com/Views/RSS/Feed?t=PodcastVideo&ct=CustomList&cst=Podcasts
curl: (35) error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
I've seen here it may be fixed by reducing possible ciphers, but trying this less strict call with restricted ciphers doesn't fix it.
curl -vlkL "https://www.joelosteen.com/Views/RSS/Feed?t=PodcastVideo&ct=CustomList&cst=Podcasts" --ciphers DHE-RSA-AES256-SHA
Any ideas how I can fix this? It's a shared account, ie no root available.
The server supports only TLS version 1.2, i.e. no TLS 1.0 or even SSL 3.0. My guess is that the curl you use is simply too old to use TLS 1.2, but without further details about the version you use this guess can not be verified.
To determine the version of curl use curl --version. If you see it is compiled with OpenSSL than it should show at least version 1.0.1 of OpenSSL to have TLS 1.2 support.
curl -vlkL ... --ciphers DHE-RSA-AES256-SHA
This is not a cipher supported by the server. One of the ciphers supported is AES128-SHA but again, no chance if your curl does not support TLS 1.2.
I'm trying to get devtools::install_github() working behind my corporate proxy on Windows 7.
So far I've had to do the following:
> library(httr)
> library(devtools)
> set_config(use_proxy("123.123.123.123",8080))
> devtools::install_github("rstudio/ggvis")
Installing github repo ggvis/master from rstudio
Downloading master.zip from https://github.com/rstudio/ggvis/archive/master.zip
Error in function (type, msg, asError = TRUE) :
SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Apparently we have some kind of certificate server replacing SSL certs with our own corporate SSL certs (confirmed by going to https://github.com and examining the cert).
Anyhow, just wondering if there's a way to ignore that cert error and proceed with the installation?
One way to handle the problem is to set the CURLOPT_SSL_VERIFYPEER to false. This option determines whether curl verifies the authenticity of the peer's certificate. A value of 1 means curl verifies; 0 (zero) means it doesn't.
http://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html
The relevant option needs to be passed to RCurl. In RCurl the CURLOPT_ is removed letters arre lowercase and the underscore is changed to ..
set_config( config( ssl.verifypeer = 0L ) )
will pass the relevant option to RCurl when using httr.
UPDATE:
The httr since this answer was written has moved from RCurl as an underlying dependence to the curl package. cURL options are now specified
with underscores so the above would be:
set_config( config( ssl_verifypeer = 0L ) )
in the current version of httr.
I tired #jdharrisonbut method but it didn't worked for me.
A Quick fix to avoid using https connection, is by replacing https by http. you install a package from cran by this method.
install.packages("http://cran.r-project.org/src/contrib/ggvis_0.4.4.tar.gz", repo=NULL, type="source")