proxy problems for install_keras with Anaconda and RStudio - r

I'm trying to install Keras by following the instructions here https://keras.rstudio.com/ when I'm using my office/work laptop.
When I get to this line install_keras() and it fails with this rather long error message shown here in its entirety:
Creating r-tensorflow conda environment for TensorFlow installation...
Fetching package metadata ...
CondaHTTPError: HTTP 000 CONNECTION FAILED for url
<https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on
your way.
ProxyError(MaxRetryError("HTTPSConnectionPool(host='repo.continuum.io',
port=443): Max retries exceeded with url: /pkgs/main/win-6
/repodata.json.bz2 (Caused by ProxyError('Cannot connect to proxy.',
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
0x000001ACD38FB780>: Failed to establish a new connection: [Errno 11001]
getaddrinfo failed',)))",),)
Error: Error 1 occurred creating conda environment r-tensorflow
In addition: Warning message:
running command '"C:\PROGRA~3\ANACON~1\Scripts\conda.exe" "create" "--
yes" "--name" "r-tensorflow" "python=3.6"' had status 1
and it says something about Anaconda. So I dutifully install the entire Anaconda beast from this Anaconda distribution.
Now when I try install_keras() I get this new error message about a proxy:
Using r-tensorflow conda environment for TensorFlow installation
Determining latest release of TensorFlow...Error in
open.connection(con, "rb") :
Unsupported proxy 'https://proxy-server.mycompanyname.com:8080', libcurl
is built without the HTTPS-proxy support.
I reviewed Jeroen's libcurl suggestion which yields the following responses:
curl::ie_proxy_info()
$AutoDetect
[1] FALSE
$AutoConfigUrl
[1] "http://mcd-server/mcd/proxy.pac"
$Proxy
NULL
$ProxyBypass
NULL
and this
curl::ie_get_proxy_for_url()
[1] "proxy-server:8080"
So I seem to have a good connection to the internet through RStudio but not when using Anaconda's software.
I have verified that I have a good internet connection using this:
httr::BROWSE("https://www.ibm.com")
It appears that libcurl is built without HTTPS-proxy support, apparently.
Any suggestions?

SOLVED!
1) Step 1: allow Anaconda to access the internet by adding proxy info with a new file named .condarc exactly as detailed in this answer. You can verify this works ok by typing conda update conda into the Anaconda Prompt app.
2) Step 2: allow R and RStudio to access the internet by adding these 2 lines into the .Renviron file (in my case this was found in C:\Users\USERNAME\Documents for Windows 10):
http_proxy=http://proxy-server:8080
https_proxy=http://proxy-server:8080
You can now run this to install Keras:
library(keras)
install_keras()

Related

Getting: "Failed to fetch metadata" when starting up Jupyter

I am using JupyterLab as my IDE and I have a couple of packages installed. Namely, the 'jupyterlab-dash' and 'juptyerlab-plotly' packages. My issue is when I go to launch 'juptyer lab' from terminal, I notice the following error:
Failed to fetch package metadata for 'jupyterlab-dash': URLError(gaierror(8, 'nodename nor servname provided, or not known'))
I'm not sure why this error is popping up but I've noticed this error only pops up when I have no internet connection (working on a train etc.). Could it be because these packages are trying to call something as I launch and because I don't have an internet connection it raises the error?
In the end I am able to use JupyterLab and the packages as intended (at-least it seems), but I'm curious why this error to 'fetch metadata' appears?
Thanks,

Installing a package from private GitLab server on Windows

I am struggling with installing a package from a GitLab repository on a Windows computer.
I found different hints but still have problems to install my package from GitLab. First of all, I generated a public and private key with puttygen.exe. The files need to be changed afterwards, I had to remove comments and stuff so they look like my the file on my Unix system. So now, both public and private key files have just a single line.
I tried to install my package via devtools::install_git which takes very long and I get the error message
Error: Failed to install 'unknown package' from Git:
Error in 'git2r_remote_ls': Failed to authenticate SSH session: Unable to send userauth-publickey request
And with devtools::install_gitlab I get a different error message and I somehow have the feeling, the link which gets generated doesn't fit to my GitLab server.
Error: Failed to install 'unknown package' from GitLab:
cannot open URL 'https://gitlab.rlp.net/api/v4/projects/madejung%2FMQqueue.git/repository/files/DESCRIPTION/raw?ref=master'
My complete code to test at the moment is
creds <- git2r::cred_ssh_key(publickey="~/.ssh/id_rsa_gitlab.pub",
privatekey="~/.ssh/id_rsa_gitlab")
devtools::install_git(
url='git#gitlab.rlp.net:madejung/MQqueue.git',
quiet=FALSE,
credentials=creds)
devtools::install_gitlab(
repo='madejung/MQqueue.git',
host='gitlab.rlp.net',
quiet=FALSE,
credentials=creds
)
My id_rsa_gitlab.pub file looks like this and is just a single line:
ssh-rsa AAAA....fiwbw== rsa-key-20200121
The id_rsa_gitlab file has just the code:
AAABA.....3WNSIAGE=
Update
On my Mac system it works as expected after installing the libssh2 library via homebrew and and recompiling git2r with install.packages("git2r", type = "source").
So the working code on my machine is:
creds <- git2r::cred_ssh_key(publickey="~/.ssh/id_rsa_gitlab.rlp.net.pub",
privatekey="~/.ssh/id_rsa_gitlab.rlp.net")
devtools::install_git(
url='git#gitlab.rlp.net:madejung/MQqueue.git',
quiet=FALSE,
credentials=creds
)
For some strange reason, the devtools::install_git call needs about a minute to fail in the end. I have no idea where the problem here is.
After struggling for almost a day, I found a solution I can live with...
I first created a PAT (Personal Access Token) in my gitlab account and granted full API access. For some reason the read_only access didn't worked and I am now tired to figure out what the problem is.
After this I had still problems to install my package and for some reason, the wininet setting for downloading doesn't work.
I used the command capabilities("libcurl") to check if libcurl is available on my windows, which was and tried to overwrite wininet to libcurl by using method='libcurl' in the install function. Somehow, this was not enough so I overwrote the options variable download.file.method directly.
options("download.file.method"='libcurl')
devtools::install_gitlab(
repo='madejung/MQqueue',
auth_token='Ho...SOMETHING...xugzb',
host='gitlab.rlp.net',
quiet=FALSE, force=TRUE
)

devtools::install_github error: SEC_E_UNTRUSTED_ROOT - The certification chain was issued by an entity that is unreliable

I am trying to install an package from behind a corporate (fire)wall using devtools:
library(devtools)
devtools::install_github("aryoda/tryCatchLog")
I get an error message:
Error: Failed to install 'unknown package' from GitHub: schannel:
next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325)
- The certification chain was issued by an entity that is unreliable.
The reason seems to be the used curl package which produces the same error with:
library(curl)
curl::curl_fetch_memory("https://httpbin.org/get")
How can I fix this?
PS: I am using MS Windows 10
I have found the solution:
The internet connection does only work within curl if you set the correct HTTPS_PROXY:
# insert your correct domain name and IP port here
Sys.setenv(https_proxy = "http://httpproxy.mycompany.com:1234")
This devtools issue comment did help me:
https://github.com/r-lib/devtools/issues/1610#issuecomment-333344548
Update 1:
This is a generic solution to set the HTTP(S)_PROXY in R:
requires(curl)
requires(devtools)
proxy <- curl::ie_get_proxy_for_url("https://www.qwant.com/")
Sys.setenv(https_proxy=proxy)
# Sys.setenv(http_proxy=proxy) # you could also set an HTTP proxy
devtools::install_github("aryoda/tryCatchLog") # should work now
You could add this line to your Rprofile.site file (in R/etc folder)

Unable to run Rmpi and spawn slaves

I'm really naive and know almost nothing about MPI. I'm trying to do some analyses in R that require Rmpi package. But I'm stuck with this problem:
Version of R 3.2.5 (x64) on Windows 10
I installed MPICH2
Added proper addresses on PATH environment variables
Successfully set smpd and mpiexec registration
Installed Rmpi using Rgui
If I call Rmpi using normal Rterm it is loaded. But when I try
mpi.spawn.Rslaves(), it returns :
Error in mpi.spawn.Rslaves() : Spawning is not implemented. Please use mpiexec with Rprofile.*
so I run on prompt :
mpiexec -n 2 "C:\Program Files\R\R-3.2.5\bin\x64\Rterm.exe" --no-save –q
So far so good.
I can load any package without problems but when it comes to call Rmpi library I get this error message:
Aborting: mpi appplication on DESKTOP-0UI5TAT is unable to connect to the smpd manager on (null):57362 error 1722
job aborted:
[ranks] message
[0] fatal error
Fatal error in MPI_Init: Other MPI error, error stack:
MPI_Init(argc_p=0x0018FA44, argv_p=0x0018FA48) failed
The RPC server is unavailable. (errno 1722)*
Even using the more recent MS-MPI software, when I call Rmpi it simply starts a new line without any prompt > and so it remains
I deep searched on the whole internet and tried several options but it's still not working
you have to create a shortcut that Rstudio is lunched by mpiexec ,otherwise the Rstudio doesn't know Mpi
follow instruction here
http://www.stats.uwo.ca/faculty/yu/Rmpi/

install.keras() in RStudio fails with http connection error

I've been trying to install and run keras in RStudio (Windows) in vain.
i installed keras package using normal package "keras"
(didn't use github)
I've installed latest python (3.6) and Anaconda.
then i use
> library(keras)
> install.keras()
and i get this error:
Creating r-tensorflow conda environment for TensorFlow installation...
Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION
FAILED for url
https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2
Elapsed: -
An HTTP error occurred when trying to retrieve this URL. HTTP errors
are often intermittent, and a simple retry will get you on your way.
ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='repo.continuum.io',
port=443): Max retries exceeded with url:
/pkgs/main/win-64/repodata.json.bz2 (Caused by
ConnectTimeoutError(, 'Connection to repo.continuum.io timed out.
(connect timeout=9.15)'))",),)
Error: Error 1 occurred creating conda environment r-tensorflow In
addition: Warning message: running command
'"C:\PROGRA~3\ANACON~1\Scripts\conda.exe" "create" "--yes" "--name"
"r-tensorflow" "python=3.6"' had status 1
I've looked up everywhere on the web and can't figure out how to install keras and tensorflow properly. Using latest version of R (3.4.2)
Every method fails somewhere.
just to add to misery, i've also tried:
> devtools::install_github("rstudio/keras")
and i get this error:
Installation failed: Timeout was reached: Connection timed out after
10015 milliseconds
I am not behind any authenticated proxies. So, after multiple failure, i just downloaded the zip file from github and manually installed it using the zip file.
i also tried install.packages("keras") and that didn't give me any error either.
when i call the library i don't get any errors (as shown above)
UPDATE: I was able to install and use the package very easily on another computer that doesn't have python/anaconda installed on it already.
UPDATE 2: my proxy does not need authentication and there is no https_proxy either.
OK,, FINALLY found a solution.
Turns out RStudio uses a lot of default proxy settings, so i needed to change all that and set up my own proxy settings.
First step:
Rstudio --> Tools --> Global Options --> packages --> uncheck both "Use secure download method for HTTP" and "Use Internet Explorer librayr/proxy for HTTP"
Second step, in RStudio type:
> file.edit('./.Renviron')
Either an empty file or some file with already existing proxy settings will open. (Mine was empty). Then I included the following two:
http_proxy=http://myusename:password#proxy.server.com:port/
https_proxy=http://myusename:password#proxy.server.com:port/
(a few notes: I didn't have a https_proxy setting but I still needed to use the http_proxy details for my https_proxy setting. This was one of the culprits for my issue. Also, I needed to include the username:password even though my proxy doesn't need secure authentication. Same thing goes with the port. Port number had to be included, otherwise it wouldn't work.
Step 3:
Saved the new changes in .Renviron file and restarted RStudio.
I checked my proxy settings in RStudio after restart by typing:
> Sys.getenv("http_proxy")
> Sys.getenv("https_proxy")
The first few times i did this i realised that the proxy settings were not being changed in RStudio because i was editing the wrong .Renviron file. So, it's best to use file.edit('~/.Renviron') in step 2 to make sure it's the right file.
After all this, when i ran install.keras(), it installed successfully, including installing Tensorflow. Again, initially i had skipped step 1 so keras started being installed but it failed at installing tensorflow.
It was only going through all the steps that i was able to install both keras and tensorflow successfully over a proxy. Hope this helps.
Uninstalling Anaconda3 and installing Anaconda2 (i.e. Python 2.7) did the trick for me: https://www.anaconda.com/download/

Resources