I am trying to use the RCurl library to access an SFTP site to download files on a MacOS running Monterrey v12.4. As has happened to others, when RCurl calls the curl libraries SFTP is not enabled.
Following SFTP Support for curl on OSX I installed curl with openssl using homebrew. I uninstalled the curl and Rcurl libraries in RStudio.
In a terminal window, running 'curl -v' shows that sftp is available.
In RStudio, running "system('which curl')" shows sftp is available.
However, when I try to retrieve a file via SFTP using the RCurl library, I receive the message:
Error in function (type, msg, asError = TRUE) :
Protocol "sftp" not supported or disabled in libcurl
I thought maybe the PATH was not set correctly. I added the following line to my .Renviron file.
PATH=/opt/homebrew/opt/curl/bin:$PATH
At this stage, it's not clear to me why sftp is not supported when trying to access a file on an SFTP site when using the RCurl library.
What else can I do to try to diagnose why this is occurring?
I found a solution. In the post
SFTP Support for curl on OSX
there is code to enable SFTP support in curl on MacOS.
I ran the three lines:
PATH <- Sys.getenv("PATH")
version <- '7.86.0'
Sys.setenv(PATH = paste0("/opt/homebrew/Cellar/curl/", version, "/bin:", PATH))
and SFTP is enabled in RCurl.
an "echo $PATH" run from the terminal shows "/opt/homebrew/opt/curl/bin" in the path. I assumed this meant the OpenSSL enabled curl was being used, but that is not the case. Apparently the R PATH is different than the MacOS PATH.
My next step is to add this line to my .Rprofile file so that it is included each time I start R.
Related
First off - I know this isn't a specific code problem, so feel free to vote to close the question, but I've spent the better part of half of my day today struggling with this problem and could use some help. I also think this thread can help anybody trying to get sftp protocol working in R, since I'll share what I've done today.
I have been attempting to update RCurl so it supports sftp protocol. In R, my protocols look as such (with version and host as well):
> library(RCurl)
> curlVersion()$protocols
[1] "dict" "file" "ftp" "ftps" "gopher" "http" "https" "imap" "imaps" "ldap" "ldaps" "pop3" "pop3s" "rtsp" "smb" "smbs"
[17] "smtp" "smtps" "telnet" "tftp"
> curlVersion()$version
[1] "7.43.0"
> curlVersion()$host
[1] "x86_64-apple-darwin15.0"
Not great - no sftp option...
I followed this thread - http://andrewberls.com/blog/post/adding-sftp-support-to-curl - to update curl on my machine, and was partially successful in doing so. The success part is reflected in when I run the following in the command line:
curl -V
curl 7.55.1 (x86_64-apple-darwin15.6.0) libcurl/7.55.1 zlib/1.2.5
libssh2/1.8.0
Release-Date: 2017-08-14
Protocols: dict file ftp gopher http imap ldap ldaps pop3 rtsp scp sftp smtp telnet tftp
Features: AsynchDNS IPv6 Largefile libz UnixSockets
This is great, because I've got curl updated. However, when I load RCurl and run curlVersion() in R, it's not updated. I've examined the lib folders of my /usr/ directory and have the following problem. In both my /usr/lib AND /usr/local/lib, there exists libcurl files. Specifically:
libcurl.3.dylib, libcurl.4.dylib, and libcurl.dylib in /usr/lib
libcurl.4.dylib, libcurl.dylib, libcurl.a and libcurl.la in /usr/local/lib
The files in my /usr/local/lib directory are the new files that I'd like used, however here is where my major headache began. I copied the four files from /usr/local/lib into /usr/lib and I BROKE MY COMPUTER. Many of my applications stopped loading, and I had to reinstall my OS X on account of being afraid of messing anything else up. When I reinstalled my OS X, it put my files back into their respective folders (how they look in the bullets above).
If I had to guess, I probably shouldn't have moved the .a or .la files... I'm not sure.
One last thing - when i run $ $PATH in console, I get:
$PATH
-bash: /Users/Home/.rvm/gems/ruby-2.3.3/bin:/Users/Home/.rvm/gems/ruby-2.3.3#global/bin:/Users/Home/.rvm/rubies/ruby-2.3.3/bin:/Users/Home/anaconda2/bin:/usr/local/git/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin:/Users/Home/.rvm/bin: No such file or directory
/usr/local/bin is before /usr/bin. but I don't see anything with /usr/local/lib or /usr/lib - could I add these (/usr/local/lib) to my path to fix this? .libPaths() in R gives me this:
.libPaths()
"/Library/Frameworks/R.framework/Versions/3.4/Resources/library"
...if it helps at all
EDIT - I should note that the root of all problems, ofcourse, is that since 'sftp' is not listed in curlVersion()$protocols in R, that when i run RCurl::getURL(), or RCurl::ftpUpload() (im hoping to use ftpUpload), for both I receive the error:
> RCurl::ftpUpload(what = 'myfile.txt', to = 'sftp://userid:userpw#ip/'myfile.txt')
Error in function (type, msg, asError = TRUE) :
Protocol "sftp" not supported or disabled in libcurl
Thanks!
Phew! Finally got this working in an Ubuntu 14.04 docker container with rstudio-server. I hit every issue! But I had to spend more hours than I care to count finding all this stuff, so trying to put it all in one place for anyone else having this problem Here's what I did...
In the linux terminal:
apt-get install libcurl4-openssl-dev
wget libssh2 and then compile with ordinary
./configure
make
make install
wget curl and then compile
./configure --with-sshlib2=/usr/local
make
make install
If you need curl to work in the terminal with sftp, it still may not at this point. Verify its using the new libcurl.so.4 that you just installed. The curl you just installed will install into /usr/local/bin, so use ldd to verify its not using an old library. Mine was using an old libcurl so I linked the old libcurl to the new libcurl in /usr/local/lib and then recompiled curl.
If you want RCurl to work, copy all of the new libs from /usr/local/lib (libcurl & libssh2) to /lib, then move to the R console.
In Rconsole:
install.packages("RCurl", type = "source")
verify sftp protocol is enabled
libcurlVersion()
RCurl::curlVersion()
These should show you enabled protocols for libcurl and RCurl respectively. Cross your fingers for sftp!
Just in case, I also did these things that probably didn't do much but I did when I was hitting a wall and just kept in place.
upgraded R and R studio-server
removed the R version of curl and RCurl prior to reinstalling RCurl. Also in newer rstudio am not sure the type="source" is needed.
Also as Canovice noted, be careful moving around libcurl and libssh libraries. I broke networking a couple of times, fortunately since it was a docker container I just restarted. I found that moving everything at once worked best,
cp /usr/local/lib/libssh* /lib && cp /usr/local/lib/libcurl* /lib
Should also note I'm a total R novice so there may be cleaner ways to do this than moving libraries all over the place. I tried utilizing .libaths() in rstudio, but gave up in the end.
Installing RCurl from source did the trick for me !
install.packages("RCurl", type="source")
I found the solution here - http://www.omegahat.net/RCurl/FAQ.html - specifically, the following lines.
"I can't use scp or sftp within RCurl but the documentation for curl seems to suggest that it can. So why does RCurl not support it?
RCurl does support it, but the likely problem is that the version of libcurl you have installed does not support it. You can check what protocols your libcurl and hence RCurl supports via the command curlVersion. If scp and sftp are not there, reinstall libcurl but with support using libssh2. You will need to have the libssh2 development libraries and headers installed before installing libcurl. On some OSes, you will need to rebuild RCurl from source."
In general, this is a great summary of my issue from start to finish. If your version of RCurl doesn't support sftp, you need to install (at the command line) libssh2 first, then reinstall libcurl, then rebuild from source for some reason.
In practice i've found it more dangerous to try updating an existing curl and openssl installation, than installing an entirely new version and updating PATH envars where they'll be used
In my case my newer curl and openssl with sftp support where installed into
/usr/local/opt/curl-openssl/bin
/usr/local/opt/openssl/lib/
Consequently I needed to set the following envars
PATH="/usr/local/opt/curl-openssl/bin:${PATH}"
LIBRARY_PATH="/usr/local/opt/openssl/lib/:${LIBRARY_PATH}"
before executing
install.packages("RCurl", type="source")
I am trying to expose a sample r model as API so I created R package. In my local machine , I am able to get the output using the command :
curl http://local host/5656/ocpu/library/mypackage/R/tv/json -F "input=#test.csv"
But when I am trying the same in my AWS linux cloud server on CentOS , I am not getting the output. I tried the command :
curl 13.228.109.233:8787/p/5656/ocpu/library/mypackage/R/tv/json -F "input=#test.csv"
and I get the output as
http://13.228.109.233:8787/auth-sign-in?appUri=%2Fp%2F5656%2Focpu%2Flibrary%2Fmypackage%2FR%2Ftv%2Fjson .
The R package is loaded into the my R studio server and I am trying to access it using putty. I installed the open cpu using the command :
yum install opencpu-server
It looks like it is asking for some authentication but I am not able to get what it means. Am I missing something here?
Thanks
If one visits the URL with a browser, one sees the RStudio-Server login screen. And indeed, you are connecting to port 8787, which is the default port for Rstudio Server. Try URLs below http://13.228.109.233/ocpu/ instead.
As Ralf suggests, that was the error , I was giving the wrong URL . Also, whenever we install opencpu in CentOS , just doing a 'yum install opencpu-server' might not be sufficient. We need to execute the entire built script: https://github.com/opencpu/opencpu-server/blob/master/rpm/buildscript.sh
Without executing the build script , I was not able to connect to opencpu server.
Just in case someone gets the same issue.
I use Linux Ubuntu 16.04. I have well installed php-cs-fixer for atom 1.9.8.
$ php-cs-fixer
PHP CS Fixer version 1.11.6 by Fabien Potencier
Usage:
command [options] [arguments]
Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
Available commands:
fix Fixes a directory or a file
help Displays help for a command
list Lists commands
readme Generates the README content, based on the fix command help
self-update Update php-cs-fixer.phar to the latest version.
selfupdate Update php-cs-fixer.phar to the latest version.
The problem comes with config of php-cs-fixer with atom. In fact, package requires:
a PHP Executable path (which is by defaut php) (for linux user it must be empty which is not possible with actual version of the package) (see isse #6 )
a PHP-CS fixer Executable path (which is simply for linux user php-cs-fixer (without the use of PHAR file))
Consequently, when I run php-cs-fixer from atom I get this error: Could not open input file: php-cs-fixer.
I can generate this error message with this console command:
php php-cs-fixer fix Class.php
Could not open input file: php-cs-fixer
So, can you make PHP Executable path optional (not compulsory) for that package of atom???
thanks,
Update on this issue:
I managed to download php-cs-fixer.phar from https://github.com/FriendsOfPHP/PHP-CS-Fixer and I put it in the folder ~/.composer/ so that:
a PHP Executable path is php
a PHP-CS fixer Executable path is: ~/.composer/php-cs-composer.phar
But now I am getting this error: Could not open input file: ~/.composer/php-cs-fixer.phar
So what's wrong??
For you information, running the console php ~/.composer/php-cs-composer.phar Class.php command is successful.
The solution is found on github:
a PHP Executable path is php
a PHP-CS fixer Executable path is: /home/username/.composer/php-cs-composer.phar and don't use the ~ in atom.
In R I want to connect to a remote server and read a file. I am using
scp("host", "file path", "password", user="username")
It is returning
could not find function scp
The scp function is provided as part of the RCurl package. If you haven't done so already, install the latest version of the package:
Open an R terminal and execute the following command to install it:
R> install.packages("RCurl", dependencies = TRUE)
If it asks about using a personal library, enter y
You will then be prompted to select a mirror. Just pick a location somewhat near you for a hopefully-quicker download speed.
Now, at the top of your R script calling scp, add the following line:
library("RCurl")
this will allow you to use scp in your R script.
When I login to RStudio server installed as an Amazon Machine Image, I, I see the message "RStudio Server Initialization Error" and "Status code 500 returned," as in the screenshot. When I click OK, nothing changes.
What does this mean and how can I fix this?
Refer the following link. It will guide you about the complete installation of R-Studio Server along with required dependencies.
Automated Installation of R-Studio Using Shell Script
It would be useful if you could state which version of RStudio Server you are using. Neverthless, you can try the following:
Install the most recent version using: wget http://download2.rstudio.org/rstudio-server-0.98.1103-amd64.deb. If you want 32-bit version wget http://download2.rstudio.org/rstudio-server-0.98.1103-i386.deb.
Delete the startup files (.Rprofile, .Renviron, and .RData) from your initial working directory, which should be user's home.
Check if you can run R from command line and if your are getting any error messages - fix that first