I am trying to copy files from a network drive location to a sharepoint library in R. The sharepoint library location requires user authentication and I was wondering how I can copy these files and pass authentication in code. A simple file.copy does not work. I was attempting to use the getURL() function from RCurl library but that hasn't worked either. I was wondering how I can accomplish this task - copying files while passing authentication.
Here are some code snippets that I have tried so far:
library(RCurl)
from <- "filename"
to <- "\\\\sharepoint.company.com\\Directory" #First attempt with just sharepoint location
to <- "file://sharepoint.company.com/Directory" #Another attempt with different format
h = getCurlHandle(header = TRUE, userpwd = "username:password")
getURL(to, verbose = TRUE, curl = h)
status <- file.copy(from, to)
Thank you!
Not the most elegant solution but if you're looking to save into a single library on SharePoint, you can first map that library as a drive on your local machine.
Simply use setwd() to point to whatever drive letter you mapped the library to. You can then treat that Sharepoint library as if it were any other shared drive location, reading and writing files from/to it.
I just use the following function to copy files to SharePoint. The only issue will be the file that was transferred will remain checked-out until the File is manually Checked-in for others to use.
saveToSharePoint <- function(fileName)
{
cmd <- paste("curl --max-time 7200 --connect-timeout 7200 --ntlm --user","username:password",
"--upload-file /home/username/FolderNameWhereTheFileToTransferExists/",fileName,
"teamsites.OrganizationName.com/sites/PageTitle/Documents/UserDocumentation/FolderNameWhereTheFileNeedsToBeCopied/",fileName, sep = " ")
system(cmd)
}
saveToSharePoint("SomeFileName.Ext")
If you have SharePoint online, you can navigate to that library and click on the "Sync to Computer" button (has an icon with arrows and a computer). Then you can have this as a oneDrive and write directly to it.
Related
I want to work on a cluster from my local Rstudio. So far, I'm using the following code to read a file:
to_read <- read.table(
pipe('ssh cluster_A "cat /path_on_cluster/file.txt"'),
h=T)
The autologin on cluster_A works fine as ssh directly gets my ~/.ssh/id_rsa file key.
There's 2 issues:
It doesn't work with fread, so can be quite slow to read
I haven't found a way to write files, only read
I was hoping to use scp as a workaround to these issues with something like that:
library(RCurl)
scp(host = "cluster_A",
path = "/path_on_cluster/file.txt",
keypasswd = NA, user = "user_name", rsa = TRUE,
key = "~/.ssh/id_rsa.pub")
But I can't find a way to make it work, as scp from R runs into some issues ("Protocol "scp" not supported or disabled in libcurl"). I can't find any good answer from a google search.
If anyone has a method to use fread and write.table (or other) between my local machine and a remote cluster, that would be very helpful!
I have been trying for three days now to download a file from an FTP server with R without a result. I have really tried everything and read all questions but still cannot manage.
The url is:
u <- "ftp://user:password#109.2.160.55/AGLO/2020/10/AGLO_00001_03-0_GDBX_1000077_202010032206_860101.CSV.gz"
When I copy paste this link in Firefox I can download the file, but with R I cannot. I tried download.file, GET, writeBin, getURL. All failed: getURL gives the following error:
Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) :
embedded nul in string: '\037‹\b\bøÙx_\0\003AGLO_00001_03-0_GDBX_1000077_202010032206_860101.CSV\0¬\\M³£¸’Ý¿_Áîº\002I Xɶ®*\fn>nÔMÇ›\231·èͼ\210î×\021óóç¤\004\006#¹.Œ§+¢ë’צŽ’TæÉ\017¡ÎUS*üï·\030ÿ±ßbñKüÛùtøþ\033#A–ýÆc\036ãgÁy,\177Ë%~f_ŽÝ{é~,é\v%}¡\\~ÐIN¿ÿùï?~ÿ\217¿þýÏ¿þ(\017±\034oY¾ýë¯?þû÷?ÿ$qù·Å/p\v÷\177{£òð]U½.umÊ¿\235»¦/{s~ƒ”\220–7ÓŸ¢Ã¿þø¯\177þã¯ÿ)ó¸ÈŠ\022_eEœã·îúz-K\031—í £¯ZÕÑW5´º+…H9/{Uéú¨K^BfNôsT©è]·n\215ŽÊ2\021œ©²¦¯”e/æ»7e\fIy‹\231Ä_"ayF?ÈñÏýsåQUGüâ3ô¼H’d\t\177\024Hàgœ•ê=ºªópR„\235笼\002á¹VÇ\aðºRFy°y\b6Ç_xz¹d¯Àf<—I2Â.\030›\004\f°3\002ZÓõ#\027\035Z£ê\023ÀDz(+\\7CwT}ÉJ\215¿»nè£þ¢\201\230Ô^>"§\033×ø3#çLäž¾éc›õ\035§‰`wàr\022\020pg-êøë »èÖjØCï\003çeý÷\037j\210êoM}n"ÝG׫ÆCÂ:£úï]S«è¢a_*°\034¹Z\016\036C\034XŽÜ¼œBð8YX\217»&ã‘\t=†›êz=´X…`yyÓ]g-G\177é¾Ü¾(ü颶9`\235Ñ\031ÎXË\016\033*RÚwÏmèûgàešf|\001Þ]\023xžYËï/F·\235}\004p\tM{Òj\200·);ݾ\033\230ýE\003úY_u\215‡`j,´Û¾\0^°8}iëf<\023Kå\217\002,àP2aÉß\006‚%åM\r¦ªì“¸Ý`jã%°“{ú±AùiÊ_s;ô_ºÄî\004Öí0\vý4ÜTÿá+ÿÚœL5þv‡¹0ž’ÃxÁ夒\025l\
There is no proxy problem whatsoever since to get the u url I am searching in the FTP directory.
How can I download this fing file ?
Also another way I could eventially work around this is using:
browseURL(u,
browser = "C:/Program Files (x86)/Mozilla Firefox/firefox.exe")
The issue with this is that it will open a Firefox browser that will:
Ask me if I am sure I want to go to this site and then
Ask me what I want to do with this file (not so much of a problem since I can choose a default to always download, but still I do not want to)
The issue with this is that simply I do not want to open a browser and I do not want to be asked if I want to go to the site and if I want to download. There are many files on this server so I want to do all of this automatically and I will need to be working in parllel, so having a browser pop up is not great, but if all else fails I can accept.
I am so desperate that I can give you the user name and password in private.
Apparently downloading the file to disk using httr solved the problem. It is possible to combine write_disk and httr::GET to download files to disk in the following way:
library(httr)
to_download <- "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
# Download pdf to disk
GET(to_download, write_disk("dummy.pdf"))
I have some files in my company's sharepint folder. I need to use r to download all files in that folder. How can I do that with R?
I googled the question and got this link. It's about uploading the file to a sharepoint folder.
[Uploading files to SharePoint from R
The code from the link is like below:
saveToSharePoint <- function(fileName)
{
cmd <- paste("curl --max-time 7200 --connect-timeout 7200 --ntlm --user","username:password",
"--upload-file /home/username/FolderNameWhereTheFileToTransferExists/",fileName,
"teamsites.OrganizationName.com/sites/PageTitle/Documents/UserDocumentation/FolderNameWhereTheFileNeedsToBeCopied/",fileName, sep = " ")
system(cmd)
}
saveToSharePoint("SomeFileName.Ext")
It's about uploading one specific file. But what I need is download files from sharepoint folder.
so, I modified the code to copy from sharepoint folder. I changed --upload-file to --download-file.
copyFromSharePoint <- function(fileName)
{
cmd <- paste("curl --max-time 7200 --connect-timeout 7200 --ntlm --user","username:password",
"--download-file teamsites.OrganizationName.com/sites/PageTitle/Documents/FolderNameWhereTheFileToTransferExists/",fileName,
"home/username/UserDocumentation/FolderNameWhereTheFileNeedsToBeCopied/",fileName, sep = " ")
system(cmd)
}
copyFromSharePoint("SomeFileName.Ext")
However, that does not seem to work. Error message below:
curl: option --download-file: is unknown
Does anyone know how I can do it with R?
In addition, what if I need to download all files in the folder, not one specific file.Does anyone know how I can do it with R?
Solution
I made a simple R Package called sharepointr to upload and download files from SharePoint.
The way I made the whole download/upload to sharepoint work from R was to:
Make a SharePoint App Registration
Add permissions to App Registration
Getting "tenant_id" and "resource_id" using cURL
Make get/post calls to API
It is all described in the package Readme.md
Example
# Installing the package
install.packages("devtools")
devtools::install_github("esbeneickhardt/sharepointr")
# Setting parameters
client_id <- "insert_from_first_step"
client_secret <- "insert_from_first_step"
tenant_id <- "insert_from_fourth_step"
resource_id <- "insert_from_fourth_step"
site_domain <- "yourorganisation.sharepoint.com"
sharepoint_url <- "https://yourorganisation.sharepoint.com/sites/MyTestSite"
# Getting a SharePoint Token
sharepoint_token <- get_sharepoint_token(client_id, client_secret, tenant_id, resource_id, site_domain)
# Getting sharepoint digest value
sharepoint_digest_value <- get_sharepoint_digest_value(sharepoint_token, sharepoint_url)
# Downloading a file
sharepoint_path <- "Shared Documents/test"
sharepoint_file_name <- "Mappe.xlsx"
out_path <- "C:/Users/User/Desktop/"
download_sharepoint_file(sharepoint_token, sharepoint_url, sharepoint_digest_value, sharepoint_path, sharepoint_file_name, out_path)
I'm hosting my first shiny app from www.shinyapps.io. My r script uses a glm I created locally that I have stored as a .RDS file.
How can I read this file into my application directly using a free file host such as dropbox or google drive? (or another better alternative?)
test<-readRDS(gzcon(url("https://www.dropbox.com/s/p3bk57sqvlra1ze/strModel.RDS?dl=0")))
however, I get the error:
Error in readRDS(gzcon(url("https://www.dropbox.com/s/p3bk57sqvlra1ze/strModel.RDS?dl=0"))) :
unknown input format
I assume this is because the URL doesn't lead directly to the file but rather a landing page for dropbox?
That being said, I can't seem to find any free file hosting sites that have that functionality.
As always, I'm sure the solution is very obvious, any help is appreciated.
I figured it out. Hosted the file in a GitHub repository. From there I was able to copy the link to the raw file and placed that link in the readRDS(gzcon(url())) wrappers.
Remotely reading using readRDS() can be disappointing. You might want to try this wrapper that saves the data set to a temporary location before reading it locally:
readRDS_remote <- function(file, quiet = TRUE) {
if (grepl("^http", file, ignore.case = TRUE)) {
# temp location
file_local <- file.path(tempdir(), basename(file))
# download the data set
download.file(file, file_local, quiet = quiet, mode = "wb")
file <- file_local
}
readRDS(file)
}
I want to deploy a basic trained R model as a webservice to AzureML. Similar to what is done here:
http://www.r-bloggers.com/deploying-a-car-price-model-using-r-and-azureml/
Since that post the publishWebService function in the R AzureML package was has changed it now requires me to have a workspace object as first parameter thus my R code looks as follows:
library(MASS)
library(AzureML)
PredictionModel = lm( medv ~ lstat , data = Boston )
PricePredFunktion = function(percent)
{return(predict(PredictionModel, data.frame(lstat =percent)))}
myWsID = "<my Workspace ID>"
myAuth = "<my Authorization code"
ws = workspace(myWsID, myAuth, api_endpoint = "https://studio.azureml.net/", .validate = TRUE)
# publish the R function to AzureML
PricePredService = publishWebService(
ws,
"PricePredFunktion",
"PricePredOnline",
list("lstat" = "float"),
list("mdev" = "float"),
myWsID,
myAuth
)
But every time I execute the code I get the following error:
Error in publishWebService(ws, "PricePredFunktion", "PricePredOnline", :
Requires external zip utility. Please install zip, ensure it's on your path and try again.
I tried installing programs that handle zip files (like 7zip) on my machine as well as calling the utils library in R which allows R to directly interact with zip files. But I couldn't get rid of the error.
I also found the R package code that is throwing the error, it is on line 154 on this page:
https://github.com/RevolutionAnalytics/AzureML/blob/master/R/internal.R
but it didn't help me in figuring out what to do.
Thanks in advance for any Help!
The Azure Machine Learning API requires the payload to be zipped, which is why the package insists on the zip utility being installed. (This is an unfortunate situation, and hopefully we can find a way in future to include a zip with the package.)
It is unlikely that you will ever encounter this situation on Linux, since most (all?) Linux distributions includes a zip utility.
Thus, on Windows, you have to do the following procedure once:
Install a zip utility (RTools has one and this works)
Ensure the zip is on your path
Restart R – this is important, otherwise R will not recognize the changed path
Upon completion, the litmus test is if R can see your zip. To do this, try:
Sys.which("zip")
You should get a result similar to this:
zip
"C:\\Rtools\\R-3.1\\bin\\zip.exe"
In other words, R should recognize the installation path.
On previous occasions when people told me this didn’t work, it was always because they thought they had a zip in the path, but it turned out they didn’t.
One last comment: installing 7zip may not work. The reason is that 7zip contains a utility called 7zip, but R will only look for a utility called zip.
I saw this link earlier but the additional clarification which made my code not work was
1. Address and Path of Rtools was not as straigt forward
2. You need to Reboot R
With regards to the address - always look where it was installed . I also used this code to set the path and ALWAYS ADD ZIP at the end
##Rtools.bin="C:\\Users\\User_2\\R-Portable\\Rtools\\bin"
Rtools.bin="C:\\Rtools\\bin\\zip"
sys.path = Sys.getenv("PATH")
if (Sys.which("zip") == "" ) {
system(paste("setx PATH \"", Rtools.bin, ";", sys.path, "\"", sep = ""))
}
Sys.which("zip")
you should get a return of
" C:\\RTools|\bin\zip"
From looking at Andrie's comment here: https://github.com/RevolutionAnalytics/AzureML/commit/9cf2c5c59f1f82b874dc7fdb1f9439b11ab60f40
Implies we can just download RTools and be done with it.
Download RTools from:
https://cran.r-project.org/bin/windows/Rtools/
During installation select the check box to modify the PATH
At first it didn't work. I then tried R32bit, and that seemed to work. Then R64 bit started working again. Honestly, not sure if I did something in the middle to make it work. Only takes a few minutes so worth a punt.
Try the following
-Download the Rtools file which usually contains the zip utility.
-Copy all the files in the "bin" folder of "Rtools"
-Paste them in "~/RStudio/bin/x64" folder