I would like to source an .R file from a private gitlab serveur. I need to use the basic authentication with user/password
I tried this kind of instruction without succes
httr::GET("http://vpsxxxx.ovh.net/root/project/raw/9f8a404b5b33c216d366d80b7d48e34577598069/R/script.R",
authenticate("user", "password",type="basic"))
any idea?
Regards
edit : I found this way... but I need to download all the project...
bundle <- tempfile()
git2r::clone("http://vpsxxx.ovh.net/root/projet.git",
bundle, credentials=git2r::cred_user_pass("user", "password"))
source(file.path(bundle,"R","script.R"))
you can use gitlab API to get a file from a repository. gitlabr can help you do that. Current version 0.9 is compatible with apiV3 and V4.
This should work (it work on my end on a private gitlab with apiv3)
library(gitlabr)
my_gitlab <- gl_connection("https://private-gitlab.com",
login = "username",
password = "password",
api_version = "v4") # by default. put v3 here if needed
my_file <- my_gitlab(gl_get_file, project = "project_name", file_path = "path/to/file")
This will get you a character version of your file. You can also get back a raw version to deal with it in another way.
raw <- gl_get_file(project = "project_name",
file_path = "file/to/path",
gitlab_con = my_gitlab,
to_char = F)
temp_file <- tempfile()
writeBin(raw, temp_file)
You can now source the code
source(temp_file)
It is one solution among others. I did not manage to source the file without using the API.
Know that :
* You can use an Access Token instead of username and password
* You can use the gitlabr several ways. It is documented in the vignette. I used 2 differents ways here.
* Version 1.0 will not be compatible with v3 api. But I think you use v4.
Feel free to get back to me so that I update this post if need a clearer answer.
Related
I'm trying to pull down all files in a given bucket, except those in a specific directory, using R.
In the aws cli, I can use...
aws s3 sync s3://my_bucket/my_prefix ./my_destination --exclude="*bad_directory*"
In aws.s3::s3sync(), I'd like to do something like...
aws.s3::s3sync(path='./my_destination', bucket='my_bucket', prefix='my_prefix', direction='download', exclude='*bad_directory*')
...but exclude is not a supported argument.
Is this possible using aws.s3 (or paws for that matter)?
Please don't recommend using aws cli - there are reasons that approach doesn't make sense for my purpose.
Thank you!!
Here's what I came up with to solve this...
library(paws)
library(aws.s3)
s3 <- paws::s3()
contents <- s3$list_objects(Bucket='my_bucket',Prefix='my_prefix/')$Contents
keys <- unlist(sapply(contents,FUN=function(x){
if(!grepl('/bad_directory/',x$Key,fixed=TRUE)){
x$Key
}
}))
for(i in keys){
dir.create(dirname(i),showWarnings=FALSE,recursive=TRUE)
aws.s3::save_object(
object = i,
bucket='my_bucket',
file = i
)
}
Still open to more efficient implementations - thanks!
When using textract from the paws package in R the start_document_analysis call requires the path to a S3Object in DocumentLocation.
textract$start_document_analysis(
DocumentLocation = list(
S3Object = list(Bucket = bucket, Name = file)
)
)
Is it possible to use DocumentLocation without a S3Object? I would prefer to just provide the path to a local PDF.
The start_document_analysis api only supports providing an s3 object as input, and not a base64 encoded string like the analyze_document api (see also CLI docs on https://docs.aws.amazon.com/cli/latest/reference/textract/start-document-analysis.html)
So unfortunately you have to use S3 as a place to (temporarily) store your data. Of course you can write your own logic to do that :). Great tutorial on that can be found at
https://www.gormanalysis.com/blog/connecting-to-aws-s3-with-r/
Since you have already set up credentials etc. you can skip a lot of the steps and start at step 3 for example.
I am trying to read one RData file from my private repository "data" in R
library(repmis)
source_data("https://github.com/**********.Rdata?raw=true")
This is my output
Error in download_data_intern(url = url, sha1 = sha1, temp_file = temp_file) :
Not Found (HTTP 404).
Other way
script <-
GET(
url = "https://api.github.com/repos/***/data/contents/01-wrangle-data-covid-ssa-mx-county.R",
authenticate(Sys.getenv("GITHUB_PAT"), ""), # Instead of PAT, could use password
accept("application/vnd.github.v3.raw")
) %>%
content(as = "text")
# Evaluate and parse to global environment
eval(parse(text = script))
Anyone knows how can I read this data from my private repo in R?
I could solve this.
Generate on GitHub your personal token
1.1 Go to GitHub
2.1 In the right corner go to "Settings"
2.2 Then in the left part go to "Developer setting"
2.3 Select the option "Personal access tokens"
2.4 Select the option "Generate new token"
2.5 Copy your personal token
On your home directory follow the next steps
2.1 Create the file .Renviron
macbook#user:~$ touch .Reviron
On this file write your personal token like this:
macbook#user:~$ nano .Reviron
GITHUB_PAT=YOUR PERSONAL TOKEN
Now on R, you can check if your personal token has been saved with this:
Sys.getenv("GITHUB_PAT")
also you can edit your token on R with this:
usethis::edit_r_environ()
Don´t forget to restart R to save your changes.
3. Finally on R these are the line codes that will load your data from private repos
library(httr)
req <- content(GET(
"https://api.github.com/repos/you_group/your_repository/contents/your_path_to your_doc/df_test.Rdata",
add_headers(Authorization = "token YOUR_TOKEN")
), as = "parsed")
tmp <- tempfile()
r1 <- GET(req$download_url, write_disk(tmp))
load(tmp)
I'm currently trying to access sharepoint folders in R. I read multiple articles addressing that issue but all the proposed solutions don't seem to work in my case.
I first tried to upload a single .txt file using the httr package, as follows:
URL <- "<domain>/<file>/<subfile>/document.txt"
r <- httr::GET(URL, httr::authenticate("username","password",type="any"))
I get the following error:
Error in curl::curl_fetch_memory(url, handle = handle) :
URL using bad/illegal format or missing URL
I then tried another package that use a similar syntax (RCurl):
URL <- "<domain>/<file>/<subfile>/document.txt"
r <- getURL(URL, userpwd = "username:password")
I get the following error:
Error in function (type, msg, asError = TRUE) :
I tried many other ways of linking R to sharepoint, but these two seemed the most straightforward. (also, my URL doesn't seem to be the problem since it works when I run it in my web browser).
Ultimately, I want to be able to upload a whole sharepoint folder to R (not only a single document). Something that would really help is to set my sharepoint folder as my working directory and use the base::list.files() function to list files in my folder, but I doubt thats possible.
Does anyone have a clue how I can do that?
I created an R library called sharepointr for doing just that.
What I basically did was:
Create App Registration
Add permissions
Get credentials
Make REST calls
The Readme.md for the repository has a full description, and here is an example:
# Install
install.packages("devtools")
devtools::install_github("esbeneickhardt/sharepointr")
# Parameters
client_id <- "insert_from_first_step"
client_secret <- "insert_from_first_step"
tenant_id <- "insert_from_fourth_step"
resource_id <- "insert_from_fourth_step"
site_domain <- "yourorganisation.sharepoint.com"
sharepoint_url <- "https://yourorganisation.sharepoint.com/sites/MyTestSite"
# Get Token
sharepoint_token <- get_sharepoint_token(client_id, client_secret, tenant_id, resource_id, site_domain)
# Get digest value
sharepoint_digest_value <- get_sharepoint_digest_value(sharepoint_token, sharepoint_url)
# List folders
sharepoint_path <- "Shared Documents/test"
get_sharepoint_folder_names(sharepoint_token, sharepoint_url, sharepoint_digest_value, sharepoint_path)
Hi I wanna using R to check dropbox and get the latest file. Now using library(rdrop2)
library(rdrop2)
# drop_auth() # username/password for 1st time
drop.file = drop_dir('daily_export')
which1 = grepl("^daily_export/hat.*.gz$", drop.file$path) # files begin with hat and end with .gz
drop.file = drop.file[which1, ]
drop.file = drop.file[ drop.file==max(drop.file$path), 'path']# max file name indicates latest
drop_get(drop.file$path) #download to current folder
It works but when I restart R, drop_dir needs my authentication - I need click agree in the browser.
I want to automate and schedule the R code. So I'm wondering if there's a way to avoid authentication every time. - Ways using other tools are welcomed too. Thanks!