All I'm trying to do is read all the repos and issues in my organizations private repos. I can from my Windows 7 cmd.exe execute
curl -u "user:pass" https://api.github.com/orgs/:org/repos
and I get back all of my repositories. I can pipe this to a file:
curl -u "user:pass" https://api.github.com/orgs/:org/repos > "C:\Users\Location\file.txt"
and this saves the JSON output. I can replicate this in R but in what seems like a terrible way.
fullRepos = system('curl -s -u "user:pass" -G https://api.github.com/orgs/:org/repos',
intern=T,show.output.on.console = F)
This captures the output (intern = T) and the -s gets rid of the progress bars so I can collapse the lines and turn it into a data frame. This gets back all the repositories, public and private.
I tried using RCurl to do the same thing but the code below only provides the public repositories. The httpheader is because otherwise it the API rejects my call.
RCurl::getURL(url="https://api.github.com/orgs/:org/repos",userpwd ="user:pass",
httpheader = c('User-Agent' = "A user agent"))
I also tried httr and it also only provides the public repositories.
httr::GET(url="https://api.github.com/orgs/:org/repos",userpwd="user:pass")
What am I doing wrong with RCurl and httr? I'd rather have a workflow that doesn't make a system command and then paste the lines together.
We can use the authenticate() helper function in httr to build the authentication header for us w/o having to manually create it. Also, verbose() can be used to debug HTTP issues:
httr::GET(url="https://api.github.com/orgs/:org/repos",
httr::authenticate("user", "pass"),
httr::verbose())
Related
I have the following bash script to download & decompress a JSON file:
#!/bin/sh -ex
# Ensure data directory (or a link) exists.
test -e results || mkdir results
# Download and decompress data.
curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
| xz -d -T8 > results/gisaid.json
Ideally I would like to have an R function to download & decompress this file in a given directory, with the environment variables above $GISAID_USERNAME, $GISAID_PASSWORD & $GISAID_FEED passed as arguments. Would anyone know how to accomplish this, e.g. using package curl or RCurl? (It would also be OK not to decompress it and leave it as .json.xz, as I would be reading the file later using
library(jsonlite)
GISAID_json <- jsonlite::stream_in(gzfile(".//data//GISAID_json//provision.json.xz"))
Something like this should work:
library(curl)
library(glue)
custom_curl <- function(user, pass, feed, dest) {
custom_handle <- curl::new_handle()
curl::handle_setopt(
custom_handle,
username = user,
password = pass
)
url <- glue::glue(
"https://www.epicov.org/epi3/3p/{feed}/export/provision.json.xz"
)
curl::curl_download(url, dest, handle = custom_handle)
}
custom_curl('my_user', 'xxxxxx', 'feed1', 'dest/filename.json.xz')
As I can't test in the real files and url, I'm not sure if little tinkering in the function is needed, but at least is a starter point for you.
Does executing your terminal commands in R using the system function already help you?
Put your terminal call into system() and it should execute and create your file. Afterwards read in the file. Of course you would have to replace the $GISAID_USERNAME, $GISAID_PASSWORD with your actual information. If the login information or the url should be flexible, you can put together a string beforehand, since system() expects a string with the command to execute.
system("curl -u $GISAID_USERNAME:$GISAID_PASSWORD --retry 4 \
https://www.epicov.org/epi3/3p/$GISAID_FEED/export/provision.json.xz \
| xz -d -T8 > results/gisaid.json")
Afterwards just read in the (hopefully) created file.
Couldn't test with your setup, but for me e.g. this small example successfully creates a file:
system("curl https://raw.githubusercontent.com/SteffenMoritz/imputeTS/master/pkgdown/favicon/favicon.ico > /Users/Steve/Downloads/x.ico")
I have the following curl command that, when run from command line, works perfectly:
curl -X POST -u "myusername|myemail#domain.com:myPassword"
-H "Content-Type: multipart/form-data"
--form file=#MyFileForUploading.csv
https://mysite-data.herokuapp.com/api/mymarket/setups/uploads
[Apologies: this is not a working example as I cannot provide the real url and credentials. I am hoping you can help me with the translation from curl to httr without running the example yourselves.]
Here's my attempt to translate the above to the language of R's httr, which did NOT work:
library(httr)
POST("https://mysite-data.herokuapp.com/api/mymarket/setups/uploads",
config = authenticate("myusername|myemail#domain.com", "myPassword"),
body = upload_file("MyFileForUploading.csv", type = "text/csv"),
encode = "multipart")
The curl command serves to upload a csv file being used as setup for a web-based trading interface. Setup includes things like trader initial holdings of objects, trader permissions (to buy and sell), etc. All this is simply stored as a csv file (columns = setup parameters; rows = traders).
Can anyone see an obvious mis-translation? I am very ignorant about both curl and httr. My translation is based on learning from examples and I wouldn't be surprised if there's an obvious failure, for example, with the content-type part of the command.
Thanks!
You're really close. This works with environment values setup in "~/Renviron":
library("httr")
post_url <- Sys.getenv("POST_URL")
username <- Sys.getenv("USERNAME")
password <- Sys.getenv("PASSWORD")
csv_file <- Sys.getenv("CSV_FILE")
POST(
url = post_url,
config = authenticate(username, password),
body = list(file = upload_file(csv_file)),
encode = "multipart",
verbose()
) -> response
The key is the file = as you used in your CURL command.
I'm trying to use download.file to get some webpages including embedded images, etc. I think using wget it's the equivalent of the -p -k options, but I can't see how to do this...
if I do:
download.file("http://guardian.co.uk","test.html")
That obviously works, but I get this error:
Warning messages:
1: running command 'wget -p -k "http://guardian.co.uk" -O "test.html"' had status 1
2: In download.file("http://guardian.co.uk", "test.html", method = "wget", :
download had nonzero exit status
When I run:
download.file("http://guardian.co.uk","test.html", method = "wget", extra = "-p -k") #no recursion (-r), but get pre-requisites, and (-k) convert for local viewing
I've done Sys.which("wget") & the path is set (and I'm not trying to access https which I think can cause issues).
Once I've done this I actually want to put it into a loop where I download a set of urls (& their embedded content) to create a single html output...
Easy solution, just use system to call wget directly:
system("wget http://guardian.co.uk -p -k")
I think the issue is that passing an output file ('test.html') means -O option specified, so you can't also invoke -r -k whereas calling wget directly means it saves the files separately.
I've been playing with httr and RCurl for a while and am struggling with specifying some certificate information.
I have a Curl CLI command that works just fine to log me in to a website that uses self-signed certs for authentication (as well as uname/pwd), it looks something like this...
curl -k -q -v --cert "../certs/foobar.pem" https://signon.foobar.com/certlogin -d "username=foo&password=bar" -H "X-Application: Curltest"
This logs me in to foobar.com and returns a session token.
The same command works if I use the corresponding .crt cert file and add a key using the --key option.
I'm trying to do the same thing with RCurl or httr and keep getting an error that indicates that the cert is wrong. Bearing in mind that I know the cert is OK (as the CLI command works) I assume this is because I'm specifying the cert incorrectly in RCurl.
So, quick question...
How would you reproduce the CURL command given above in RCurl or httr?
BTW this is on Ubuntu - so should not be related to the issues lots of folks are having with Mac OS X 10.9 +
Many thanks
Eluxoso
(first time poster - be nice....!)
Here is my (probably very inelegant & certainly non-functioning) code
RCurl:
Library(RCurl)
url <- "https://signon.foobar.com/certlogin"
uname <- "foo"
pswd <-"bar"
headers <- list("X-Application" = "Curltest")
opts=curlOptions(verbose=TRUE,
httpheader=headers,
cainfo="../certs/foobar.pem",
ssl.verifypeer=FALSE)
postForm(url,
"username" = uname, "password"=pswd,
.opts=opts,
style='POST'
)
httr:
library(httr)
cafile="../certs/foobar.pem"
resp<-POST("https://signon.foobar.com/certlogin",
body = list(username="foo", password = "bar"),
add_headers("X-Application" = "Curltest" ),
verbose(),
encode = "form",
config(cainfo = cafile, ssl.verifypeer=FALSE))
content(resp,"text")
Having used a full pathname for the cert I still get the same server response, despite the CURL CLI command working.
In the HTTP requests I noticed this:
SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
Is that relevant/helpful to someone?..
I had this issue, and it was resolved by installing libcurl4-openssl-dev and updating the Rcurl package to the most recent version (1.95)
$ sudo apt-get install libcurl4-openssl-dev
$ R
> install.packages("RCurl")
I set up a local OpenCPU single-user server using RStudio. I also create my own R package(Package name: test) which includes only a simple test.R file. The source code is
f1 <- function(x, y) {x+y}
I started the OpenCPU server by typing library(opencpu) in RStudio's console. I got the following print.
Initiating OpenCPU server...
OpenCPU started.
[httpuv] http://localhost:6067/ocpu
OpenCPU single-user server ready.
I was able to run the script by typing curl http://localhost:6067/ocpu/library/test/R/f1 -d "x=33&y=3".
But when I tried to display the R script(test.R) by typing curl http://localhost:6067/ocpu/library/test/R/test.R, it printed
object 'test.R' not found
In call:
get(reqobject, paste("package", reqpackage, sep = ":"), inherits = FALSE)
In addition, It's failed when I ran the test.R script by typing curl http://localhost:6067/ocpu/library/test/R/test.R -X POST -d "x=3&y=4". Could I run the script like that?
Could anyone help with this? Thanks.
When you install the R package, scripts under /R are turned into functions/objects. To read the source of the function, just do on of these:
curl http://localhost:6067/ocpu/library/test/R/f1/print
curl http://localhost:6067/ocpu/library/test/R/f1/ascii