I am trying to make a cURL API call with R and I am unable to retrieve data. Or more specifically I am unable to figure out how to translate a multi-line curl call into an R command.
I am trying to get data from Twitch, the Twitch Developers API page offers the following curl code. Though I am unsure about the syntax of the call.
curl -H 'Accept: application/vnd.twitchtv.v5+json' \
-H 'Client-ID: uo6dggojyb8d6soh92zknwmi5ej1q2' \
-X GET 'https://api.twitch.tv/kraken/games/top'
I have attempted variations of:
library(curl)
library(httr)
library(jsonlite)
df <- GET('https://api.twitch.tv/kraken/games/top', add_headers('Accept: application/vnd.twitchtv.v5+json', 'Client-ID: uo6dggojyb8d6soh92zknwmi5ej1q2'))
fromJSON(df)
df <- curl_download('https://api.twitch.tv/kraken/games/top', destfile = 'C:\\....\\curldta.csv')
fromJSON(df)
Thanks for any help in advance.
I wrote a package that is a wrapper of twitch API for R language (you can install packages from github it with devtools package). The data frame you're trying to get can be obtained with
library(rTwitchAPI)
twitch_auth("YOUR_CLIENT_ID")
df = get_top_games()$data
Related
I'm trying to understand internet connections in R little bit better and while there is some information scattered around, I find it hard to follow. Hopefully this question can bring information together. My problem is following. I'm working from my office computer and there is firewall in place. I have tested out how to scrape webpages in R and found out following.
This code works now, because I have deleted curl from my package list.
library(rvest)
lego_movie <- read_html("http://www.imdb.com/title/tt1490017/")
lego_movie %>%
html_node("strong span") %>%
html_text() %>%
as.numeric()
[1] 7.8
But I'm trying to follow https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/ this example which needs V8 package (which needs curl). When I install curl package following takes place:
lego_movie <- read_html("http://www.imdb.com/title/tt1490017/")
Error in open.connection(x, "rb") : Empty reply from server
So there is proxy problem but how do I go around it with curl installed as many packages are using it? If I again remove curl I can continue with rvest only scraping. How curl can block my normal rvest scraping even if it's not using curl (httr seems to be using curl, but rvest still works)? I find this very confusing.
curl installation 3.1
rvest installation 0.3.2
R 3.4.3
I can access API data using "curl" on terminal but get error when i use R. Looking for some advice
Curl command:
curl -H "token:jQyrbzexeCEaWFIDBAwCWqbrkrVQTVhM" "https://www.ncdc.noaa.gov/cdo-web/api/v2/datasets"
R command:
books_key <- "&token=jQyrbzexeCEaWFIDBAwCWqbrkrVQTVhM"
url <- "https://www.ncdc.noaa.gov/cdo-web/api/v2/datasets"
req <- fromJSON(paste0(url, books_key))
Error in open.connection(con, "rb") : HTTP error 400.
Usually this error comes when there are spaces in URL (from other similar questions) but in my case there is no space in URL
Some info on using token from website https://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted
Not a R issue because following example of another website works
movie_key <- "&api-key=b75da00e12d54774a2d362adddcc9bef"
url <- "http://api.nytimes.com/svc/movies/v2/reviews/dvd-picks.json?order=by-date"
req <- fromJSON(paste0(url, movie_key))
You're supposed to pass the token as a header not a query param, e.g. with the crul pkg
cli <- crul::HttpClient$new(
url = "https://www.ncdc.noaa.gov",
headers = list(token = "yourtoken"))
cli$get(path = "cdo-web/api/v2/datasets")
A quick look at your Curl and your R call - in Curl your passing the token as a header variable and in the R call it appears to be a query variable.
Also the NYT call has a query param in the URI so & would be appropriate in the R call.
Hi I am new to writing R packages.
r package development imports not loaded advised me to use roxygen2.
I once called devtools::document() and the namespace was generated.
However when I load this simple package (or try it via opencpu) the functions are NOT available.
calling the code in native R seems to work
test2::hello()
[1] "Hello, world!"
Starting opencpu like:
1) start opencpu simple server via library(opencpu)
2) execute opencpu$restartwhich will show a port number
3) http://localhost:myPortNumber/ocpu/library/myPackage/info ---> this endpoint works
As mentioned in the comments this is not a "proper" way of calling a function. However opencpu defaults to myfunction/print if a function is called via HTTP as http://public.opencpu.org/ocpu/library/stats/R/quantile/printand even that does not work when I call the hello function.
This is a demonstration of how to call a more complex function:
curl http://localhost:myPortNumber/ocpu/library/stats/R/quantile/json -d '{"type":1,"x":[1,2,3,4,5,6,7,8,9,10],"probs":[0.05,0.25,0.75,0.95]}' -H "Content-Type: application/json"
You can simply test it via:
curl http://public.opencpu.org/ocpu/library/stats/R/quantile/json -d \
'{"type":1,"x":[1,2,3,4,5,6,7,8,9,10],"probs":[0.05,0.25,0.75,0.95]}' \
-H "Content-Type: application/json"
I did install it via sudo like:
sudo R CMD INSTALL test2_0.1.tgz
that means it should be available via in the /library/test2 endpoint.
Solution:
It still was the wrong API endpoint --> I was missing the R sub-directory
http://localhost:myPort/ocpu/library/myPackage/R/hello/
Example-code is here: https://github.com/geoHeil/rSimplePackageForOpenCpu
It still was the wrong API endpoint --> I was missing the R sub-directory
http://localhost:myPort/ocpu/library/myPackage/R/hello/
Hi I'm trying to write a script using R where I post data to a specific weblink and get the response back. I'm not really sure how to set curl headers for Content-type and Application.
this is what I have so far:
library("RCurl")
httpPOST("http://localhost", '{"title":"ello World!"}')
I get the following error:
Error in curlOptions(..., .opts = .opts) : unnamed curl option(s):
....
If use curl from command line the data does get posted.
Is there another library in R that does CURL and JSON posts better?
I just started playing around with the Twitter Streaming API and using the command line, redirect the raw JSON reponses to a file using the command below:
curl https://stream.twitter.com/1/statuses/sample.json -u USER:PASSWORD -o "somefile.txt"
Is it possible to stay completely within R and leverage RCurl to do the same thing? Instead of just saving the output to a file, I would like to parse each response that is returned. I have parsed twitter search results in the past, but I would like to do this as each response is received. Essentially, apply a function to each JSON response.
Thanks in advance.
EDIT: Here is the code that I have tried in R (I am on Windows, unfortunately). I need to include the reference to the .pem file to avoid the error. However, the code just "runs" and I can not seem to see what is returned. I have tried print, cat, etc.
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")
getURL("https://stream.twitter.com/1/statuses/sample.json",
userpwd="USER:PWD",
cainfo = "cacert.pem")
I was able to figure out the basics, hopefully this helps.
#==============================================================================
# Streaming twitter using RCURL
#==============================================================================
library(RCurl)
library(rjson)
# set the directory
setwd("C:\\")
#### redirects output to a file
WRITE_TO_FILE <- function(x) {
if (nchar(x) >0 ) {
write.table(x, file="Twitter Stream Capture.txt", append=T, row.names=F, col.names=F)
}
}
### windows users will need to get this certificate to authenticate
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")
### write the raw JSON data from the Twitter Firehouse to a text file
getURL("https://stream.twitter.com/1/statuses/sample.json",
userpwd=USER:PASSWORD,
cainfo = "cacert.pem",
write=WRITE_TO_FILE)
Try the twitter api package for R.
install.packages('twitteR')
library(twitteR)
I think this is what you need.