Does anyone have a working R script to access the API? I see there is a webscraping package, but I'd prefer to use the official API if possible.
Everything I've got and research I've done to authenticate via oauth 1 or 2 seems broken using the httr package.
For oauth 2:
library(httr)
app = '<OAuth 2.0 Client ID>'
key <- '<Client (Consumer) Key>'
secret <- '<Client (Consumer) Secret>'
accessTokenURL <- 'https://api.fitbit.com/oauth2/token'
authorizeURL <- 'https://www.fitbit.com/oauth2/authorize'
fbr <- oauth_app(key,app,NULL)
fitbit <- oauth_endpoint(NULL, authorizeURL,accessTokenURL)
token <- oauth2.0_token(fitbit,fbr,scope='profile',as_header=TRUE, cache=FALSE)
When I check token$credentials I get the error message:
$errors
$errors[[1]]
$errors[[1]]$errorType
[1] "oauth"
$errors[[1]]$fieldName
[1] "n/a"
$errors[[1]]$message
[1] "No Authorization header provided in the request. Each call to Fitbit API should be OAuth signed"
$success
[1] FALSE
I have tried all the misc settings in fitbit setting up my app, both as a server and a client, and I have my callback URL correctly set up as http://localhost:1410
Any ideas?
Thanks,
Isaac
ps: I've crossposted on fitbit's forums: https://twitter.com/IsaacWyatt/status/637768649290350592 and also tweeted at Hadley Wickham for help. Retweets welcome!
Current session info:
R version 3.1.2 (2014-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] httr_1.0.0
loaded via a namespace (and not attached):
[1] curl_0.9.3 httpuv_1.3.2 jsonlite_0.9.14 R6_2.0.1 Rcpp_0.11.4 stringr_0.6.2
[7] tools_3.1.2
Fitbit's API access via OAuth1.0a is deprecated from 2016-03-14, so here is working script using OAuth2.0. There was an issue with the httr package not being able to request a token using HTTP Basic authentication (required by Fitbit), but this is fixed by a new use_basic_auth argument to oauth2.0_token().
Fitbit Application Settings
OAuth 2.0 Application Type: Personal
Callback URL: http://localhost:1410 (this is the default expected by httr)
Example
# Until the version *following* httr 1.0.0 is available on CRAN, get the
# development version by uncommenting the following 2 lines
# library(devtools)
# install_github("hadley/httr")
library(httr)
# 1. Set up credentials
fitbit_endpoint <- oauth_endpoint(
request = "https://api.fitbit.com/oauth2/token",
authorize = "https://www.fitbit.com/oauth2/authorize",
access = "https://api.fitbit.com/oauth2/token")
myapp <- oauth_app(
appname = "data_access",
key = "Your OAuth 2.0 Client ID",
secret = "Your Client (Consumer) Secret")
# 2. Get OAuth token
scope <- c("sleep","activity") # See dev.fitbit.com/docs/oauth2/#scope
fitbit_token <- oauth2.0_token(fitbit_endpoint, myapp,
scope = scope, use_basic_auth = TRUE)
# 3. Make API requests
resp <- GET(url = "https://api.fitbit.com/1/user/-/sleep/date/2015-10-10.json",
config(token = fitbit_token))
content(resp)
So I asked for help on fitbit's forums as well, and user grahamrp posted the following using oauth 1.0:
library(httr)
token_url = 'https://api.fitbit.com/oauth/request_token'
access_url = 'https://api.fitbit.com/oauth/access_token'
auth_url = 'https://www.fitbit.com/oauth/authorize'
key <- '<Client (Consumer) Key>'
secret <- '<Client (Consumer) Secret>'
myapp = oauth_app('data_access', key, secret)
fb_ep = oauth_endpoint(token_url, auth_url, access_url)
token = oauth1.0_token(fb_ep, myapp)
gtoken = config(token = token)
resp = GET('https://api.fitbit.com/1/user/-/sleep/date/today.json', gtoken)
content(resp)
This works for me as of 8/31/2015
Link: https://community.fitbit.com/t5/Web-API/Working-R-script-Using-API-R-Mac-OS-X/m-p/931586/highlight/false#M3002
Thanks all for your attention to this!
Best,
Isaac
Related
I notice a few people have encountered issues with rtweet authorization. I followed this vignette which assumes people are familiar with authentication protocols. Newbies such as myself, struggle with this.
vignette("auth", package = "rtweet")`
I stored all keys and tokens in my .Renviron file. This is the code I use to access keys and tokens:
require(rtweet)
require(httpuv)
api_key <- Sys.getenv("TWITTER")
api_secret_key <- Sys.getenv("TWITTER_SECRET")
access_token <- Sys.getenv("ACCESS_TOKEN")
access_token_secret <- Sys.getenv("ACCESS_SECRET")
I then create a token using this code:
token <- create_token(
app = "my_app",
consumer_key = api_key,
consumer_secret = api_secret_key,
access_token = access_token,
access_secret = access_token_secret)
Next, I check if a token has been created:
get_token()
I get this response:
<Token>
<oauth_endpoint>
request: https://api.twitter.com/oauth/request_token
authorize: https://api.twitter.com/oauth/authenticate
access: https://api.twitter.com/oauth/access_token
<oauth_app> my_app
key: XXXXXXXXXXXXXXXXXXXX
secret: <hidden>
<credentials> oauth_token, oauth_token_secret
---
I restarted R and run a query as follows:
tweet_data <- search_tweets("#plasticrecycling",
n = 2000,
include_rts = F,
lang = "en")
And end up with this error:
Warning: 32 - Could not authenticate you.
Warning message:
Could not authenticate you.
Can someone please explain why this is happening? A similar issue was reported here https://github.com/ropensci/rtweet/issues/439 but it was closed without a clear answer. I am not sure if create_token() is a once-off thing or must be initiated each time I use R. I notice this added to my .Renviron file:
TWITTER_PAT=/Users/bob/.rtweet_token.rds
Thanks in advance.
Loading the latest development version of rtweet solves the problem I was encountering:
remotes::install_github("rOpenSci/rtweet")
packageVersion("rtweet")
[1] ‘0.7.0.9026’
This version makes it easier to authenticate things.
auth <- rtweet_app(bearer_token = Sys.getenv("BEARER_TOKEN"))
auth_save(auth, "my_app")
auth_list()
[1] "my_app"
auth_as(auth = "my_app")
tweets <- search_tweets("#plasticrecycling",
include_rts = F,
n = 1000,
lang = "en")
One only has to run the rtweet_app() once and then save the output. Every time one restarts R and loads a Twitter script, one simply has to execute auth_as(auth = "your_app")
I was confused by OAuth1.0 versus OAuth2.0 on the Twitter API page. It also did not help reading the wrong online manual pages for a different version of rtweet!
Kudos to llrs for helping me resolve my issue.
I am trying to fetch tweets using searchTwitter() and/or userTimeline()
I want to fetch maximum number tweets allowed to fetch by twitterR API (I believe that limit is around 3000.)
But in result I'm only getting very few posts (like 83 or 146). I'm sure there are more number of posts, when I check the Timeline of that user (via browser or app) I see there are more than 3000 posts.
Below is the message I get.
r_stats <- searchTwitter("#ChangeToMeIs", n=2000)
Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit, :
2000 tweets were requested but the API can only return 83
Is there anything I am missing on?
PS: I've checked all related question before posting. Before marking duplicate, please help me with the solution.
Actually, you are using is the Twitter Search API and it only returns a sample of results, not the comprehensive search.
What you need is Twitter Streaming API.
Please note that the Twitter search API does not return an exhaustive
list of tweets that match your search criteria, as Twitter only makes
available a sample of recent tweets. For a more comprehensive search,
you will need to use the Twitter streaming API, creating a database of
results and regularly updating them, or use an online service that can
do this for you.
Source: https://colinpriest.com/2015/07/04/tutorial-using-r-and-twitter-to-analyse-consumer-sentiment/
I install the library twitteR from git hub and this is quite important that the version is from git not CRAN
Than set up
setup_twitter_oauth("xxxxxxx", "xxxxx")
and than you can use the commends as
To get twitts from users timeline
ut <- userTimeline('xxxx', n=2000)
ut <- twListToDF(ut)
or to search for specific hastags
tweets<-twListToDF(searchTwitter("#f1", n=5000))
It works perfect for me
R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=Swedish_Sweden.1252 LC_CTYPE=Swedish_Sweden.1252 LC_MONETARY=Swedish_Sweden.1252 LC_NUMERIC=C LC_TIME=Swedish_Sweden.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] twitteR_1.1.9
loaded via a namespace (and not attached):
[1] bit_1.1-12 httr_1.1.0 rjson_0.2.15 plyr_1.8.3 R6_2.1.2 rsconnect_0.4.1.11 DBI_0.3.1 tools_3.2.2
[9] whisker_0.3-2 yaml_2.1.13 Rcpp_0.12.4 bit64_0.9-5 rCharts_0.4.5 RJSONIO_1.3-0 grid_3.2.2 lattice_0.20-33
Since twitteR is going to be deprecated, what you need to do is install rtweet.
Here is the code:
# Install and load the 'rtweet' package
install.packages("rtweet")
library(rtweet)
# whatever name you assigned to your created app
appname <- "tweet-search-app"
# api key (example below is not a real key)
key <- "9GmBeouvgfdljlBLryeIeqCHEt"
# api secret (example below is not a real key)
secret <- "ugdfdgdgrxOzjhlkhlxgdxllhoiofdtrrdytszghcv"
# create token named "twitter_token"
twitter_token <- create_token(
app = appname,
consumer_key = key,
consumer_secret = secret)
# Retrieve tweets for a particular hashtag
r_stats <- search_tweets("#ChangeToMeIs", n = 2000, token = twitter_token)
I am retrieving online XML data using the XML R packages. My issue is that the UTF-8 encoding is lost during the call to xmlToList : for instance, 'é' are replaced by 'é'. This happens during the XML parsing.
Here is a code snippet, with an example of encoding lost and another where encoding is kept (depending of the data source) :
library(XML)
library(RCurl)
url = "http://www.bdm.insee.fr/series/sdmx/data/DEFAILLANCES-ENT-FR-ACT/M.AZ+BE.BRUT+CVS-CJO?lastNObservations=2"
res <- getURL(url)
xmlToList(res)
# encoding lost
url2 = "http://www.bdm.insee.fr/series/sdmx/conceptscheme/"
res2 <- getURL(url2)
xmlToList(res2)
# encoding kept
Why the behaviour about encoding is different ? I tried to set .encoding = "UTF-8" in getURL, and to enc2utf8(res) but that makes no change.
Any help is welcome !
Thanks,
Jérémy
R version 3.2.1 (2015-06-18)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 7 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252
[3] LC_MONETARY=French_France.1252 LC_NUMERIC=C
[5] LC_TIME=French_France.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RCurl_1.95-4.7 bitops_1.0-6 XML_3.98-1.3
loaded via a namespace (and not attached):
[1] tools_3.2.1
You are trying to read SDMX documents in R. I would suggest to use the rsdmx package that makes easier the reading of SDMX documents. The package is available on CRAN, you can also access the latest version on Github.
rsdmx allows you to read SDMX documents by file or url, e.g.
require(rsdmx)
sdmx = readSDMX("http://www.bdm.insee.fr/series/sdmx/data/DEFAILLANCES-ENT-FR-ACT/M.AZ+BE.BRUT+CVS-CJO?lastNObservations=2")
as.data.frame(sdmx)
Another approach is to use the web-service interface to embedded data providers, and INSEE is one of them. Try:
sdmx <- readSDMX(providerId = "INSEE", resource = "data",
flowRef = "DEFAILLANCES-ENT-FR-ACT",
key = "M.AZ+BE.BRUT+CVS-CJO", key.mode = "SDMX",
start = 2010, end = 2015)
as.data.frame(sdmx)
AFAIK the package also contains issues to the character encoding, but i'm currently investigating a solution to make available soon in the package. Calling getURL(file, .encoding="UTF-8") properly retrieves data, but encoding is lost calling xml functions.
Note: I also see you use a parameter lastNObservations. For the moment the web-service interface does not support extra parameters, but it may be made available quite easily if you need it.
twitteR R Package OAuth issue: Error in registerTwitterOAuth(cred) :
oauth argument must be of class OAuth
This is the code I am using in R to load in the Authentication once I have authenticated the first time.
I keep getting the error:
Error in registerTwitterOAuth(cred) : oauth argument must be of class OAuth
from the following code:
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="/xxx/xxx/xxx/cacert.pem")
# to get your consumerKey and consumerSecret see the twitteR documentation for instructions
cred <- OAuthFactory$new(consumerKey='xxxxxxxxx',
consumerSecret='xxxxxxxxx',
requestURL= NOTE UNABLE TO POST LINKS IN THE POST
accessURL = NOTE UNABLE TO POST LINKS IN THE POST
authURL= NOTE UNABLE TO POST LINKS IN THE POST)
# necessary step for Windows / Mac
cred$handshake(cainfo="/xxx/xxx/xxx/cacert.pem")
# save for later use for Windows / Mac
save(cred, file="/xxx/xxx/xxx/twitter authentication.Rdata")
cred <- load("/xxx/xxx/xx/twitter authentication.Rdata")
registerTwitterOAuth(cred)
Just do:
load("/xxx/xxx/xx/twitter authentication.Rdata")
Instead of:
cred <- load("/xxx/xxx/xx/twitter authentication.Rdata")
I'd like to be able to access the bitly OAuth2 api from R and was wondering whether there were any example implementations, or better still, R libraries/wrappers for the bitly API around?
The twitteR library span off an R OAuth library, ROAuth, but this presumably doesn't support OAuth2? Or will OAuth2 accept the OAuth1 overtures?
Here is a way using httr - it's in the veins of the packages' examples on GitHub:
require(jsonlite)
require(httr)
# 1. Find OAuth settings for bit.ly:
# http://dev.bitly.com/authentication.html
bitly <- oauth_endpoint(
authorize = "https://bitly.com/oauth/authorize",
access = "https://api-ssl.bitly.com/oauth/access_token")
# 2. Register an application at http://dev.bitly.com/my_apps.html
# Insert your values below - if secret is omitted, it will look it up in
# the BITLY_CONSUMER_SECRET environmental variable.
myapp <- oauth_app("bitly",
key = ".............................", # Client ID
secret = "............................") # Client Secret
bitly_token <- oauth2.0_token(bitly, myapp, cache = FALSE)
# 4. Use API
req <- GET("https://api-ssl.bit.ly/v3/user/info", query = list(access_token = bitly_token$credentials$access_token))
stop_for_status(req)
content(req)$data$profile_url
# [1] "http://bitly.com/u/lukeanker"
I have written three fxns so far within a package that hits other APIs here: https://github.com/ropensci/raltmet/tree/master/R
The three fxns gets clickc based on users, expand URLs and shorten URLs
Install via:
install.packages("devtools")
require(devtools)
install_github("raltmet", "ropensci")
require(raltmet)
It is possible to do it with the yet to be updated on CRAN version of ROAUth ( ROAuth 0.92). I have a working copy available here. Once you install ROAuth from this source, download a copy of RMendeley to test out how R works with oauth.
library(devtools)
install_github("rmendeley", "ropensci")