I have just received an Academic Twitter Developer privileges and am attempting to scrape some tweets. I updated RStudio, regenerated a new bearer token once I got updated Academic Twitter access, and get_bearer() returns my new bearer token. However, I continue to get the following error:
Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token, :
something went wrong. Status code: 403
In addition: Warning messages:
1: Recommended to specify a data path in order to mitigate data loss when ingesting large amounts of data.
2: Tweets will not be stored as JSONs or as a .rds file and will only be available in local memory if assigned to an object.
Additionally, I have tried specifying a data path, but I think I am confused as to what these means? I think that's where my issue lies, but does the data path way mean like a specific file pathway on my computer?
Below is the code I was attempting to use.This code worked previously with my professor's bearer token that they used to just show the output:
`tweets <-
get_all_tweets(
query = "#BlackLivesMatter",
start_tweets = "2020-01-01T00:00:00Z",
end_tweets = "2020-01-05T00:00:00Z",
n = 100)`
Thanks in advance!
Status code 403 means forbidden. You may want to check the error codes reference page of Twitter API here.
Perhaps your bearer token is misspelled?
Related
I am trying to use pdftables package to extract data into csv.
install.packages("pdftables")
library(pdftables)
write.csv(head(iris), file = "test.csv", row.names = FALSE)
Open test.csv and print as PDF to "test.pdf"
convert_pdf("test.pdf", "test2.csv")
However, I am getting the following error:
Error in get_content(input_file, format, api_key) : Bad Request
(HTTP 400).
What's the fix here?
Did you get an API token?
To use the package the user first needs to sign up to the PDFTables API to get an API token (they offer a free package that allows up to 50 pages).
See: https://cran.r-project.org/web/packages/pdftables/README.html
To use the PDFTables R package, you need to the run the following command:
convert_pdf('test/index.pdf', output_file = NULL, format = "xlsx-single", message = TRUE, api_key = "insert_API_key")
Make sure you replace insert_API_key with your API key, and change the file path and/or format.
More info here: https://pdftables.com/blog/convert-pdf-to-excel-r
I have seen a few posts about this, but I have not been able to use any of the suggested code for the way I have my program set up. I am not very good at R but I need this information for a Political Science class. Could someone please help me figure out a way to avoid receiving the error message "Warning message: In twInterfaceObj$doAPICall(cmd, params, method, ...) : Rate limit encountered & retry limit reached - returning partial results." Thank you in advance! The code I am running is below:
# install and load packages
install.packages("twitteR")
install.packages("ROAuth")
install.packages("base64enc")
install.packages("plyr")
library("twitteR")
library("ROAuth")
library("base64enc")
library("plyr")
# Establish the connection (do this once)
cred$handshake(cainfo="cacert.pem")
#setup Twitter
setup_twitter_oauth('Consumer Key - XXXXXXXXXX', Consumer Secret -
'XXXXXXXXXX', API Key - 'XXXXXXXXXX', API Secret - 'XXXXXXX')
#store the user
user <- getUser('JackPosobiec')
#return a list of followers
Posobiecfollowers <- user$getFollowers(n=NULL)
Posobiecfollowers
#save followers in a data set
followers.df = ldply(followers, function(t) t$toDataFrame())
write.csv(followers.df, file = "JackPosobiecfollowersfile.csv")
I have successfully logged in google account with
gconnect(usr,password)
But I got request problem when I want to query the data
gt.us <- gtrends("USA", geo="US", start_date="2004-01-01", end_date="2004-01-30")
Error: Not enough search volume. Please change your search terms.
In addition: Warning message:
In request_GET(x, url, ...) : Gone (HTTP 410).
Could anyone help me out ?
Get gtrendsR 2.0.0:
devtools::install_github('PMassicotte/gtrendsR')
library(gtrendsR)
# do not need to log in anymore
# syntax change!
gt.us <- gtrends("USA", geo="US", time = "2004-01-01 2004-01-30")
See Philippe Massicotte answer to this issue.
I can not throttle-up my downloads by using the token issued to my app (on data.chicago.com portal, where I had to register)
Error 1:
token <- "___my_app_token__";
fdf <- read.socrata("h___s://data.cityofchicago.org/resource/7edu-s3u7.csv?$where=station_name=\"Foster Weather Station\"", token)
2016-10-06 10:39:53.685 getResponse:
Error in httr GET: 403 h___s://data.cityofchicago.org/resource/7edu-s3u7.csv?%24where=station_name%3D%22Foster%20Weather%20Station%22&app_token=%2524%2524app_token%3D___my_app_token_______
I have NO IDEA where did the first 'token' (2524 2524) come from, do you? Can somebody tell me? Maybe the author of the package is here?
Non-error:
fdf <- read.socrata("h___s://data.cityofchicago.org/resource/7edu-s3u7.csv?$where=station_name=\"Foster Weather Station\"")
WITHOUT A TOKEN (and not throttled-up) works perfectly well!
and this 'open source' h___s://github.com/Chicago/RSocrata/blob/master/R/RSocrata.R doesn't answer the question as well.
It looks like the syntax you're using to pass your app token is wrong. I'm no R expert, but I found this example in the documentation for the RSocrata library:
df <- read.socrata("http://soda.demo.socrata.com/resource/4334-bgaj.csv",
app_token = "__my_app_token__")
Try passing your app token as a named parameter instead of an indexed parameter, and see if that helps.
So I am trying to find tweets based on date range with this code:
tweets <- searchTwitter(c("Alzheimer"), n=500, lang="en",
since="2011-03-01", until="2011-03-02")
and I get the warning message
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit
= retryOnRateLimit, :
500 tweets were requested but the API can only return 0
BUT I don't get this warning message with the code
tweets <- searchTwitter(c("Alzheimer"), n=500, lang="en", since="2011-08-01")
I've read many posts previously about twitter not allowing a date range past a few days...is this still the case currently?? I'm new to coding so any help is greatly appreciated.
MrFlick is right in his comment, the twitter API only returns tweets for the past few days, as to why you are not getting the warning on the second command, my guess is because its working. I tried this command from my terminal and got back 500 tweets.