I have paid premium access to Twitter so can access historical data. I am trying to retrieve tweets on a combined search for the terms 'delivery' and 'accident', ie as if I go to a Twitter search and type 'delivery accident' or 'delivery+accident' in the 'latest' tab. My code for doing so is the following:
deliv_list <- searchTwitter("delivery+accident", n=500)
I am getting results which correspond to my search, but far fewer than I had expected. R returns the error message:
Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit, :
500 tweets were requested but the API can only return 248
I subsequently checked and I am able to return 500 records for a different single search, so it is not that I have reached a download limit. There are multiple tweets every day on this theme so I'm not sure why so few results n=248 are returned. Appreciate any help.
It seems twitteR cannot handle full archive premium accounts. search-tweets-python seems to be the way to go. Tantalizingly, this may be able to work inside R via R's reticulate package. See: https://dev.to/twitterdev/running-search-tweets-python-in-r-45eo
Related
I am trying to get the full timeline of twitter user through the academictwitteR package and the twitter academic track API (v2). However, I get an error 400. Unfortunately, the explanation here https://developer.twitter.com/en/support/twitter-api/error-troubleshooting
says "The request was invalid or cannot be otherwise served. An accompanying error message will explain further. Requests without authentication or with invalid query parameters are considered invalid and will yield this response.", but the error code does not contain any explanation. I do not know what I am doing wrong. I tried other R packages as well, same error. I have access to all other functions of the API, but cannot adjust the timeline to download either more than 3200 tweets at a time (as my access should allow me to do). I want to full timeline of this user, not just the newest 3200 tweets.
The errorcode looks like that:
Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token, :
something went wrong. Status code: 400
This is my request (I tried using the user ID, it does not change anything).
tmlne <- get_user_timeline("25390350",
start_tweets = "2009-03-19T00.00.00Z",
end_tweets = "2021-11-27T00.00.00Z",
bearer_token = get_bearer(),
data_path = "twitter/data/timelines/",
n = 100000,
bind_tweets = F,
file = "tmlnw")
I tried revoking and renewing the bearer token, I tested the token by downloading all tweets from a user (which is not what I am after, but just to see if my access works), that works, but the timelines do not. I can get 3200 with the get_timelines_v2 function of the TwitterV2 package, but I cannot circumvent the 3200 tweets limit and I do not know how to change the request to get older tweets than the most recent 3200, and I cannot get the academictwitteR package to use the timeline function (here I know how to make multiple requests).
What would help me is either an information regarding the Error 400 or how to adjust the get_timelines_v2 function to include older tweets.
I've tried two cryptocurrency API package for R, to name it:
coinmarketcapr package
riingo <- included in tidyquant
I really want to get updated historical data of cryptocurrency, my goal is to predict the thing with some timeseries analysis, but with these packages I kept getting error message, for coinmarketcapr it's understandable because apparently my API subscription plan doesn't cover the historical data, but for the riingo the message shows just like this...
> riingo_crypto_latest("btcusd", resample_frequency = "10min", base_currency = NULL)
Request failed [401]. Retrying in 1.6 seconds...
Request failed [401]. Retrying in 1.5 seconds...
Error: There was an error, but riingo isn't sure why. See Tiingo msg for details.
Tiingo msg) Invalid token.
Can somebody help me? Or maybe suggesting other source for taking cryptocurrency historical data? Thank you in advance for any answer!
P.S. I've already inserted the API key, so it's not the authentication problem.
If you trying to scrape information about new listings at crypto exchanges and some more useful info, you can be interested at using this API:
https://rapidapi.com/Diver44/api/new-cryptocurrencies-listings/
Example request in R:
library(httr)
url <- "https://new-cryptocurrencies-listings.p.rapidapi.com/new_listings"
response <- VERB("GET", url, add_headers(x_rapidapi-host = 'new-cryptocurrencies-listings.p.rapidapi.com', x_rapidapi-key = 'your-api-key', '), content_type("application/octet-stream"))
content(response, "text")
It includes an endpoint with New Listings from the biggest exchanges and a very useful endpoint with information about exchanges where you can buy specific coins and prices for this coin at that exchanges. You can use this information for trading. When currency starts to list at popular exchanges like Binance, KuCoin, Huobi, etc the price increases about 20-30%
I am working in R with the twitteR package, which is meant to get information from Twitter through their API. After getting authentificated, I can download information about any user with the getUser function. However, I am not able to do so with usernames which are only numbers (for example, 1234). With the line getUser("1234") I get the following error message:
Error in twInterfaceObj$doAPICall(paste("users", "show", sep = "/"),
params = params, : Not Found (HTTP 404).
Is there any way to get user information when the username is made completely of numbers? The function tries to search by ID instead of screenname when it finds only numbers.
Thanks in advance!
First of all, twitteR is deprecated in favour of rtweet, so you might want to look into that.
The specific user ID you've provided is a protected account, so unless your account follows it / has access to it, you will not be able to query it anyway.
Using rtweet and some random attempts to find a valid numerical user ID, I succeeded with this:
library(rtweet)
users <- c("989", "andypiper")
usr_df <- lookup_users(users)
usr_df
rtweet also has some useful coercion functions to force the use of screenname or ID (as_screenname and as_userid respectively)
I am using R and I need to retrieve the few most recent posts from a Twitter user (#ExpressNewsPK) using twitteR api. I have created an account and have an access token, etc. I have used the following command to extract the tweets:
setup_twitter_oauth(consumerkey,consumersecret,accesstoken,accesssecret)
express_news_tweets <- searchTwitter("#ExpressNewsPK", n = 10, lang = "en" )
However, the posts that are returned aren't the most recent ones from this user. Where have I made a mistake?
I think searchTwitter would search with the search string provided (here #ExpressNewsPK). So instead of giving tweets by #ExpressNewsPK it would give tweets which are directed to #ExpressNewsPK.
To get tweets from #ExpressNewsPK, you have a function named userTimeline which would give tweets from a particular user.
So after you are done with setup_twitter_oauth, you can try
userTimeline("ExpressNewsPK")
read more about it at ?userTimeline
When you use searchTwitter(), you call the Twitter Search API. Search API only returns a sample history of the tweets.
What you really need to do is to call Twitter Streaming API. Using it you'll be able to download tweets in near real time. You can read more about the Streaming API here: https://dev.twitter.com/streaming/overview
I recently tried Rfacebook package by pablobarbera, which works quite well. I am having this slight issue, for which I am sharing the code.
install.packages("Rfacebook") # from CRAN
library(devtools)
install_github("Rfacebook", "pablobarbera", subdir = "Rfacebook")
library(Rfacebook)
# token generated here: https://developers.facebook.com/tools/explorer
token <- "**********"
page <- getPage("DarazOnlineShopping", token, n = 1000)
getPage command works, but it only retrieves 14 records from the Facebook page I used in the command. In the example used by pablobarbera in the original post he retreived all the posts from "Humans of New York", but when I tried the same command, facebook asked me to reduce the number of posts, and I hardly managed to get 20 posts. This is the command used by Pablo bera:
page <- getPage("humansofnewyork", token, n = 5000)
I thought I was using temporary token access that why Facebook is not giving me the required data, but I completed the wholo Facebook Oauth Process, and the same result.
Can somebody look into this, and tell why this is happening.
The getPage() command looks fine to me, I manually counted 14 posts (including photos) on the main page. It could be that Daraz Online Shopping has multiple pages and that the page name you are using only returns results from the main page, when (I assume) you want results from all of them.
getPage() also accepts page IDs. You might want to collect a list of IDs associated with Daraz Online Shopping, loop through and call each of them and combine the outputs to get the results you need.
To find this out these IDs you could write a scraper (or manually search for them all) that views the page source and searches for the unique page ID. Searching for content="fb://page/?id= will highlight the location of the page ID in the source code.