Issue scraping flairs from Reddit: Arguments imply differing number of rows? - r

Question

I'm trying to scrape a subreddit using a RedditExtractoR library that I've modified, but I keep running into this error. I modified the get_thread_content.R file in the source code so that it looks like this:
#' Get thread contents of Reddit URLs
#'
#' This function takes a collection of URLs and returns a list with 2 data frames:
#' 1. a data frame containing meta data describing each thread
#' 2. a data frame with comments found in all threads
#'
#' The URLs are being retained in both tables which would allow you to join them if needed
#'
#' #param urls A vector of strings pointing to a Reddit thread
#' #return A list with 2 data frames "threads" and "comments"
#' #export
get_thread_content <- function(urls){
data <- lapply(urls, parse_thread_url)
list(
threads = lapply(data, function(z) z[["thread"]]) |> rbind_list(),
comments = lapply(data, function(z) z[["comments"]]) |> remove_na() |> rbind_list()
)
}
# Build a data frame with thread attributes of interest
build_thread_content_df <- function(json, request_url) {
if (is.null(json$link_flair_text))
{thread_flair_info <- 'no flair'}
else {thread_flair_info <- json$link_flair_text}
if (is.null(json$author_flair_text))
{author_flair_info <- 'no flair'}
else {author_flair_info <- json$author_flair_text}
df <- data.frame(
url = strip_json(request_url),
author = json$author,
author_flair_text = author_flair_info,
date = timestamp_to_date(json$created_utc),
timestamp = json$created_utc,
title = json$title,
text = json$selftext,
thread_flair = thread_flair_info,
subreddit = json$subreddit,
score = json$score,
upvotes = json$ups,
downvotes = json$downs,
up_ratio = json$upvote_ratio,
total_awards_received = json$total_awards_received,
golds = json$gilded,
cross_posts = json$num_crossposts,
comments = json$num_comments,
stringsAsFactors = FALSE
)
return(df)
}
nullfix <- function(x){
if(is.null(x))
{x <- "no flair"}
else {x}
}
# Build a data frame with comments and their attributes.
build_comments_content_df <- function(json, request_url) {
data.frame(
url = strip_json(request_url),
author = extract_comments_attributes(json, "author"),
comment_author_flair = nullfix(extract_comments_attributes(json, "author_flair_text")),
date = extract_comments_attributes(json, "created_utc") |> timestamp_to_date(),
timestamp = extract_comments_attributes(json, "created_utc"),
score = extract_comments_attributes(json, "score"),
upvotes = extract_comments_attributes(json, "ups"),
downvotes = extract_comments_attributes(json, "downs"),
golds = extract_comments_attributes(json, "gilded"),
comment = extract_comments_attributes(json, "body"),
comment_id = build_comment_ids(json),
stringsAsFactors = FALSE
)
}
Everything works except the comment_author_flair part. I initially tried using similar code to what I used for the get_thread_content function, but that failed, so I separated out the function (see: nullfix) and tried to apply it to the basic code, but that still isn't working, as I get the same error.
As a reproducable example, after modifying that library:
thread <- get_thread_content("https://www.reddit.com/r/SSBM/comments/10ys20y/who_were_the_least_clutch_players/")
I would expect to get 69 flair values with this code, one for each comment, with NULL values replaced by "no flair" because of the nullfix function. Instead I get the error:
Error in data.frame(url = strip_json(request_url), author = extract_comments_attributes(json, :
arguments imply differing number of rows: 1, 69, 29