Twitter Data Sentiment Analysis - r

Question

I'm very novice so, apologies if my question is trivial.
I am trying to do sentiment analysis on some twitter data I downloaded but am having trouble. I am trying to follow this example:
which creates a bar plot that shows positive/negative sentiment counts. The code for the example is here**
original_books %>%
unnest_tokens(output = word,input = text) %>%
inner_join(get_sentiments("bing")) %>%
count(book, index, sentiment) %>%
pivot_wider(names_from = sentiment,
values_from = n) %>%
mutate(sent_score = positive - negative) %>%
ggplot() +
geom_col(aes(x = index, y = sent_score,
fill = book),
show.legend = F) +
facet_wrap(~book,scales = "free_x")
Here is the code I have so far for my own analysis:
#twitter scraping
ref <- search_tweets(
"#refugee", n = 18000, include_rts = FALSE,lang = "en"
)
data(stop_words)
new_stops <- tibble(word = c("https", "t.co", "1", "refugee", "#refugee", "amp", "refugees",
"day", "2022", "dont", "0", "2", "#refugees", "4", "2021") ,lexicon = "sabs")
full_stop <- stop_words %>%
bind_rows(new_stops) #bind_rows adds more rows (way to merge data)
Now I want to make a bar graph similar to the one above but I get an error because I don't have a column called "index." I tried making one but it didn't work. Here is the code I am trying to use:
ref %>%
unnest_tokens(word,text,token = "tweets") %>%
anti_join(full_stop) %>%
inner_join(get_sentiments("bing")) %>%
count(word, index, sentiment) %>%
pivot_wider(names_from = sentiment,
values_from = n) %>%
mutate(sent_score = positive - negative) %>%
ggplot() + #plot the overall sentiment (pos - neg) versus index,
geom_col(aes(x = index, y = sent_score), show.legend = F)
Here is an image of the error
Any suggestions are really appreciated! Thank you
Contents of ref
enter image description here
enter image description here

Answer 1

In the example, index just refers to a group of lines from the book, in order (i.e., 1, 2, 3...) . It's a way to group the text -- you could think of it like a page, which would also be in numerical order. The text just be split up into some kind of groups in order to compute the sentiment within each group. Tweets are natural groups of words, and you want to compute the sentiment within a single tweet -- you don't need to split it up farther. In the example, the figure has a bar for each "page" of the book. You'll have a bar for each tweet. You need to assign the tweets consecutive numbers because they don't have a natural order. I did that below using rowid_to_column(), and I named the new column "tweet". It just contains the row numbers of the tweets, so once the ref dataframe is split up by word, each word is still tied back to the original tweet it belonged to by that number.
Note that many tweets don't have enough words with an associated sentiment to even calculate their sentiment score, so I then re-assigned a consecutive number to those that did -- this one is called "index".
I also added the argument values_fill = 0 to the pivot_wider() line because tweets with only positive (or negative) sentiment were not getting included, because the other value was NA instead of 0.
Along the way there are a couple of places where I just stop and look at the data -- this is really helpful in understanding errors.
library(tidyverse)
library(rtweet)
library(tidytext)
#twitter scraping
ref <- search_tweets(
"#refugee", n = 18000, include_rts = FALSE,lang = "en"
)
data(stop_words)
new_stops <- tibble(word = c("https", "t.co", "1", "refugee", "#refugee", "amp", "refugees",
"day", "2022", "dont", "0", "2", "#refugees", "4", "2021") ,lexicon = "sabs")
full_stop <- stop_words %>%
bind_rows(new_stops) #bind_rows adds more rows (way to merge data)
ref_w_sentiments <- ref %>%
rowid_to_column("tweet") %>%
unnest_tokens(word, text, token = "tweets") %>%
anti_join(full_stop) %>%
inner_join(get_sentiments("bing"))
# look at what the data looks like
select(ref_w_sentiments, tweet, word, sentiment)
#> # A tibble: 811 × 3
#> tweet word sentiment
#> <int> <chr> <chr>
#> 1 2 helping positive
#> 2 3 inspiring positive
#> 3 4 support positive
ref_w_scores <- ref_w_sentiments %>%
group_by(tweet) %>%
count(sentiment) %>%
pivot_wider(names_from = sentiment,
values_from = n, values_fill = 0) %>%
mutate(sent_score = positive - negative) %>%
# not all tweets were scored, so create a new index
rowid_to_column("index")
# look at the data again
ref_w_scores
#> # A tibble: 418 × 5
#> # Groups: tweet [418]
#> index tweet positive negative sent_score
#> <int> <int> <int> <int> <int>
#> 1 1 2 1 0 1
#> 2 2 3 1 0 1
#> 3 3 4 1 0 1
ggplot(ref_w_scores) + #plot the overall sentiment (pos - neg) versus index,
geom_col(aes(x = index, y = sent_score), show.legend = F)