How use docvars in Quanteda to create document variables from raw text - r

Question

I am using Quanteda to analyze party platforms, which are txt files. I have binded two sets of party platforms into one dfm:
corp20dr <- corp20d + corp20r
summary(corp20dr)`
Document-feature matrix of: 28 documents, 6,595 features (85.0% sparse).
> summary(corp20dr)
Corpus consisting of 28 documents:
Text Types Tokens Sentences
akdem20.txt 1895 7624 332
azdem20.txt 908 2921 94
cadem20.txt 3255 19881 150
medem20.txt 355 863 39
.....................................
wvgop20.txt 1419 5013 106
wygop20.txt 428 1085 45
I would like to compare the Democratic (corp20d) and Republican platforms (corp20r). But, I seem to need to use docvars to make comparisons between the different groups (15 Dem, 13 GOP). When I use textplot_keyness, I intend to get a comparison of all the texts, but the result is to draw the first text against all other texts in the corpus.
corp20dr_dfm <- dfm(corpus(corp20dr),
remove = stopwords("english"), stem = TRUE, remove_numbers = TRUE, ```
remove_punct = TRUE)
corp20dr_dfm
result_keyness <- textstat_keyness(corp20dr_dfm)
textplot_keyness(result_keyness,
color = c('blue', 'red'))
The result is a comparison of the Alaska platform to the "reference" which seems to be the other 27 documents. I was hoping to use the compare differences in word usage between the two groups of corpora (15 Democratic platforms compared to the 13 Republican platforms), but I seem to have to identify each group using docvars. But I am not sure how to do this. Any help would be appreciated.

Answer 1

The keyness function only compares one reference document to all others, so you should group the documents by the original corpus, before calling textstat_keyness(). You can do this by using dfm_group() on a new docvar that identifies the corpus. See below for a reproducible example.
library("quanteda")
## Package version: 2.1.2
corp_a <- corpus(data_corpus_inaugural[1:5])
corp_b <- corpus(data_corpus_inaugural[6:10])
# this is the key: identifying the original corpus
# will be used to group the dfm later into just two combined "documents"
corp_a$source <- "a"
corp_b$source <- "b"
corp <- corp_a + corp_b
summary(corp)
## Corpus consisting of 10 documents, showing 10 documents:
##
## Text Types Tokens Sentences Year President FirstName
## 1789-Washington 625 1537 23 1789 Washington George
## 1793-Washington 96 147 4 1793 Washington George
## 1797-Adams 826 2577 37 1797 Adams John
## 1801-Jefferson 717 1923 41 1801 Jefferson Thomas
## 1805-Jefferson 804 2380 45 1805 Jefferson Thomas
## 1809-Madison 535 1261 21 1809 Madison James
## 1813-Madison 541 1302 33 1813 Madison James
## 1817-Monroe 1040 3677 121 1817 Monroe James
## 1821-Monroe 1259 4886 131 1821 Monroe James
## 1825-Adams 1003 3147 74 1825 Adams John Quincy
## Party source
## none a
## none a
## Federalist a
## Democratic-Republican a
## Democratic-Republican a
## Democratic-Republican b
## Democratic-Republican b
## Democratic-Republican b
## Democratic-Republican b
## Democratic-Republican b
Now we can go through the steps of forming the dfm, grouping, and getting the keyness statistics. (Here, I've removed stopwords and punctuation as well.)
# using the separate package since we are moving textstat_*() functions
# to this module package with quanteda v3 release planned in 2021
library("quanteda.textstats")
corp %>%
tokens(remove_punct = TRUE) %>%
tokens_remove(stopwords("en")) %>%
dfm() %>%
dfm_group(groups = "source") %>%
textstat_keyness() %>%
head()
## feature chi2 p n_target n_reference
## 1 love 11.236174 0.0008021834 10 1
## 2 mind 10.108762 0.0014756604 11 3
## 3 good 9.971163 0.0015901101 17 8
## 4 may 9.190508 0.0024327341 38 31
## 5 can 8.887529 0.0028712512 27 19
## 6 shall 7.728615 0.0054352433 23 16