Seperating characters in dfm object R - r

Question

all,
I have imported the sotu corpus from quanteda in R. I am somewhat new to dfm objects and am wanting to separate the doc_id column to give me a name and a year column. If this was a tibble, this code works:
library(quanteda)
library(quanteda.corpora)
library(tidyverse)
sotu <- as_tibble(data_corpus_sotu)
sotusubsetted <- sotu %>%
separate(doc_id, c("name","year"),"-")
However, since I am new with dfm and regex, I am not sure if there is an equivalent process if I load in the data as:
library(quanteda)
library(quanteda.corpora)
library(tidyverse)
sotu <- corpus(data_corpus_sotu)
sotudfm <- dfm(sotu)
Is there some equivalent way to do this with dfm objects?

Answer 1

The safest method is also one that will work for any core quanteda object, meaning equally for a corpus, tokens, or dfm object. These involve using the accessor functions, not addressing the internals of the corpus or dfm objects directly, which is strongly discouraged. You can do that, but your code could break in the future if those object structures are changed. In addition, our accessor functions are generally also the most efficient method.
For this task, you want to use the docnames() functions or accessing the document IDs, and this works for the corpus as well as for the dfm.
library("quanteda")
## Package version: 2.1.2
data("data_corpus_sotu", package = "quanteda.corpora")
data.frame(doc_id = docnames(data_corpus_sotu[1:5])) %>%
tidyr::separate(doc_id, c("name", "year"), "-")
## name year
## 1 Washington 1790
## 2 Washington 1790b
## 3 Washington 1791
## 4 Washington 1792
## 5 Washington 1793
data.frame(doc_id = docnames(dfm(data_corpus_sotu[1:5]))) %>%
tidyr::separate(doc_id, c("name", "year"), "-")
## name year
## 1 Washington 1790
## 2 Washington 1790b
## 3 Washington 1791
## 4 Washington 1792
## 5 Washington 1793
You could also have taken this from the "President" docvar field and the "Date":
data.frame(
name = data_corpus_sotu$President,
year = lubridate::year(data_corpus_sotu$Date)
) %>%
head()
## name year
## 1 Washington 1790
## 2 Washington 1790
## 3 Washington 1791
## 4 Washington 1792
## 5 Washington 1793
## 6 Washington 1794
Created on 2021-02-13 by the reprex package (v1.0.0)

Answer 2

The following code will do exactly what you want, albeit it might break some operations in quanteda that will look for docid_ in sotudfm#docvars, the data frame that stores the documents relational data. For instance, it will break any filtering by sotudfm#Dimnames$docs, which is where the dimension names of the documents are listed.
sotudfm#docvars <- sotudfm#docvars %>% separate(col = docid_, c("name","year"),"-")
> sotudfm#docvars %>% as_tibble()
# A tibble: 241 x 10
docname_ name year segid_ FirstName President Date delivery type party
<chr> <chr> <chr> <int> <chr> <chr> <date> <fct> <fct> <fct>
1 Washington-1790 Washington 1790 1 George Washington 1790-01-08 spoken SOTU Independent
2 Washington-1790b Washington 1790b 1 George Washington 1790-12-08 spoken SOTU Independent
3 Washington-1791 Washington 1791 1 George Washington 1791-10-25 spoken SOTU Independent
4 Washington-1792 Washington 1792 1 George Washington 1792-11-06 spoken SOTU Independent
5 Washington-1793 Washington 1793 1 George Washington 1793-12-03 spoken SOTU Independent
6 Washington-1794 Washington 1794 1 George Washington 1794-11-19 spoken SOTU Independent
7 Washington-1795 Washington 1795 1 George Washington 1795-12-08 spoken SOTU Independent
8 Washington-1796 Washington 1796 1 George Washington 1796-12-07 spoken SOTU Independent
9 Adams-1797 Adams 1797 1 John Adams 1797-11-22 spoken SOTU Federalist
10 Adams-1798 Adams 1798 1 John Adams 1798-12-08 spoken SOTU Federalist

Answer 3

Here is the code that ended up working for me:
sotudfm#docvars <- sotudfm#docvars %>%
separate(col = docname_, c("name","year"),"-")
This kept the doc_id intact when I ran
head(sotudfm, 10)
It appears that docid_ and docname_ are identical.