Stopwords Remaining in Corpus After Cleaning - r

Question

I am attempting to remove the stopword "the" from my corpus, however not all instances are being removed.
library(RCurl)
library(tm)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/1.txt"
file1 <- getURL(url)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/2.txt"
file2 <- getURL(url)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/3.txt"
file3 <- getURL(url)
shakespeare <- VCorpus(VectorSource(c(file1,file2,file3)))
list<-inspect(
DocumentTermMatrix(shakespeare,list(dictionary = c("the","thee")))
)
shakespeare <- tm_map(shakespeare, stripWhitespace)
shakespeare <- tm_map(shakespeare, stemDocument)
shakespeare <- tm_map(shakespeare, removePunctuation)
tm_map(shakespeare, content_transformer(tolower))
#taken directly from tm documentation
shakespeare <- tm_map(shakespeare, removeWords, c(stopwords("english"),"the"))
list<-inspect(
DocumentTermMatrix(shakespeare,list(dictionary = c("the","thee")))
)
The first inspect call reveals:
Terms
Docs the thee
1 11665 752
2 11198 660
3 4866 382
And the second, after cleaning:
Terms
Docs the thee
1 1916 1298
2 1711 1140
3 760 740
What am I missing here about the logic of removeWords that it would ignore all these instances of "the"?
EDIT
I was able to get the instances of "the" down to below 1000 by a slight call change and making the removewords call the very first cleaning step:
shakespeare <- tm_map(shakespeare, removeWords, c(stopwords("english"),"the","The"))
Which gets me down to:
Docs the thee
1 145 752
2 130 660
3 71 382
Still though, I'd like to know why I can't seem to eliminate them all.

Answer 1

Hereby reproducable code which leads to 0 instances of "the". I solved your typo and used your code from before the edit.
library(RCurl)
library(tm)
library(SnowballC)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/1.txt"
file1 <- getURL(url)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/2.txt"
file2 <- getURL(url)
url <- "https://raw.githubusercontent.com/angerhang/statsTutorial/master/src/textMining/data/3.txt"
file3 <- getURL(url)
shakespeare <- VCorpus(VectorSource(c(file1,file2,file3)))
list<-inspect(
DocumentTermMatrix(shakespeare,list(dictionary = c("the","thee")))
)
leads to:
<<DocumentTermMatrix (documents: 3, terms: 2)>>
Non-/sparse entries: 6/0
Sparsity : 0%
Maximal term length: 4
Weighting : term frequency (tf)
Sample :
Terms
Docs the thee
1 11665 752
2 11198 660
3 4866 382
and after cleaning and solving the typo:
shakespeare <- tm_map(shakespeare, stripWhitespace)
shakespeare <- tm_map(shakespeare, stemDocument)
shakespeare <- tm_map(shakespeare, removePunctuation)
shakespeare = tm_map(shakespeare, content_transformer(tolower)) ## FIXED TYPO
#taken directly from tm documentation
shakespeare <- tm_map(shakespeare, removeWords, c(stopwords("english"),"the"))
list<-inspect(
DocumentTermMatrix(shakespeare,list(dictionary = c("the","thee")))
)
it leads to:
<<DocumentTermMatrix (documents: 3, terms: 2)>>
Non-/sparse entries: 3/3
Sparsity : 50%
Maximal term length: 4
Weighting : term frequency (tf)
Sample :
Terms
Docs the thee
1 0 1298
2 0 1140
3 0 740