Split words in r - r

Question

I have a huge list of text files (50,000+) that contain normal sentences. Some of these sentences have words that have merged together because some of the endlines have been placed together. How do I go about unmerging some of these words in R?
The only suggestion I could get was here and kind of attempted something from here but both suggestions require big matrices which I can't use because I either run out of memory or RStudio crashes :( can someone help please? Here's an example of a text file I'm using (there are 50,000+ more where this came from):
Mad cow disease, BSE, or bovine spongiform encephalopathy, has cost the country dear.
More than 170,000 cattle in England, Scotland and Wales have contracted BSE since 1988.
More than a million unwanted calves have been slaughtered, and more than two and a quarter million older cattle killed, their remains dumped in case they might be harbouring the infection.
In May, one of the biggest cattle markets, at Banbury in Oxfordshire, closed down. Avictim at least in part, of this bizarre crisis.
The total cost of BSE to the taxpayer is set to top £4 billion.
EDIT: for example:
"It had been cushioned by subsidies, living in an unreal world. Many farmers didn't think aboutwhat happened beyond the farm gate, because there were always people willing to buy what they produced."
See the 'aboutwhat' part. Well that happens to about 1 in every 100 or so articles. Not this actual article, I just made the above up as an example. The words have been joined together somehow (I think when I read in some articles some of them have missed spaces or my notepad reader joins the end of one line with another).
EDIT 2: here's the error I get when I use variation of what they have here replacing the created lists with read-in lists:
Error: assertion 'tree->num_tags == num_tags' failed in executing regexp: file 'tre-compile.c', line 627
I've never seen that error before but it does come up here and here but no solution to it on either :(

Answer 1

Based on your comments, I'd use an environment which is basically a hashtable in R. Start by building a hash of all known words:
words <- new.env(hash=TRUE)
for (w in c("hello","world","this","is","a","test")) words[[tolower(w)]] <- T
(you'd actually want to use the contents of /usr/share/dict/words or similar), then we define a function that does what you described:
dosplit <- function (w) {
if(is.null(words[[tolower(w)]])) {
n <- nchar(w)
for (i in 1:(n-1)) {
a <- substr(w,1,i)
b <- substr(w,i+1,n)
if(!is.null(words[[tolower(a)]]) && !is.null(words[[tolower(b)]]))
return (c(a,b))
}
}
w
}
then we can test it:
test <- 'hello world, this isa test'
ll <- lapply(strsplit(test,'[ \t]')[[1]], dosplit)
and if you want it back into a space separated list:
do.call(paste, as.list(unlist(ll,use.names=FALSE)))
Note that this is going to be slow for large amounts of text, R isn't really built for this sort of thing. I'd personally use Python for this sort of task, and a compiled language if it got much larger.