Encoding Issues when reading R Object - r

Question

I am reading an R object with the readRDS. It should have two columns, a year and a character string. For most rows, the character string is OK, but some have a strange white blob and others seem to have a character vector with escaped special characters and some have special characters like â.
I think its an encoding issue with the original data (which is not mine), but am unsure what the blobs are or what causes the character vectors / escaping. I realise its probably the original data, but trying to understand a little more of what I am seeing so I can investigate.
I'm using macOS 10.14.6.
Any ideas welcome.
The original data is here and I used the following to pull out some of the rows with strange characters.
data <- readRDS("all_speech.rds") %>%
select(year, speech) %>%
filter(str_detect(speech, "â"))
str(hansardOrig)
'data.frame': 2286324 obs. of 2 variables:
$ year : num 1979 1979 1979 1979 1979 ...
$ speech: chr "Mr. Speaker ...
Added
sample <- data %>% mutate(speech = substr(speech, 1, 200))
dput(head(sample))
structure(list(year = c(1982, 1982, 1982, 1984, 1986, 1986),
speech = c("With this it will be convenient to take amendment No. 112, in title, line 10, leave out 'section 163 1) of’.\n",
"I am not so much surprised as astonished by the amendment. It would create tremendous problems. Police officers have a vital role in visiting places of entertainment—without a warrant—particularly in ",
"I note the hon. Gentleman's desire to retire there.\nMy right hon. Friend mentioned that we are setting up a pilot scheme with three experimental homes. They will be in adapted, domestic-style, buildin",
"The British forces in the Lebanon had their headquarters at Haddâsse. From that position they would have been totally unable to help British nationals in west Beirut. They are better able to help, thr",
"We know that soon more cars will be manufactured in the United Kingdom, as the hon. Member for Edinburgh, Central Mr. Fletcher) wishes.\nhirdly, the decision will have a domino effect—that American phr",
"I beg to move,\nThat leave be given to bring in a Bill to make illegal the display of pictures of naked or partially naked women in sexually provocative poses in newspapers.\nThis is a simple but import"
)), row.names = c(NA, 6L), class = "data.frame")

Answer 1

You've got a difficult problem ahead of you. The sample you show has inconsistent encodings, so fixups will be hard to do.
The first entry in sample$speech displays like this on my Mac:
> sample$speech[1]
[1] "With this it will be convenient to take amendment No. 112, in title,
line 10, leave out 'section 163 1) of’.\n"
This looks okay up to the end, where the ’ characters look like a UTF-8 encoding for a directional quote "’", interpreted in the WINDOWS-1252 encoding. I can fix that with this code:
> iconv(sample$speech[1], from="utf-8", to="WINDOWS-1252")
[1] "With this it will be convenient to take amendment No. 112, in title,
line 10, leave out 'section 163 1) of’.\n"
However, this messes up the second entry, because it has em-dashes correctly encoded, so the translation converts them to hex 97 characters, not legal in the native UTF-8 encoding on the Mac:
> sample$speech[2]
[1] "I am not so much surprised as astonished by the amendment. It would
create tremendous problems. Police officers have a vital role in visiting
places of entertainment—without a warrant—particularly in "
> iconv(sample$speech[2], from="utf-8", to="WINDOWS-1252")
[1] "I am not so much surprised as astonished by the amendment. It would
create tremendous problems. Police officers have a vital role in visiting
places of entertainment\x97without a warrant\x97particularly in "
There are functions in various packages to guess encodings and to fix them, e.g. rvest::repair_encoding, stringi::stri_enc_detect, but I couldn't get them to work on your data. I wrote one myself, based on these ideas: use utf8ToInt to convert each string to its Unicode code point, then look for which ones contain multiple high values in a sequence. sample$speech[1] looks like this:
> utf8ToInt(sample$speech[1])
[1] 87 105 116 104 32 116 104 105 115 32 105 116 32 119 105 108 108
[18] 32 98 101 32 99 111 110 118 101 110 105 101 110 116 32 116 111
[35] 32 116 97 107 101 32 97 109 101 110 100 109 101 110 116 32 78
[52] 111 46 32 49 49 50 44 32 105 110 32 116 105 116 108 101 44
[69] 32 108 105 110 101 32 49 48 44 32 108 101 97 118 101 32 111
[86] 117 116 32 39 115 101 99 116 105 111 110 32 49 54 51 32 49
[103] 41 32 111 102 226 8364 8482 46 10
and that sequence near the end 226 8364 8482 is typical for a misinterpreted UTF-8 character. (The Wikipedia page describes the encoding in detail. Two byte chars start with 192 to 223, three byte chars start with 224 to 239, and four byte chars start with 240 to 247. Chars after the first are all in the range 128 to 191. The tricky part is figuring out how these high order chars will be displayed, because that depends on the wrongly assumed encoding.) Here's a quick and dirty function that tries every encoding known to iconv() and reports on what it does:
fixEncoding <- function(s, guess = iconvlist()) {
firstbytes <- list(as.raw(192:223),
as.raw(224:239), as.raw(240:247))
nextbytes <- as.raw(128:191)
for (i in seq_along(s)) {
str <- utf8ToInt(s[i])
if (any(str > 127)) {
fixes <- c()
encs <- c()
for (g in guess) {
high <- which(str > 127)
firsts <- lapply(firstbytes, function(s) utf8ToInt(iconv(rawToChar(s), from = g, to = "UTF-8", sub="")))
nexts <- utf8ToInt(iconv(rawToChar(nextbytes), from = g, to = "UTF-8", sub = ""))
for (try in 1:3) {
starts <- high[str[high] %in% firsts[[try]]]
starts <- starts[starts <= length(str) - try]
for (hit in starts) {
if (str[hit+1] %in% nexts &&
(try < 2 || str[hit+2] %in% nexts) &&
(try < 3 || str[hit+3] %in% nexts))
high <- setdiff(high, c(hit, hit + 1,
if (try > 1) hit + 2,
if (try > 2) hit + 3))
}
}
if (!length(high)) {
fixes <- c(fixes, iconv(s[i], from = "UTF-8", to = g, mark = FALSE))
encs <- c(encs, g)
}
}
if (length(fixes)) {
if (length(unique(fixes)) == 1) {
s[i] <- fixes[1]
message("Fixed s[", i, "] using one of ", paste(encs, collapse=","), "\n", sep = "")
} else {
warning("s[", i, "] has multiple possible fixes.")
message("It could be")
uniq <- unique(fixes)
for (u in seq_along(uniq))
message(paste(encs[fixes == uniq[u]], collapse = ","), "\n")
message("Not fixed!\n")
}
}
}
}
s
}
When I try it on your sample, I see this:
> fixed <- fixEncoding(sample$speech)
Fixed s[1] using one of CP1250,CP1252,CP1254,CP1256,CP1258,MS-ANSI,MS-ARAB,MS-EE,MS-TURK,WINDOWS-1250,WINDOWS-1252,WINDOWS-1254,WINDOWS-1256,WINDOWS-1258
You can make it less verbose by calling it as
fixed <- suppressMessages(fixEncoding(sample$speech))
The other issue you had in your original post was that some strings were being displayed as single characters. I think that's an RStudio bug. If I put too many characters in a single element in a dataframe, the RStudio viewer can't display it. For me the limit is around 10240 chars. This dataframe won't display properly:
d <- data.frame(x = paste(rep("a", 10241), collapse=""))
but any smaller number works. This isn't an R issue; it can display that dataframe in the console with no problem. It's only View(d) that is bad, and only in RStudio.