I am trying to download a CSV from my database online and it appears to be working,
tempCSV <- postForm(myDB_URL, .params=myParameters)
but when I try to read the file,
> dat <- read.csv(textConnection(tempCSV))
I get this error:
Error in textConnection(tempCSV) : invalid 'text' argument
I've tried this too (don't laugh if I'm completely grabbing for straws here, it is 2:19am on a Friday night)
> dat <- read.csv(tempCSV)
with this error:
Error in read.table(file = file, header = header, sep = sep, quote = quote, :
'file' must be a character string or connection
For reference, this is what the data actually looks like:
> tempCSV[1:20]
[1] 43 69 74 79 73 70 61 6e 20 43 6c 61 73 73 20 2d 20 46 69 6e
RStudio says it's raw[225758]
Here's what happens if I print the whole thing, if that helps at all:
> tempCSV
...
[ reached getOption("max.print") -- omitted 215758 entries ]
attr(,"Content-Type")
"application/x-comma-separated-values"
Related
I was doing some file merging, and two files wouldn't - despite having a key column that matches (I actually generated one key column by copy-pasting from the other). It's the damndest thing, and I worry that I'm either going crazy or missing something fundamental. As an example (and I cannot figure out how to make it reproducible, as when I copy and paste these strings into new objects, they compare just fine), here's my current console:
> q
[1] "1931 80th Anniversary"
> z
[1] "1931 80th Anniversary"
> q == z
[1] FALSE
I str-ed both, just in case I missed something, and...
> str(q)
chr "1931 80th Anniversary"
> str(z)
chr "1931 80th Anniversary"
What could be going on here?
This was a great puzzler. To answer - to diagnose the problem, charToRaw() was the answer.
> charToRaw(q)
[1] 31 39 33 31 c2 a0 38 30 74 68 c2 a0 41 6e 6e 69 76 65
[19] 72 73 61 72 79
> charToRaw(z)
[1] 31 39 33 31 20 38 30 74 68 20 41 6e 6e 69 76 65 72 73
[19] 61 72 79
Oh! Different! It seems to lie in the encoding, which, given that these were both plain ole' CSVs I loaded from, I never would have guessed, but
> Encoding(q)
[1] "UTF-8"
> Encoding(z)
[1] "unknown"
In the end, I used iconv() on q to make it work
> iconv(q, from = 'UTF-8', to = 'ASCII//TRANSLIT') == z
[1] TRUE
This has been a weird journey, and I hope this helps someone else who is as baffled as I was - and they learn a few new functions along the way.
It looks like you have non-breaking spaces in your string, which isn't really an encoding issue. This happens to me all the time because alt + space inserts a non-breaking space on a Mac, and I use alt on my German keyboard for all sorts of special characters, too. My pinkies are my slowest fingers and they don't always release alt fast enough when I transition from some special character to a space. I discovered this problem writing bash scripts, where <command> | <command> is common and | is alt + 7.
I think stringr::str_replace_all(q, "\\s", " ") should fix your current issue. Alternatively, you can try targeting specific non-printables, e.g. in your situation stringr::str_replace_all(q, "\uA0", " "). To expose the offending characters you can use stringi::stri_escape_unicode(q), which would return "1931\\u00a080th\\u00a0Anniversary". You can then just copy and paste to get the same results as above: stringr::str_replace_all(q, "\u00a0", " ")
I am reading an R object with the readRDS. It should have two columns, a year and a character string. For most rows, the character string is OK, but some have a strange white blob and others seem to have a character vector with escaped special characters and some have special characters like â.
I think its an encoding issue with the original data (which is not mine), but am unsure what the blobs are or what causes the character vectors / escaping. I realise its probably the original data, but trying to understand a little more of what I am seeing so I can investigate.
I'm using macOS 10.14.6.
Any ideas welcome.
The original data is here and I used the following to pull out some of the rows with strange characters.
data <- readRDS("all_speech.rds") %>%
select(year, speech) %>%
filter(str_detect(speech, "â"))
str(hansardOrig)
'data.frame': 2286324 obs. of 2 variables:
$ year : num 1979 1979 1979 1979 1979 ...
$ speech: chr "Mr. Speaker ...
Added
sample <- data %>% mutate(speech = substr(speech, 1, 200))
dput(head(sample))
structure(list(year = c(1982, 1982, 1982, 1984, 1986, 1986),
speech = c("With this it will be convenient to take amendment No. 112, in title, line 10, leave out 'section 163 1) of’.\n",
"I am not so much surprised as astonished by the amendment. It would create tremendous problems. Police officers have a vital role in visiting places of entertainment—without a warrant—particularly in ",
"I note the hon. Gentleman's desire to retire there.\nMy right hon. Friend mentioned that we are setting up a pilot scheme with three experimental homes. They will be in adapted, domestic-style, buildin",
"The British forces in the Lebanon had their headquarters at Haddâsse. From that position they would have been totally unable to help British nationals in west Beirut. They are better able to help, thr",
"We know that soon more cars will be manufactured in the United Kingdom, as the hon. Member for Edinburgh, Central Mr. Fletcher) wishes.\nhirdly, the decision will have a domino effect—that American phr",
"I beg to move,\nThat leave be given to bring in a Bill to make illegal the display of pictures of naked or partially naked women in sexually provocative poses in newspapers.\nThis is a simple but import"
)), row.names = c(NA, 6L), class = "data.frame")
You've got a difficult problem ahead of you. The sample you show has inconsistent encodings, so fixups will be hard to do.
The first entry in sample$speech displays like this on my Mac:
> sample$speech[1]
[1] "With this it will be convenient to take amendment No. 112, in title,
line 10, leave out 'section 163 1) of’.\n"
This looks okay up to the end, where the ’ characters look like a UTF-8 encoding for a directional quote "’", interpreted in the WINDOWS-1252 encoding. I can fix that with this code:
> iconv(sample$speech[1], from="utf-8", to="WINDOWS-1252")
[1] "With this it will be convenient to take amendment No. 112, in title,
line 10, leave out 'section 163 1) of’.\n"
However, this messes up the second entry, because it has em-dashes correctly encoded, so the translation converts them to hex 97 characters, not legal in the native UTF-8 encoding on the Mac:
> sample$speech[2]
[1] "I am not so much surprised as astonished by the amendment. It would
create tremendous problems. Police officers have a vital role in visiting
places of entertainment—without a warrant—particularly in "
> iconv(sample$speech[2], from="utf-8", to="WINDOWS-1252")
[1] "I am not so much surprised as astonished by the amendment. It would
create tremendous problems. Police officers have a vital role in visiting
places of entertainment\x97without a warrant\x97particularly in "
There are functions in various packages to guess encodings and to fix them, e.g. rvest::repair_encoding, stringi::stri_enc_detect, but I couldn't get them to work on your data. I wrote one myself, based on these ideas: use utf8ToInt to convert each string to its Unicode code point, then look for which ones contain multiple high values in a sequence. sample$speech[1] looks like this:
> utf8ToInt(sample$speech[1])
[1] 87 105 116 104 32 116 104 105 115 32 105 116 32 119 105 108 108
[18] 32 98 101 32 99 111 110 118 101 110 105 101 110 116 32 116 111
[35] 32 116 97 107 101 32 97 109 101 110 100 109 101 110 116 32 78
[52] 111 46 32 49 49 50 44 32 105 110 32 116 105 116 108 101 44
[69] 32 108 105 110 101 32 49 48 44 32 108 101 97 118 101 32 111
[86] 117 116 32 39 115 101 99 116 105 111 110 32 49 54 51 32 49
[103] 41 32 111 102 226 8364 8482 46 10
and that sequence near the end 226 8364 8482 is typical for a misinterpreted UTF-8 character. (The Wikipedia page describes the encoding in detail. Two byte chars start with 192 to 223, three byte chars start with 224 to 239, and four byte chars start with 240 to 247. Chars after the first are all in the range 128 to 191. The tricky part is figuring out how these high order chars will be displayed, because that depends on the wrongly assumed encoding.) Here's a quick and dirty function that tries every encoding known to iconv() and reports on what it does:
fixEncoding <- function(s, guess = iconvlist()) {
firstbytes <- list(as.raw(192:223),
as.raw(224:239), as.raw(240:247))
nextbytes <- as.raw(128:191)
for (i in seq_along(s)) {
str <- utf8ToInt(s[i])
if (any(str > 127)) {
fixes <- c()
encs <- c()
for (g in guess) {
high <- which(str > 127)
firsts <- lapply(firstbytes, function(s) utf8ToInt(iconv(rawToChar(s), from = g, to = "UTF-8", sub="")))
nexts <- utf8ToInt(iconv(rawToChar(nextbytes), from = g, to = "UTF-8", sub = ""))
for (try in 1:3) {
starts <- high[str[high] %in% firsts[[try]]]
starts <- starts[starts <= length(str) - try]
for (hit in starts) {
if (str[hit+1] %in% nexts &&
(try < 2 || str[hit+2] %in% nexts) &&
(try < 3 || str[hit+3] %in% nexts))
high <- setdiff(high, c(hit, hit + 1,
if (try > 1) hit + 2,
if (try > 2) hit + 3))
}
}
if (!length(high)) {
fixes <- c(fixes, iconv(s[i], from = "UTF-8", to = g, mark = FALSE))
encs <- c(encs, g)
}
}
if (length(fixes)) {
if (length(unique(fixes)) == 1) {
s[i] <- fixes[1]
message("Fixed s[", i, "] using one of ", paste(encs, collapse=","), "\n", sep = "")
} else {
warning("s[", i, "] has multiple possible fixes.")
message("It could be")
uniq <- unique(fixes)
for (u in seq_along(uniq))
message(paste(encs[fixes == uniq[u]], collapse = ","), "\n")
message("Not fixed!\n")
}
}
}
}
s
}
When I try it on your sample, I see this:
> fixed <- fixEncoding(sample$speech)
Fixed s[1] using one of CP1250,CP1252,CP1254,CP1256,CP1258,MS-ANSI,MS-ARAB,MS-EE,MS-TURK,WINDOWS-1250,WINDOWS-1252,WINDOWS-1254,WINDOWS-1256,WINDOWS-1258
You can make it less verbose by calling it as
fixed <- suppressMessages(fixEncoding(sample$speech))
The other issue you had in your original post was that some strings were being displayed as single characters. I think that's an RStudio bug. If I put too many characters in a single element in a dataframe, the RStudio viewer can't display it. For me the limit is around 10240 chars. This dataframe won't display properly:
d <- data.frame(x = paste(rep("a", 10241), collapse=""))
but any smaller number works. This isn't an R issue; it can display that dataframe in the console with no problem. It's only View(d) that is bad, and only in RStudio.
I'm using the code below to query genders from FOAF linked data. The function works when I search for 'Bowie' but not 'David Bowie'.
sparql_foaf <- function(term) {
endpoint <- "http://live.dbpedia.org/sparql"
prefix <- c("db","http://dbpedia.org/resource/",
"rdfs","http://www.w3.org/2000/01/rdf-schema#",
"foaf","http://xmlns.com/foaf/0.1/")
query <- paste0("
SELECT str(?lbl) as ?names_r, str(?gender) as ?gender WHERE {
?sub a foaf:Person .
?sub rdfs:label ?lbl .
FILTER regex(?lbl, 'Bowie')
FILTER(langMatches(lang(?lbl), 'en'))
OPTIONAL {?sub foaf:gender ?gender}
}
LIMIT 1")
SPARQL(endpoint,query,ns=prefix)$results
}
The issue being, Bowie works
FILTER regex(?lbl, 'Bowie')
And David Bowie does not
FILTER regex(?lbl, 'David Bowie')
The issue is likely something simple that I'm overlooking with the SPARQL library. However, the query does works as expected in a query environment: https://api.triplydb.com/s/r8cBeIuo
Am I missing some sort of character enclosing technique for spaces in the SPARQL R library?
Using the information in the ASKW comment and therefor removing the "person requirement" we see this result:
sparql_foaf <- function(term) {
endpoint <- "http://live.dbpedia.org/sparql"
prefix <- c("db","http://dbpedia.org/resource/",
"rdfs","http://www.w3.org/2000/01/rdf-schema#",
"foaf","http://xmlns.com/foaf/0.1/")
query <- paste0("
SELECT str(?lbl) as ?names_r, str(?gender) as ?gender WHERE {
?sub rdfs:label ?lbl .
FILTER regex(?lbl, 'Bowie')
FILTER(langMatches(lang(?lbl), 'en'))
OPTIONAL {?sub foaf:gender ?gender}
}
LIMIT 100")
SPARQL(endpoint,query,ns=prefix)$results
}
grep("David", sparql_foaf (1)[[1]] )
[1] 4 5 6 9 10 11 12 33 40 41 42 43 46 47 48 50 55 59 63 69 71 73 74 75 78 83
[27] 85 91 96 97
And:
sparql_foaf (1)[[1]][4]
[1] "Albums produced by David Bowie"
How can I convert Ａb９８７６５４３２１０ into Ab9876543210? Is there a solution by regular expression?
test <- dput("Ａb９８７６５４３２１０")
Disclaimer: The following works on my machine, but since I can't replicate your full width string based purely on the example provided, this is a best guess based on my version of the problem (pasting the string into a text file, save it with UTF-8 encoding, & loading it in with coding specified as UTF-8.
Step 1. Reading in the text (I added a half width version for comparison):
> test <- readLines("fullwidth.txt", encoding = "UTF-8")
> test
[1] "Ａb９８７６５４３２１０" "Ab9876543210"
Step 2. Verifying that the full & half width versions are not equal:
# using all.equal()
test1 <- test[1]
test2 <- test[2]
> all.equal(test1, test2)
[1] "1 string mismatch"
# compare raw bytes
> charToRaw(test1)
[1] ef bb bf ef bc a1 62 ef bc 99 ef bc 98 ef bc 97 ef bc 96 ef bc 95 ef
[24] bc 94 ef bc 93 ef bc 92 ef bc 91 ef bc 90
> charToRaw(test2)
[1] 41 62 39 38 37 36 35 34 33 32 31 30
For anyone interested, if you paste the raw byte version into a utf-8 decoder as hexadecimal input, you'll see that except for letter b (mapped from 62 in the 7th byte), the rest of the letters were formed by 3-byte sequences. In addition, the first 3-byte sequence maps to "ZERO WIDTH NO-BREAK SPACE character", so it's not visible when you print the string to console.
Step 3. Converting from full width to half width using the Nippon package:
library(Nippon)
test1.converted <- zen2han(test1)
> test1.converted
[1] "Ab9876543210"
# If you want to compare against the original test2 string, remove the zero
# width character in front
> all.equal(substring(test1.converted, 2), test2)
[1] TRUE
Here is a base R solution
Full width characters are in the range 0xFF01:0xFFEF, and can be offset like this.
x <- "Ａb９８７６５４３２１０"
iconv(x, to = "utf8") |>
utf8ToInt() |>
(\(.) ifelse(. > 0xFF01 & . <= 0xFFEF, . - 65248, .))() |>
intToUtf8()
[1] "Ab9876543210"
This question is related to the my previous question reading raw data in R to be saved as .RData file using the dropbox api
I am running into problems when my path includes non-url standard characters
the db.file.name in the previous question is just the path to the relevant file in dropbox.
however the path has a space in it along with exclamation marks. I have a feeling that these need to be converted to a relevant format so that the GET request can work...but not too sure what the conversion is....
so using and continuing from my previous example...
require(httr)
require(RCurl)
db.file.name <- "!! TEST FOLDER/test.RData"
db.app <- oauth_app("db",key="xxxxx", secret="xxxxxxx")
db.sig <- sign_oauth1.0(db.app, token="xxxxxxx", token_secret="xxxxxx")
response <- GET(url=paste0("https://api-content.dropbox.com/1/files/dropbox/",curlEscape(db.file.name)),config=c(db.sig,add_headers(Accept="x-dropbox-metadata")))
The response is an error, and no file is downloaded...using the documentation page https://www.dropbox.com/developers/reference/api it suggests putting the URL into a UTF-8 encoding...which I'm not sure how to do/not sure it works.
Any help would be greatly appreciated.
I was close before...I just needed to re-insert the slashes using gsub in order for the GET request to work... so the result was
response <- GET(url=paste0("https://api-content.dropbox.com/1/files/dropbox/",gsub("%2F","/",curlEscape(db.file.name))),config=c(db.sig,add_headers(Accept="x-dropbox-metadata")))
the quick copy-past from ?iconv,
x <- "fa\xE7ile"
Encoding(x) <- "latin1"
charToRaw(xx <- iconv(x, "latin1", "UTF-8"))
[1] 68 74 74 70 3a 2f 2f 73 74 61 63 6b 6f 76 65 72 66 6c 6f 77 2e 63 6f 6d
Encoding(x)
[1] "latin1"
Encoding(xx)
[1] "UTF-8"
does this answer your question?