rvest | read_xml claims error where read_html does not - r

Question

Taking the following url
URL <- "http://www.google.de/complete/search?output=toolbar&q=TDS38311DE"
doc <- read_xml(URL)
I get the following error:
Error: Input is not proper UTF-8, indicate encoding !
Bytes: 0xDF 0x20 0x2F 0x20 [9]
Using read_html instead everything is fine.
Am i doing something wrong? Why does this error occur?

Answer 1

First: rvest uses xml2 for the acquisition of content so you should file any issue relating to it under that package gh vs rvest.
Second, read_xml takes an encoding parameter for a reason and says so: "Unless otherwise specified XML documents are assumed to be in UTF-8 or UTF-16. If the document is not UTF-8/16, and lacks an explicit encoding directive, this allows you to supply a default."
XML files have the ability to specify an encoding but this "AJAX-y" response from google clearly isn't (and it's not something it's expecting you to be pilfering and it knows it's being read—usually—by an HTML parsing engine [a.k.a. a browser], not an XML parsing engine).
rvest used to do this:
encoding <- encoding %||% default_encoding(x)
xml2::read_xml(httr::content(x, "raw"), encoding = encoding, base_url = x$url,
as_html = as_html)
And default_encoding does this:
default_encoding <- function(x) {
type <- httr::headers(x)$`Content-Type`
if (is.null(type)) return(NULL)
media <- httr::parse_media(type)
media$params$charset
}
but rvest now only exposes read_xml methods for session and response objects (where it does the encoding guessing).
So, you can either:
do some manual introspection prior to scraping (after reading a site's ToS),
use httr to grab a page and pass that to read_xml, or
hook up your own reader function into your script with the same idiom