iconvlist() inconsistency on alpine linux - r

Question

I have a docker container set up that is based on artemklevtsov/r-alpine:latest. When I run my R scripts I see this error:
Invalid encoding UTF-8: defaulting to UTF-8.
I tracked this down to this code in the httr library:
https://github.com/hadley/httr/blob/master/R/content-parse.r#L5
It looks like iconvlist() on alpine returns encodings that have a trailing comma at the end, ex:
iconvlist()
[1] "..." "ISO8859-1," "ISO8859-2," "ISO8859-3," "ISO8859-4,"
[6] "ISO8859-5," "ISO8859-6," "ISO8859-7," "UCS-2BE," "UCS-2LE,"
[11] "US_ASCII," "UTF-16BE," "UTF-16LE," "UTF-32BE," "UTF-8,"
Therefore UTF-8 never matches UTF-8,. Has anyone ran into this issue before? The list of encodings I get on my local Mac (OSX) is correct and doesn't have trailing commas. It also doesn't happen on CentOS, so it looks like it's specific to alpine.
Is there a way to get around this? Maybe through a configuration in R or by modifying the iconvlist() output?

Answer 1

I have the same issue, this time from calling read::read_csv, which uses base::iconvlist and gives the same error message Invalid encoding UTF-8: defaulting to UTF-8.. This is on alpine:3.12 using R 3.6.3 provided by apk add R and, based on the details below, I think the issue will be present on any version of alpine and R unless steps have been taken to address it directly.
I found a couple of solutions. TLDR:
Remove the commas from the file at system.file("iconvlist", package = "utils"), or
Recompile R using the gnu-libiconv library for more comprehensive iconv support.
Solution 1
The base::iconvlist() function uses an iconvlist file as a fallback method to get the list of encodings the system supports. On alpine this fallback method will always be used, for reasons outlined below, but the iconvlist file has commas in, which R is not expecting.
The easiest solution is to remove the commas from the iconvlist file, which can be found with base::system.file().
> system.file("iconvlist", package = "utils")
[1] "/usr/lib/R/library/utils/iconvlist"
One way to remove the commas, from the command line (not R) is:
sed -i 's/,//g' /usr/lib/R/library/utils/iconvlist
Subsequent calls to base::iconvlist() will read and parse the new file without the commas, and other functions that rely on base::iconvlist() will be able to successfully check for support, e.g. for "UTF-8".
> iconvlist()
[1] "..." "ISO8859-1" "ISO8859-2" "ISO8859-3" "ISO8859-4" "ISO8859-5"
[7] "ISO8859-6" "ISO8859-7" "UCS-2BE" "UCS-2LE" "US_ASCII" "UTF-16BE"
[13] "UTF-16LE" "UTF-32BE" "UTF-8" "UTF32-LE" "WCHAR_T"
> "UTF-8" %in% iconvlist()
[1] TRUE
Why is this necessary?
International conversion (iconv) of character encodings is a feature that R expects to be provided by the operating system, stipulated in the R Administration and Installation Manual. Operating systems provide their own implementations of iconv functionality, sometimes with fewer features. Since alpine is designed to be minimal, it is not surprising that it provides only what is necessary to meet the POSIX standards.
When R is built on a system it first checks the extent of iconv support from the host's C development libraries, before it compiles features into R's internals. Crucially, support for the C function iconvlist is checked for, which is not present on alpine, as shown in the apk build log for R: checking for iconvlist... no, so this C function is not available to R internally.
R's base::iconvlist() function will first try to get encodings using pre-compiled C code via .Internal(iconv(..., which will call iconvlist (in C) if available. As the iconvlist C function is not present on alpine, this .Internal call will always return NULL, and the R function will fall back to reading the info from the iconvlist file:
> iconvlist
function ()
{
int <- .Internal(iconv(NULL, "", "", "", TRUE, FALSE))
if (length(int))
return(sort.int(int))
icfile <- system.file("iconvlist", package = "utils")
# ... (truncated)
Why is the iconvlist file in an unexpected format?
The iconvlist file is created when R is built, from the command iconv -l which lists the available encodings. This is the utility program at /usr/bin/iconv not an R or C function. There is no standard for the format of the output of iconv -l. Alpine tries to conform to POSIX standards, and these only require that the -l option writes "values to standard output in an unspecified format".
R is expecting the file format to contain values separated by spaces (base::iconvlist() parses the file with strsplit(ext, "[[:space:]]")), which is true for other Linux variants, e.g. Debian, CentOS, but not for alpine's musl libc version, which has the commas.
Solution 2
A more rigorous solution is to build R from source using an alternative iconv C library implementation that provides the iconvlist C function. base::iconvlist() can then fetch the encodings via its .Internal(iconv(... call, and never needs to fall back to the iconvlist file.
An implementation that provides iconvlist is GNU libiconv, which has been packaged for alpine and can be installed with:
apk add gnu-libiconv gnu-libiconv-dev
The package gnu-libiconv-dev provides headers in /usr/include/gnu-libiconv/, so the compiler needs to be pointed here in preference to the existing ones in /usr/include. This is outside my expertise but can be done by adding -I/usr/include/gnu-libiconv to the CFLAGS environment variable.
export CFLAGS=-I/usr/include/gnu-libiconv $CFLAGS
Running ./configure should yield check results similar to:
... (truncated)
checking for iconv.h... yes
checking for iconv... in libiconv
checking whether iconv accepts "UTF-8", "latin1", "ASCII" and "UCS-*"... yes
checking whether iconv accepts "CP1252"... yes
checking for iconvlist... yes
... (truncated)
After make I can run ./bin/R and, even if the iconvlist file still contains commas, calls to base::iconvlist() yields well-formatted results:
> iconvlist()
[1] "850"
[2] "862"
[3] "866"
[4] "ANSI_X3.4-1968"
[5] "ANSI_X3.4-1986"
... (truncated)
# The unsorted list is coming from the internal C functions, not the file
> .Internal(iconv(NULL, "", "", "", TRUE, FALSE))
[1] "ANSI_X3.4-1968"
[2] "ANSI_X3.4-1986"
[3] "ASCII"
[4] "CP367"
[5] "IBM367"
... (truncated)