What's the "internal method" of R's download.file? - r

Question

I'm trying to download the following dataset with download.file, which only works when method = "wget")
# Doesn't work
download.file('http://uofi.box.com/shared/static/bba3968d7c3397c024ec.dta', tempfile(), method = "auto")
download.file('http://uofi.box.com/shared/static/bba3968d7c3397c024ec.dta', tempfile(), method = "curl")
# Works
download.file('http://uofi.box.com/shared/static/bba3968d7c3397c024ec.dta', tempfile(), method = "wget")
According to help(download.file),
If method = "auto" is chosen (the default), the internal method is
chosen for file:// URLs, and for the others provided
capabilities("http/ftp") is true (which it almost always is).
Looking at the source code, "internal method" refers to:
if (method == "internal") {
status <- .External(C_download, url, destfile, quiet,
mode, cacheOK)
if (!quiet)
flush.console()
}
But still, I don't know what .External(C_download) does, especially across platform. It's important for me to know this instead of relying on wget because I'm writing a package that should work cross-platform.

Answer 1

The source code for this is in the R sources (download the current version from http://cran.r-project.org/sources.html). The relevant code (as of R 3.2.1) is in "./src/modules/internet/internet.c" and "./src/modules/internet/nanohttp.c".
According to the latter, the code for the minimalist HTTP GET functionality is based on libxml2-2.3.6.
The files are also available on the R svn site at https://svn.r-project.org/R/branches/R-3-2-branch/src/modules/internet/internet.c and https://svn.r-project.org/R/branches/R-3-2-branch/src/modules/internet/nanohttp.c if you'd prefer not to download the whole .tgz file and decompress it.
If you look at the code, most of it is consistent across platforms. However, on Windows, the wininet code seems to be used.
The code was identified by looking initially in the utils package, since that is where the R command download.file is found. I grepped for download in the c files in the "./src/library/utils/src" directory and found that the relevant code was in "sock.c". There was a comment high up in that file which read /* from src/main/internet.c */ and so I next went to "internet.c".
With respect to your specific file, the issue is that the link you have returns a 302 Found status code. On Windows and using wget, the download routine follows the Location field of the 302 response and gets the actual file. Using the curl method works but only if you supply the parameter extra="-L".
download.file('http://uofi.box.com/shared/static/bba3968d7c3397c024ec.dta', tempfile(), method = "curl", extra="-L")
There's a package called downloader which claims to offer a good cross-platform solution for https. Given an http URL, it just passes the call onto download.file. Here's a version that works for http too. It also defaults to binary transfers, which seems generally to be a good idea.
my_download <- function(url, destfile, method, quiet = FALSE,
mode = "wb", cacheOK = TRUE, extra = getOption("download.file.extra")) {
if (.Platform$OS.type == "windows" && (missing(method) || method %in% c("auto", "internal", "wininet"))) {
seti2 <- utils::"setInternet2"
internet2_start <- seti2(NA)
on.exit(suppressWarnings(seti2(internet2_start)))
suppressWarnings(seti2(TRUE))
} else {
if (missing(method)) {
if (nzchar(Sys.which("wget")[1])) {
method <- "wget"
} else if (nzchar(Sys.which("curl")[1])) {
method <- "curl"
if (!grepl("-L", extra)) {
extra <- paste("-L", extra)
}
} else if (nzchar(Sys.which("lynx")[1])) {
method <- "lynx"
} else {
stop("no download method found")
}
}
}
download.file(url = url, destfile = destfile, method = method, quiet = quiet, mode = mode,
cacheOK = cacheOK, extra = extra)
}

Answer 2

You can answer this yourself. Just type download.file at the console prompt and you should see this near the top of the function definition:
if (method == "auto") { # this is actually the default from
# getOption("download.file.method", default = "auto")
if (capabilities("http/ftp"))
method <- "internal"
else if (length(grep("^file:", url))) {
method <- "internal"
url <- URLdecode(url)
}
else if (system("wget --help > /dev/null") == 0L)
method <- "wget"
else if (system("curl --help > /dev/null") == 0L)
method <- "curl"
else if (system("lynx -help > /dev/null") == 0L)
method <- "lynx"
else stop("no download method found")
}
if (method == "internal") {
status <- .External(C_download, url, destfile, quiet,
mode, cacheOK)
if (!quiet)
flush.console()
}