Windows system call fails while building the book "R Packages" - r

Question

I'm trying to build Hadley's book "R Packages" from source. I have downloaded and unzipped the source and am have combined the code from Getting started (with some modifications) and the code in r-pkgs-first-edition-freeze\book\build-book.r to build the book (original code at the of this post). The first part of the code from Getting started works as expected:
# install.packages(c("devtools", "roxygen2", "testthat", "knitr")) # orginal code
library(devtools); library(roxygen2); library(testthat); library(knitr) # what I'm using
# install.packages("rstudioapi") # orginal code
library(rstudioapi) # what I'm using
rstudioapi::isAvailable("0.99.149")
[1] TRUE # expected result
In the next part, has_devel() doesn't give the verbose output described in Getting started, but it also doesn't throw an error, and .Last.value == TRUE, as expected, so I think I'm OK:
devtools::install_github("r-lib/devtools")
has_devel()
.Last.value
[1] TRUE
The last part of the code from Getting started also works. Many of my packages are newer versions compared to what is listed in Getting started, but I don't think this is the source of my subsequent problems.
library(roxygen2)
library(testthat)
devtools::session_info()
─ Session info ───────────────────────────────────────────────────────────
setting value
version R version 3.5.1 (2018-07-02)
os Windows 7 x64 SP 1
system x86_64, mingw32
ui RStudio
language (EN)
collate English_United States.1252
tz America/Chicago
date 2019-04-10
─ Packages ───────────────────────────────────────────────────────────────
package * version date source
assertthat 0.2.0 2017-04-11 CRAN (R 3.5.1)
backports 1.1.2 2017-12-13 CRAN (R 3.5.0)
callr 2.0.4 2018-05-15 CRAN (R 3.5.1)
cli 1.0.0 2017-11-05 CRAN (R 3.5.1)
clisymbols 1.2.0 2017-05-21 CRAN (R 3.5.1)
commonmark 1.5 2018-04-28 CRAN (R 3.5.1)
crayon 1.3.4 2017-09-16 CRAN (R 3.5.1)
desc 1.2.0 2018-05-01 CRAN (R 3.5.1)
devtools * 2.0.2.9000 2019-04-11 Github (r-lib/devtools#ab6c878)
digest 0.6.15 2018-01-28 CRAN (R 3.5.1)
evaluate 0.11 2018-07-17 CRAN (R 3.5.1)
fs 1.2.5 2018-07-30 CRAN (R 3.5.1)
glue 1.3.0 2018-07-17 CRAN (R 3.5.1)
htmltools 0.3.6 2017-04-28 CRAN (R 3.5.1)
knitr * 1.20 2018-02-20 CRAN (R 3.5.1)
magrittr 1.5 2014-11-22 CRAN (R 3.5.1)
memoise 1.1.0 2017-04-21 CRAN (R 3.5.1)
oldbookdown * 0.1 2019-04-11 Github (hadley/oldbookdown#0ffc6fb)
pkgbuild 1.0.3 2019-04-11 Github (r-lib/pkgbuild#79cb7a0)
pkgload 1.0.1.9000 2019-04-11 Github (r-lib/pkgload#0ef4f58)
prettyunits 1.0.2 2015-07-13 CRAN (R 3.5.1)
processx 3.1.0 2018-05-15 CRAN (R 3.5.1)
R6 2.2.2 2017-06-17 CRAN (R 3.5.0)
Rcpp 0.12.18 2018-07-23 CRAN (R 3.5.1)
remotes 2.0.3 2019-04-09 url
RevoUtils * 11.0.1 2018-08-01 local
RevoUtilsMath * 11.0.0 2018-08-01 local
rlang 0.2.1 2018-05-30 CRAN (R 3.5.1)
rmarkdown * 1.10 2018-06-11 CRAN (R 3.5.1)
roxygen2 * 6.1.0 2018-07-27 CRAN (R 3.5.1)
rprojroot 1.3-2 2018-01-03 CRAN (R 3.5.1)
rstudioapi * 0.7 2017-09-07 CRAN (R 3.5.1)
sessioninfo 1.0.0 2017-06-21 CRAN (R 3.5.1)
stringi 1.2.4 2018-07-20 CRAN (R 3.5.1)
stringr 1.3.1 2018-05-10 CRAN (R 3.5.1)
testthat * 2.0.0 2017-12-13 CRAN (R 3.5.1)
usethis * 1.4.0 2018-08-14 url
withr 2.1.2 2018-03-15 CRAN (R 3.5.1)
xml2 1.2.0 2018-01-24 CRAN (R 3.5.1)
yaml 2.2.0 2018-07-25 CRAN (R 3.5.1)
The first part of the code in build-book.r works for me:
install_github("hadley/oldbookdown#0ffc6fb") # code I needed to add
library(oldbookdown)
library(rmarkdown)
# Render chapters into tex ------------------------------------------------
needs_update <- function(src, dest) {
if (!file.exists(dest)) return(TRUE)
mtime <- file.info(src, dest)$mtime
mtime[2] < mtime[1]
}
render_chapter <- function(src) {
dest <- file.path("book/tex/", gsub("\\.rmd", "\\.tex", src))
if (!needs_update(src, dest)) return()
message("Rendering ", src)
command <- bquote(rmarkdown::render(.(src), oldbookdown::tex_chapter(),
output_dir = "book/tex", quiet = TRUE, env = globalenv()))
writeLines(deparse(command), "run.r")
on.exit(unlink("run.r"))
source_clean("run.r")
}
source_clean <- function(path) {
r_path <- file.path(R.home("bin"), "R")
cmd <- paste0(shQuote(r_path), " --quiet --file=", shQuote(path))
out <- system(cmd, intern = TRUE)
status <- attr(out, "status")
if (is.null(status)) status <- 0
if (!identical(as.character(status), "0")) {
stop("Command failed (", status, ")", call. = FALSE)
}
}
chapters <- dir(".", pattern = "\\.rmd$")
But the code fails when I try to actually render the chapters:
lapply(chapters, render_chapter)
Rendering check.rmd
Error: Command failed (1)
In addition: Warning message:
In system(cmd, intern = TRUE) :
running command '"C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R" --quiet --file="run.r"' had status 1
Called from: source_clean("run.r")
Browse[1]
The system call fails, but since I don't know what the Windows call "C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R" --quiet --file="run.r" is supposed to do, I have no idea how to trouble shoot this. I tried changing r_path <- file.path(R.home("bin"), "R") to r_path <- file.path(R.home("bin")) (since there is no "R" directory in C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64), but I still get the error below:
lapply(chapters, render_chapter)
Rendering check.rmd
Error in system(cmd, intern = TRUE) :
'"C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64"' not found
I also tried changing file.path(R.home("bin"), "R") to file.path(R.home("bin"), "R"), since there is a "R.exe" executable file in this directory, also with no luck. Any guidance would be appreciated.
Additional information, in response to #onlyphantom: it appears that the broken path is found by R.home("bin") has also affected other system variables, including R_DOC_DIR and R_HOME. But PATH points to the correct location, C:\Program Files\Microsoft\R Open\R-3.5.1\bin\x64.
file.path(R.home("bin"))
[1] "C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64"
Sys.getenv() # returned values below are redacted
APPDATA C:\Users\JT\AppData\Roaming
HOME C:/Users/JT/R/Projects
HOMEDRIVE C:
HOMEPATH \Users\JT\R
LOCALAPPDATA C:\Users\JT\AppData\Local
PATH C:\Program Files\Microsoft\R Open\R-3.5.1\bin\x64;...
ProgramData C:\ProgramData
ProgramFiles C:\Program Files
ProgramFiles(x86) C:\Program Files (x86)
R_DOC_DIR C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/doc
R_HOME C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1
R_LIBS_USER C:/Program Files/Microsoft/R Open/R-3.5.1/library
R_USER C:/Users/JT/R
I tried uninstalling and then reinstalling R Open, but still get this:
file.path(R.home("bin"), "R")
[1] "C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R"
I tried switching to regular R (instead of R Open) and continued to have the same problem.
Windows Command Line Problem: I walked step-by-step through the R code to find the exact location of the problem. It occurs when render_chapter() calls source_clean("run.r"), which eventually calls out <- system(cmd, intern = TRUE). cmd (for the first object in chapters) is "C:\Program Files\R\R-3.5.3\bin\x64\R" --quiet --file="run.r". When I open a Windows command prompt and run this code I get the following:
C:\>cd C:\Users\JT\R\Books\r-pkgs-first-edition-freeze
C:\Users...freeze>"C:\Program Files\R\R-3.5.3\bin\x64\R" --quiet --file="run.r"
> rmarkdown::render("check.rmd", oldbookdown::tex_chapter(), output_dir = "book/tex",
+ quiet = TRUE, env = globalenv())
--chapters has been removed. Use --top-level-division=chapter instead.
Try pandoc.exe --help for more information.
Error: pandoc document conversion failed with error 2
Execution halted
When I run pandoc.exe --help I see an entry --top-level-division=section|chapter|part. The switch away from --chapters happened after pandocs v1.17.2, so I uninstalled the current version and installed pandocs v1.17.2. I think this fixed the problem, but created a new problem. I now get this error (with either the short or long form of the path name):
C:\Users...freeze>"C:/PROGRA~1/R/R-35~1.3/bin/x64/R" --quiet --file="run.r"
#or#
C:\Users...freeze>"C:\Program Files\R\R-3.5.3\bin\x64\R" --quiet --file="run.r"
> rmarkdown::render("check.rmd", oldbookdown::tex_chapter(), output_dir = "book/
tex",
+ quiet = TRUE, env = globalenv())
pandoc.exe: Unknown reader: markdown_style
Error: pandoc document conversion failed with error 7
Execution halted
My assumption is that the old version of pandocs is not compatible with the new markdown_style. So I'm going to try installing the newest version of pandocs and then edit the run.r file to change --chapters to --top-level-division=section|chapter|part. If that's a stupid idea, please let me know.

Answer 1

The problem lies with a wrong R home directory. C:/PROGRA~1 is just the short form of path elements, and really points to C:/PROGRAM FILES/....
The same for R-35~1.1, it points to /R/R-3.5.1 or something. It's just a "short name".
The system call fails, but since I don't know what the Windows call
"C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R" --quiet
--file="run.r" is supposed to do, I have no idea how to trouble shoot this.
Your script includes the following code:
source_clean <- function(path) {
r_path <- file.path(R.home("bin"), "R")
...
}
The value of r_path at this point is C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R, which may have resulted from a previous (or failed) installation of R Open.
I tried changing r_path <- file.path(R.home("bin"), "R") to r_path <-
file.path(R.home("bin")) (since there is no "R" directory in
C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64), but I still get the
error below:
"C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64"' not found
What you can do instead, is to set r_path directly to the directory containing your R installation. Open up R (in Rstudio or R console) and verify this:
file.path(R.home("bin"))
[1] "/Library/Frameworks/R.framework/Resources/bin"
Sys.getenv("R_HOME")
"/Library/Frameworks/R.framework/Resources"
The R.home function returns the R home directory. If you navigate to that exact location on your machine, you should see that's where your R installation is that.
Setting R environment variable on Windows
Let's say you find your R installation here C:\Program Files\R\R-3.3.2\bin\x86, then do the following:
Open Start Menu, type in "View advanced system settings" and hit "Environment variables".
Under "System variables", select Path and click on edit
Click “New”, and add the folder address for R to there (C:\Program Files\R\R-3.3.2\bin\x86)
Refer also to this answer and this for tips to set the path of your R home.

Answer 2

Turns out, there were two problems, both of which gave essentially the same error when lapply(chapters, render_chapter) was called. And it had nothing to do with the short version of the path (i.e. C:/PROGRA~1/MIE74D~1/ROPEN~1/R-35~1.1/bin/x64/R).
First, oldbookdown::tex_chapter() returned a list that was not compatible with the latest version of pandocs. So I changed the render_chapter function as follows:
render_chapter <- function(src) {
dest <- file.path("book/tex/", gsub("\\.rmd", "\\.tex", src))
if (!needs_update(src, dest)) return()
message("Rendering ", src)
# "changed oldbookdown::tex_chapter()" to "temp"
command <- bquote(rmarkdown::render(.(src), temp, output_dir = "book/tex", quiet = TRUE, env = globalenv()))
# added definition for temp (with appropriate changes to
# oldbookdown::tex_chapter()) to the writeLines() call
writeLines(
c(
"temp <- oldbookdown::tex_chapter()",
'temp$pandoc$from <- "markdown"',
'temp$pandoc$args[1] <- "--top-level-division=chapter"',
deparse(command)
),
"run.r"
)
on.exit(unlink("run.r"))
source_clean("run.r")
}
Second, pandocs was calling multiple R packages in the background and failing if they were not installed. So I had to call render_chapter on each object in chapters individually to identify and install the missing packages.
Now system("xelatex -interaction=batchmode r-packages ") is failing, but that's a topic for a different question.