I work in a corporate environment that uses Microsoft Windows Defender Application Control (WDAC) to provide security. This blocks unsigned EXE and DLL files from being installed on devices. R packages which use DLLs fail to install. The workaround to this is provide an R installation from an approved central source which also copies over a default set of packages, such as tidyverse, data.table etc. to the R library. Users can continue to install additional packages which are built with native R, but run into issues if they try to install, build from source, or update packages with DLL files in.
Is there a way to check whether a package uses DLL files in advance of installation?
Output something like:-
check_dll(foo)
result: "This package and its dependencies have no DLL files. You can install this package"
check_dll(bar)
result: "bar does not have any DLL files, but one dependency, OOF, uses DLL files.
You have already have a version of OOF installed so it should be safe to install bar"
check_dll(foobar)
result: "foobar has a DLL. Do not attempt to install foobar".
check_dll(RABOOF)
result: "RABOOF does not have any DLL files, but one of it's dependencies,
foobar, does have a DLL file. Do not attempt to install RABOOF".
tools::package_dependencies() will list the package dependencies, but nothing else.
Downloading the zip file from CRAN and inspecting it for a libs/x64 folder with contents will work, but seems a heavyweight approach. Theoretically if a package has lots of dependencies this could result in downloading a lot of files unnecessarily.
Look for the NeedsCompilation field in the DESCRIPTION file. If it is "yes", there will be a DLL. If it is "no", there probably won't be. (If it is not there, the package wasn't built properly, so all bets are off.)
The test is not perfect, because packages can put DLLs into the inst folder to get them installed without compiling them, though CRAN isn't supposed to allow that: "Source packages may not contain any form of binary executable code." But packages like pak (mentioned in the comments) may be allowed to get around this rule, e.g. by downloading binaries, so the test isn't perfect. You will also need to put together a blacklist of packages that will fail your WDAC tests even though they claim not to need compilation, containing pak and others like it.
The NeedsCompilation field is included as a column of the result of available.packages(), so it is very easy to access without trying to install the package.
I have accepted the answer from user2554330 as the best solution. It makes use of the normal set of commands used for package management; and the matrix generated by available.packages() can be passed to tools::package_dependencies(), removing the need for multiple internet queries.
For completeness I am documenting another possible solution. A script could query the unofficial CRAN Github mirror https://docs.r-hub.io/#cranatgh and look for a /src directory in each package project.
Related
I know there are plenty of ways to install packages but I obtained a zipped folder containing lots of folders with r packages. Is that ok to simply unzip the folder, copy all the folders and paste them to R-3.6.1\library location? Will that work properly?
You're talking about the installed package folders that were built and installed on another computer?
You'd want to be sure they are from the same operating system and version of R.
It also depends if there are any unmet or potentially conflicting dependencies. If not, in theory it should work.
But you'll then get notified of package updates. So it will only save you time installing them in the first place.
You could always switch between the package libs location when starting RStudio, by setting the "R_LIBS_USER" environment variable. Then update those packages (and get a sense of how safe it is), and you'll probably run into less chance of issues when you soft-copy them across to your primary location.
And please backup the primary location first if you need to restore it to that point.
I have been using git for a while but just recently started using packrat. I would like my repository to be self contained but at the same time I do not want to include CRAN packages as they are available. It seems once R is opened in a project with packrat it will try to use packages from project library; if they are not available then it will try to install from src in the project library; if they are not available it will look at libraries installed in that computer. If a library is not available in the computer; would it look at CRAN next?
What files should I include in my git repo as a minimum (e.g., packrat.lock)?
You can choose to set an external CRAN-like repository with the source tarballs of the packages and their versions that you'd like available for your project. The default behaviour, however, is to look to CRAN next, as you've identified in your question. Check out the packrat.lock file, you will see that for each package you use in packrat, there is an option called source: CRAN (if you've downloaded the file from CRAN, that is).
When you have a locally stored package source file, the contents of the lockout for said package change to the following:
Package: FooPackage
Source: source
Version: 0.4-4
Hash: 44foo4036fb68e9foo9027048d28
SourcePath: /Users/MyName/Documents/code/myrepo/RNetica
I'm a bit unclear on your final question: What files should I include in my git repo as a minimum (e.g., packrat.lock)? But I'm going to take this as a) combination of what files should be present for packrat to run, and b) which of those files should be committed to the git-repo. To answer the first question, I illustrate with initialising packrat on an existing R project.
When you run packrat::init(), two important things happen (among others):
1. All the packrat scaffolding, including source tarballs etc are created under: PackageName/packrat/.
2. packrat/lib*/ is added to your .gitignore file.
So from this, we can see that anything under packrat/lib*/ doesn't need to be committed to your git-repo. This leaves the following 3 files to be committed:
packrat/init.R
packrat/packrat.lock
packrat/packrat.opts
packrat.lock is needed for collaborating with others through a version control system; it helps keep your private libraries in sync. packrat.opts allows you to specify different project specific options for packrat. The file is automatically generated using get_opts and set_opts. Committing this file to the git-repo will ensure that any options you specify are maintained for all collaborators. A final file to be committed to the repo is .Rprofile. This file tells R to use the private package library (when R is started from the project directory).
Depending on your needs, you can choose to commit the source tar balls to the repository, or not. If you don't want them available in your git-repo, you simply add packrat/src/ to the .gitignore. But, this will mean that anyone accessing the git-repo will not have access to the package source code, and the files will be downloaded from CRAN, or from wherever the source line dictates within the packrat.lock file.
From your question, it sounds like committing the packrat/src/ folder contents to your repo might be what you need.
I'm at a loss as to how to install the RBloomberg package. The only source for the files seems to be GitHub. The offered zip file is called blpwrapper-master.zip which embeds an rbloomberg folder. When I try to install the zip file (in RStudio), I get an error message that it cannot open a compressed file. I rezipped just the rbloomberg folder, but that led nowhere either. How does one go about this?
In general things can be installed from Github using the devtools package. For example:
library("devtools")
install_github("username/packagename")
I don't know who authors so Rbloomberg, but you can swap in the appropriate Github username in the above.
Note: Sometimes this won't work because a developer uses a non-traditional .git directory structure, but it should work in most cases. Indeed, that is the case here (as #rawr points out), where you need to use a slightly different path to point to package (which is in a subdirectory of the git repo):
install_github("johnlaing/blpwrapper/rbloomberg")
Question: How do I move all of the most up-to-date R packages into one simple location that R (and everything else) will use from now and forever for my packages?
I have been playing around with R on Ubuntu 10.04 using variously RGedit, RCmdr, R shell, and RStudio. Meanwhile, I have installed packages, updated packages, and re-updated packages via apt, synaptic, install.packages(), etc... which apparently means these packages get placed everywhere, and (with the occasional sudo tossed in) with different permissions.
Currently I have different versions of different (and repeated) packages in:
/home/me/R/i486-pc-linux-gnu-library/2.10
/home/me/R/i486-pc-linux-gnu-library/2.14
/home/me/R/i486-pc-linux-gnu-library/
/usr/local/lib/R/site-library
/usr/lib/R/site-library
/usr/lib/R/library
First - I'm a single user, on a single machine - I don't want multiple library locations, I just want it to work.
Second - I am on an extremely slow connection, and can't keep just downloading packages repeatedly.
So - is there an easy way to merge all these library locations into one simple location? Can I just copy the folders over?
How do I set it in concrete that this is and always will be where anything R related looks for and updates packages?
This is maddening.
Thanks for your help.
Yes, it should almost work to just copy the folders over. But pre-2.14 packages WITHOUT a NAMESPACE file probably won't work in R 2.14 where all packages must have a namespace...
And you'd want to manually ensure you only copy the latest version of each package if you have multiple versions...
If you type .libPaths(), it will tell you where R looks for packages. The first in the list is where new packages are typically installed. I suspect that .libPaths() might return different things from RStudio vs. Rcmd etc.
After piecing together various bits of info here goes: A complete moron's guide to the R packages directory organization:
NB1 - this is my experience with Ubuntu - your mileage may vary
NB2 - I'm a single user on a single machine, and I like things simple.
Ubuntu puts anything installed via apt, or synaptic in:
/usr/lib/R/site-library
/usr/lib/R/library
directories. The default vanilla R install will try install packages here:
/usr/local/lib/R/site-library
Since these are system directories the user does not have write privileges to, depending on what method you are interacting with R you might be prompted with a friendly - "Hey buddy - we can't write there, you want us to put your packages in your home directory?" which seems innocent and reasonable enough... assuming you never change your GUI, or your working environment. If you do, the new GUI / environment might not be looking in the directory where the packages were placed, so won't find them. (Most interfaces have a way for you to point where your personal library of packages is, but who wants to muck about in config files?)
What seems to be the best practice for me (and feel free to correct me if I'm wrong) with a default install setup on Ubuntu, is to do any package management from a basic R shell as sudo: > sudo R and from there do your install.packages() voodoo. This seems to put packages in the usr/local/lib/R/site-library directory.
At the same time, update.packages() will update the files in /usr/lib/R/site-library and usr/lib/R/library directories, as well as usr/local/lib/R/site-library
(As for usr/lib/R/ division, it looks like /library/ has the core packages, while /site-library/ holds anything added, assuming they were installed by apt...)
Any packages previously installed and in the wrong place can be moved to the /usr/local/lib/R/site-library directory (assuming you are sudoing it) just by moving the directories (thanks #Tommy), but as usr/lib/R/ is controlled by apt - best not add or subtract anything from there...
Whew. Anyway - simple enough, and in simple language. Thanks everyone for the help.
Does R have a dependency management tool to facilitate project-specific dependencies? I'm looking for something akin to Java's maven, Ruby's bundler, Python's virtualenv, Node's npm, etc.
I'm aware of the "Depends" clause in the DESCRIPTION file, as well as the R_LIBS facility, but these don't seem to work in concert to provide a solution to some very common workflows.
I'd essentially like to be able to check out a project and run a single command to build and test the project. The command should install any required packages into a project-specific library without affecting the global R installation. E.g.:
my_project/.Rlibs/*
Unfortunately, Depends: within the DESCRIPTION: file is all you get for the following reasons:
R itself is reasonably cross-platform, but that means we need this to work across platforms and OSs
Encoding Depends: beyond R packages requires encoding the Depends in a portable manner across operating systems---good luck encoding even something simple such as 'a PNG graphics library' in a way that can be resolved unambiguously across systems
Windows does not have a package manager
AFAIK OS X does not have a package manager that mixes what Apple ships and what other Open Source projects provide
Even among Linux distributions, you do not get consistency: just take RStudio as an example which comes in two packages (which all provide their dependencies!) for RedHat/Fedora and Debian/Ubuntu
This is a hard problem.
The packrat package is precisely meant to achieve the following:
install any required packages into a project-specific library without affecting the global R installation
It allows installing different versions of the same packages in different project-local package libraries.
I am adding this answer even though this question is 5 years old, because this solution apparently didn't exist yet at the time the question was asked (as far as I can tell, packrat first appeared on CRAN in 2014).
Update (November 2019)
The new R package renv replaced packrat.
As a stop-gap, I've written a new rbundler package. It installs project dependencies into a project-specific subdirectory (e.g. <PROJECT>/.Rbundle), allowing the user to avoid using global libraries.
rbundler on Github
rbundler on CRAN
We've been using rbundler at Opower for a few months now and have seen a huge improvement in developer workflow, testability, and maintainability of internal packages. Combined with our internal package repository, we have been able to stabilize development of a dozen or so packages for use in production applications.
A common workflow:
Check out a project from github
cd into the project directory
Fire up R
From the R console:
library(rbundler)
bundle('.')
All dependencies will be installed into ./.Rbundle, and an .Renviron file will be created with the following contents:
R_LIBS_USER='.Rbundle'
Any R operations run from within this project directory will adhere to the project-speciic library and package dependencies. Note that, while this method uses the package DESCRIPTION to define dependencies, it needn't have an actual package structure. Thus, rbundler becomes a general tool for managing an R project, whether it be a simple script or a full-blown package.
You could use the following workflow:
1) create a script file, which contains everything you want to setup and store it in your projectd directory as e.g. projectInit.R
2) source this script from your .Rprofile (or any other file executed by R at startup) with a try statement
try(source("./projectInit.R"), silent=TRUE)
This will guarantee that even when no projectInit.R is found, R starts without error message
3) if you start R in your project directory, the projectInit.R file will be sourced if present in the directory and you are ready to go
This is from a Linux perspective, but should work in the same way under windows and Mac as well.