My R script has evolved over many months with many additions and subtractions. It is long and rambling and I would like to find out which packages I am actually using in the code so I can start deleting library() references. Is there a way of finding redundant dependencies in my R script?
I saw this question so I tried:
library(mvbutils)
library(MyPackage)
library(dplyr)
foodweb( funs=find.funs( asNamespace( 'EndoMineR')), where=
asNamespace( 'EndoMineR'), prune='filter')
But that really tells me where I am using a function from a package whereas I don't necessarily remember which functions I have used from which package.
I tried packrat but this is looking for projects whereas mine is a directory of scripts I am trying to build into a package.
To do this you first parse your file and then use getAnywhere to check for each terminal token in which namespace it is defined. Compare the result with the searchpath and you will have the answer. I compiled a function that takes a filename as argument and returns the packages which are used in the file. Note that it can only find packages that are loaded when the function is executed, so be sure to load all "candidates" first.
findUsedPackages <- function(sourcefile) {
## get parse tree
parse_data <- getParseData(parse(sourcefile))
## extract terminal tokens
terminal_tokens <- parse_data[parse_data$terminal == TRUE, "text"]
## get loaded packages/namespaces
search_path <- search()
## helper function to find the package a token belongs to
findPackage <- function(token) {
## get info where the token is defined
token_info <- getAnywhere(token)
##return the package the comes first in the search path
token_info$where[which.min(match(token_info$where, search_path))]
}
packages <- lapply(unique(terminal_tokens), findPackage)
##remove elements that do not belong to a loaded namespace
packages <- packages[sapply(packages, length) > 0]
packages <- do.call(c, packages)
packages <- unique(packages)
##do not return base and .GlobalEnv
packages[-which(packages %in% c("package:base", ".GlobalEnv"))]
}
Related
Criteria for answer to this question
Given the following function (within its own script)
# something.R
hello <- function(x){
paste0("hello ", x)
}
What is the most minimal amount of setup which will enable the following
library(something)
x <- something::hello('Sue')
# x now has value: "hello Sue"
Context
In python it's very simple to have a directory containing some code, and utilise it as
# here foo is a directory
from foo import bar
bar( ... )
I'm not sure how to do something similar in R though.
I'm aware there's source(file.R), but this puts everything into the global namespace. I'm also aware that there's library(package) which provides package::function. What I'm not sure about is whether there's a simple approach to using this namespacing within R. The packaging tutorials that I've searched for seem to be quite involved (in comparison to Python).
I don't know if there is a real benefit in creating a namespace just for one quick function. It is just not the way it is supposed to be (I think).
But anyway here is a rather minimalistic solution:
First install once: install.packages("namespace")
The function you wanted to call in the namespace:
hello <- function(x){
paste0("hello ", x)
}
Creating your namespace, assigning the function and exporting
ns <- namespace::makeNamespace("newspace")
assign("hello",hello ,env = ns)
base::namespaceExport(ns, ls(ns))
Now you can call your function with your new namespace
newspace::hello("you")
Here's the quickest workflow I know to produce a package, using RStudio. The default package already contains a hello function, that I overwrote with your code.
Notice there was also a box "create package based on source files", which I didn't use but you might.
A package done this way will contain exported undocumented untested functions.
If you want to learn how to document, export or not, write tests and run checks, include other objects than functions, include compiled code, share on github, share on CRAN.. This book describes the workflow used by thousands of users, and is designed so you can usually read sections independently.
If you don't want to do it from GUI you can useutils::package.skeleton() to build a package folder, and remotes::install_local() to install it :
Reproducible setup
# create a file containing function definition
# where your current function is located
function_path <- tempfile(fileext = ".R")
cat('
hello <- function(x){
paste0("hello ", x)
}
', file = function_path)
# where you store your package code
package_path <- tempdir()
Solution :
# create package directory at given location
package.skeleton("something", code_file = file_path, path = package_path)
# remove sample doc to make remotes::install_local happy
unlink(file.path(package_path, "something", "man/"), TRUE)
# install package
remotes::install_local(file.path(package_path, "something"))
I have written a function to get the name and version of all of my loaded packages:
my_lib <- function(){
tmp <- (.packages())
tmp_base <- sessionInfo()$basePkgs
tmp <- setdiff(tmp, tmp_base)
tmp <- sort(tmp)
tmp <- sapply(tmp, function(x){
x <- paste(x, utils::packageVersion(x), sep = ' v')
})
tmp <- paste(tmp, collapse=', ')
return(tmp)
}
This also returns all packages loaded as dependencies to other packages (eg I load car and carData is loaded as a dependency).
I am wondering if there is a way to only return the packages I manually loaded (eg just car)? Can R tell the difference between manually loaded vs loaded as a dependency?
Edit:
Added line to remove base packages using sessionInfo()
R has a subtle difference between a loaded package and an attached package.
A package is attached when you use the library function,
and it makes its exported functions "visible" to the user's global environment.
If a package is attached,
its namespace has been loaded,
but the opposite is not necessarily true.
Each package can define two main types of dependencies: Depends and Imports.
The packages in the former get attached as soon as the dependent package is attached,
but the packages in the latter only get loaded.
This means you can't completely differentiate,
because you may call library for a specific package,
but any packages it Depends on will also be attached.
Nevertheless, you can differentiate between loaded and attached packages with loadedNamespaces() and search().
EDIT: It just occurred to me that if you want to track usage of library
(ignoring require),
you could write a custom tracker:
library_tracker <- with(new.env(), {
packages <- character()
function(flag) {
if (missing(flag)) {
packages <<- union(packages, as.character(substitute(package, parent.frame())))
}
packages
}
})
trace("library", library_tracker, print = FALSE)
library("dplyr")
library(data.table)
# retrieve packages loaded so far
library_tracker(TRUE)
[1] "dplyr" "data.table"
The flag parameter is just used to distinguish between calls made by trace,
which call the function without parameters,
and those made outside of it,
in order to easily retrieve packages loaded so far.
You could also use environment(library_tracker)$packages.
Occasionally one wants to patch a function in a package, without recompiling the whole package.
For example, in Emacs ESS, the function install.packages() might get stuck if tcltk is not loaded. One might want to patch install.packages() in order to require tcltk before installation and unload it after the package setup.
A temp() patched version of install.packages() might be:
## Get original args without ending NULL
temp=rev(rev(deparse(args(install.packages)))[-1])
temp=paste(paste(temp, collapse="\n"),
## Add code to load tcltk
"{",
" wasloaded= 'package:tcltk' %in% search()",
" require(tcltk)",
## Add orginal body without braces
paste(rev(rev(deparse(body(install.packages))[-1])[-1]), collapse="\n"),
## Unload tcltk if it was not loaded before by user
" if(!wasloaded) detach('package:tcltk', unload=TRUE)",
"}\n",
sep="\n")
## Eval patched function
temp=eval(parse(text=temp))
# temp
Now we want to replace the original install.packages() and perhaps insert the code in Rprofile.
To this end it is worth nothing that:
getAnywhere("install.packages")
# A single object matching 'install.packages' was found
# It was found in the following places
# package:utils
# namespace:utils
# with value
#
# ... install.packages() source follows (quite lengthy)
That is, the function is stored inside the package/namespace of utils. This environment is sealed and therefore install.packages() should be unlocked before being replaced:
## Override original function
unlockBinding("install.packages", as.environment("package:utils"))
assign("install.packages", temp, envir=as.environment("package:utils"))
unlockBinding("install.packages", asNamespace("utils"))
assign("install.packages", temp, envir=asNamespace("utils"))
rm(temp)
Using getAnywhere() again, we get:
getAnywhere("install.packages")
# A single object matching 'install.packages' was found
# It was found in the following places
# package:utils
# namespace:utils
# with value
#
# ... the *new* install.packages() source follows
It seems that the patched function is placed in the right place.
Unfortunately, running it gives:
Error in install.packages(xxxxx) :
could not find function "getDependencies"
getDependencies() is a function inside the same utils package, but not exported; therefore it is not accessible outside its namespace.
Despite the output of getAnywhere("install.packages"), the patched install.packages() is still misplaced.
The problem is that we need to reload the utils library to obtain the desired effect, which also requires unloading other libraries importing it.
detach("package:stats", unload=TRUE)
detach("package:graphics", unload=TRUE)
detach("package:grDevices", unload=TRUE)
detach("package:utils", unload=TRUE)
library(utils)
install.packages() works now.
Of course, we need to reload the other libraries too. Given the dependencies, using
library(stats)
should reload everything. But there is a problem when reloading the graphics library, at least on Windows:
library(graphics)
# Error in FUN(X[[i]], ...) :
# no such symbol C_contour in package path/to/library/graphics/libs/x64/graphics.dll
Which is the correct way of (re)loading the graphics library?
Patching functions in packages is a low-level operation that should be avoided, because it may break internal assumptions of the execution environment and lead to unpredictable behavior/crashes. If there is a problem with tck/ESS (I didn't try to repeat that) perhaps it should be fixed or there may be a workaround. Particularly changing locked bindings is something to avoid.
If you really wanted to run some code at the start/end of say install.packages, you can use trace. It will do some of the low-level operations mentioned in the question, but the good part is you don't have to worry about fixing this whenever some new internals of R change.
trace(install.packages,
tracer=quote(cat("Starting install.packages\n")),
exit=quote(cat("Ending install packages.\n"))
)
Replace tracer and exit accordingly - maybe exit is not needed anyway, maybe you don't need to unload the package. Still, trace is a very useful tool for debugging.
I am not sure if that will solve your problem - if it would work with ESS - but in general you can also wrap install.packages in a function you define say in your workspace:
install.packages <- function(...) {
cat("Entry.\n")
on.exit(cat("Exit.\n"))
utils::install.packages(...)
}
This is the cleanest option indeed.
I wanted to try some new package. I installed it, it required a lot of dependencies, so it installed plenty of other packages. I tried it and I am not impressed - now I would like to uninstall that package including all the dependencies!
Is there any way to remove given packages including all dependencies which are not needed by any other package in the system?
I looked at ?remove.packages but there is no option to do this.
Here is some code that will all you to remove a package and its unneeded dependencies. Note that its interpretation of "unneeded" dependent packages is the set of packages that this package depends on but that are not used in any other package. This means that it will also default to suggesting to uninstall packages that have no reverse dependencies. Thus I've implemented it as an interactive menu (like in update.packages) to give you control over what to uninstall.
library("tools")
removeDepends <- function(pkg, recursive = FALSE){
d <- package_dependencies(,installed.packages(), recursive = recursive)
depends <- if(!is.null(d[[pkg]])) d[[pkg]] else character()
needed <- unique(unlist(d[!names(d) %in% c(pkg,depends)]))
toRemove <- depends[!depends %in% needed]
if(length(toRemove)){
toRemove <- select.list(c(pkg,sort(toRemove)), multiple = TRUE,
title = "Select packages to remove")
remove.packages(toRemove)
return(toRemove)
} else {
invisible(character())
}
}
# Example
install.packages("YplantQMC") # installs an unneeded dependency "LeafAngle"
c("YplantQMC","LeafAngle") %in% installed.packages()[,1]
## [1] TRUE TRUE
removeDepends("YplantQMC")
c("YplantQMC","LeafAngle") %in% installed.packages()[,1]
## [1] FALSE FALSE
Note: The recursive option may be particularly useful. If package dependencies further depend on other unneeded packages, setting recursive = TRUE is vital. If dependencies are shallow (i.e., only one level down the dependency tree), this can be left as FALSE (the default).
There is in fact a function remove.packages() in base R, but it's in the package utils, which you need to load first:
library(utils)
remove.packages()
It's not entirely clear to me how much recursive cleanup this function does.
There are base R ways to handle this but I'm going to recommend a package (I know you're trying to get rid of these). I'm recommending this package for 2 reasons (1) it solves two problems you're having & (2) Dason K. and I are developing this package (full disclosure). This package's value stands in that the functions are easier to remember names that are consistent. It also does some combined operations. Note you could do all of this in base but this question is already pretty localized so thus I'm going to use a tool that makes answering easier.
This package will:
allow you to delete package and dependencies
allow you to install packages in a temporary directory rather than main library
The caveat is that you can't be 100% certain that the package dependency wasn't already there, installed by the user previously. Therefore I would take caution with every step of this solution that you're not deleting things that are of importance. This solution relies on 2 factors (1) pacman (2) file.info. We'll assume that dependencies that were modified within a certain (user defined) threshold of time are indeed unwanted packages. Note the word assume here.
I made this reproducible for the folks at home in that the answer will randomly install a package from CRAN with additional dependencies (this installs a package you do not already have locally with 3 or more dependencies; used random to not single out any package).
Making a reproducible example
library(pacman)
(available <- p_cran())
(randoms <- setdiff(available, p_lib()))
(mypackages <- p_lib())
ndeps <- 1
while(ndeps < 3) {
package <- sample(randoms, 1)
deps <- unlist(p_depends(package, character.only=TRUE), use.names=FALSE)
ndeps <- length(setdiff(deps, mypackages))
}
package
p_install(package, character.only = TRUE)
Uninstalling package
We will assign the package name from the first part to package or the OP can use the unwanted package they installed and assign that to package (my random package happened to be package <- "OrdinalLogisticBiplot"). This deletions process should, ideally, be done in a clean R session with no add-on packages (except pacman) loaded.
## function to grab file info date/time modified
infograb <- function(x) file.info(file.path(p_path(), x))[["mtime"]]
## determine the differences in times modified for "package"
## and all other packages in library
diffs <- as.numeric(infograb(package)) - sapply(p_lib(), infograb)
## user defined threshold
threshold <- 15
## determine packages just installed within the time frame of the unwanted package
(delete_deps <- diffs[diffs < threshold & diffs >= 0])
## recursively find all packages that could have been installed
potential_depends <- unlist(lapply(unlist(p_depends(package, character=TRUE)),
p_depends, character=TRUE, recursive=TRUE))
## delete packages that are both on the lists of (1) installed within time
## frame of unwanted package and a dependency of that package
p_delete(intersect(names(delete_deps), potential_depends), character.only = TRUE)
This approach makes some big assumptions.
A better approach from the get go
p_temp(package_to_try)
This allows you to try it out first and not have it muddy your local library.
If you're unimpressed with pacman you can use the method described above to delete it.
Here is a quick solution to have a look at the packages that are not required by any other locally installed packages.
library(pacman)
ip <- installed.packages()[,1]
deps <- lapply(1:length(ip), function(i) tryCatch(p_depends(ip[i], local = TRUE)$Imports, error = function(e) return(NULL)))
packages.on.which.things.depend <- sort(unique(unlist(deps)))
packages.on.which.nothing.depends <- setdiff(ip, packages.on.which.things.depend)
packages.on.which.nothing.depends
Then, have a quick look at it and remove the ones you are not using (just be careful and do not try to uninstall base).
After you have determined which ones you use and which you don’t use, you may proceed with something like
i.need <- c("AER", "car", "devtools", "glmnet", "gmm", "Hmisc", "pacman", "plm", "RcppArmadillo", "RcppEigen", "rmarkdown", "rugarch", "base", "datasets")
un <- setdiff(packages.on.which.nothing.depends, i.need)
un
remove.packages(un)
Rinse and repeat until there are no unneeded orphanes packages. R should not allow you to remove built-in system packages.
I have a package that I wrote while learning R and its dependency list is quite long. I'm trying to trim it down, for two cases:
I switched to other approaches, and packages listed in Suggests simply aren't used at all.
Only one function out of my whole package relies on a given dependency, and I'd like to switch to an approach where it is loaded only when needed.
Is there an automated way to track down these two cases? I can think of two crude approaches (download the list of functions in all the dependent packages and automate a text search for them through my package's code, or load the package functions without loading the required packages and execute until there's an error), but neither seems particularly elegant or foolproof....
One way to check dependancies in all functions is to use the byte compiler because that will check for functions being available in the global workspace and issue a notice if it does not find said function.
So if you as an example use the na.locf function from the zoo package in any of your functions and then byte compile your function you will get a message like this:
Note: no visible global function definition for 'na.locf'
To correctly address it for byte compiling you would have to write it as zoo::na.locf
So a quick way to test all R functions in a library/package you could do something like this (assuming you didn't write the calls to other functions with the namespace):
Assuming your R files with the functions are in C:\SomeLibrary\ or subfolders there of and then you define a sourceing file as C:\SomeLibrary.r or similar containing:
if (!(as.numeric(R.Version()$major) >=2 && as.numeric(R.Version()$minor) >= 14.0)) {
stop("SomeLibrary needs version 2.14.0 or greater.")
}
if ("SomeLibrary" %in% search()) {
detach("SomeLibrary")
}
currentlyInWorkspace <- ls()
SomeLibrary <- new.env(parent=globalenv())
require("compiler",quietly=TRUE)
pathToLoad <- "C:/SomeLibraryFiles"
filesToSource <- file.path(pathToLoad,dir(pathToLoad,recursive=TRUE)[grepl(".*[\\.R|\\.r].*",dir(pathToLoad,recursive=TRUE))])
for (filename in filesToSource) {
tryCatch({
suppressWarnings(sys.source(filename, envir=SomeLibrary))
},error=function(ex) {
cat("Failed to source: ",filename,"\n")
print(ex)
})
}
for(SomeLibraryFunction in ls(SomeLibrary)) {
if (class(get(SomeLibraryFunction,envir=SomeLibrary))=="function") {
outText <- capture.output(with(SomeLibrary,assign(SomeLibraryFunction,cmpfun(get(SomeLibraryFunction)))))
if(length(outText)>0){
cat("The function ",SomeLibraryFunction," produced the following compile note(s):\n")
cat(outText,sep="\n")
cat("\n")
}
}
}
attach(SomeLibrary)
rm(list=ls()[!ls() %in% currentlyInWorkspace])
invisible(gc(verbose=FALSE,reset=TRUE))
Then start up R with no preloaded packages and source in C:\SomeLibrary.r
And then you should get notes from cmpfun for any call to a function in a package that's not part of the base packages and doesn't have a fully qualified namespace defined.