We have recently got RStudio Connect in my office. For our work, we have made custom packages, which we have updated amongst ourselves by opening the project and build+reloading.
I understand the only way I can get our custom packages to work within apps with RSConnect is to get up a local repo and set our options(repos) to include this.
Currently I have the following:
library(drat)
RepoAddress <- "C:/<RepoPath>" # High level path
drat::insertPackage(<sourcePackagePath>, repodir = RepoAddress)
# Add this new repo to Rs knowledge of repos.
options(repos = c(options("repos")$repos,LocalCurrent = paste0("file:",RepoAddress)))
# Install <PackageName> from the local repo :)
install.packages("<PackageName>")
Currently this works nicely and I can install my custom package from the local repo. This indicates to me that the local repo is set up correctly.
As an additional aside, I have changed the DESCRIPTION file to have an extra line saying repository:LocalCurrent.
However when I try to deploy a Shiny app or Rmd which references , I get the following error on my deploy:
Error in findLocalRepoForPkg(pkg, repos, fatal = fatal) :
No package '<PackageName> 'found in local repositories specified
I understand this is a problem with packrat being unable to find my local repos during the deploy process (I believe at a stage where it uses packrat::snapshot()).This is confusing since I would have thought packrat would use my option("repos") repos similar to install.packages. If I follow through the functions, I can see the particular point of failure is packrat:::findLocalRepoForPkg("<PackageName", repos = packrat::get_opts("local.repos")), which fails even after I define packrat::set_opts("local.repos" = c(CurrentRepo2 = paste0("file:",RepoAddress)))
If I drill into packrat:::findLocalRepoForPkg, it fails because it can't find a file/folder called: "C://". I would have thought this is guaranteed to fail, because repos follow the C://bin/windows/contrib/3.3/ structure. At no point would a repo have the structure it's looking for?
I think this last part is showing I'm materially misunderstanding something. Any guidance on configuring my repo so packrat can understand it would be great.
One should always check what options RStudio connect supports at the moment:
https://docs.rstudio.com/connect/admin/r/package-management/#private-packages
Personally I dislike all options for including local/private packages, as it defeats the purpose of having a nice easy target for deploying shiny apps. In many cases, I can't just set up local repositories in the organization because, I do not have clearance for that. It is also inconvenient that I have to email IT-support to make them manually install new packages. Overall I think RS connect is great product because it is simple, but when it comes to local packages it is really not.
I found a nice alternative/Hack to Rstudio official recommendations. I suppose thise would also work with shinyapps.io, but I have not tried. The solution goes like:
add to global.R if(!require(local_package) ) devtools::load_all("./local_package")
Write a script that copies all your source files, such that you get a shinyapp with a source directory for a local package inside, you could call the directory for ./inst/shinyconnect/ or what ever and local package would be copied to ./inst/shinyconnect/local_package
manifest.
add script ./shinyconnect/packrat_sees_these_dependencies.R to shiny folder, this will be picked up by packrat-manifest
Hack rsconnet/packrat to ignore specifically named packages when building
(1)
#start of global.R...
#load more packages for shiny
library(devtools) #need for load_all
library(shiny)
library(htmltools) #or what ever you need
#load already built local_package or for shiny connection pseudo-build on-the-fly and load
if(!require(local_package)) {
#if local_package here, just build on 2 sec with devtools::load_all()
if(file.exists("./DESCRIPTION")) load_all(".") #for local test on PC/Mac, where the shinyapp is inside the local_package
if(file.exists("./local_package/DESCRIPTION")) load_all("./local_package/") #for shiny conenct where local_package is inside shinyapp
}
library(local_package) #now local_package must load
(3)
make script loading all the dependencies of your local package. Packrat will see this. The script will never be actually be executed. Place it at ./shinyconnect/packrat_sees_these_dependencies.R
#these codelines will be recognized by packrat and package will be added to manifest
library(randomForest)
library(MASS)
library(whateverpackageyouneed)
(4) During deployment, manifest generator (packrat) will ignore the existence of any named local_package. This is an option in packrat, but rsconnect does not expose this option. A hack is to load rsconnect to memory and and modify the sub-sub-sub-function performPackratSnapshot() to allow this. In script below, I do that and deploy a shiny app.
library(rsconnect)
orig_fun = getFromNamespace("performPackratSnapshot", pos="package:rsconnect")
#packages you want include manually, and packrat to ignore
ignored_packages = c("local_package")
#highjack rsconnect
local({
assignInNamespace("performPackratSnapshot",value = function (bundleDir, verbose = FALSE) {
owd <- getwd()
on.exit(setwd(owd), add = TRUE)
setwd(bundleDir)
srp <- packrat::opts$snapshot.recommended.packages()
packrat::opts$snapshot.recommended.packages(TRUE, persist = FALSE)
packrat::opts$ignored.packages(get("ignored_packages",envir = .GlobalEnv)) #ignoreing packages mentioned here
print("ignoring following packages")
print(get("ignored_packages",envir = .GlobalEnv))
on.exit(packrat::opts$snapshot.recommended.packages(srp,persist = FALSE), add = TRUE)
packages <- c("BiocManager", "BiocInstaller")
for (package in packages) {
if (length(find.package(package, quiet = TRUE))) {
requireNamespace(package, quietly = TRUE)
break
}
}
suppressMessages(packrat::.snapshotImpl(project = bundleDir,
snapshot.sources = FALSE, fallback.ok = TRUE, verbose = FALSE,
implicit.packrat.dependency = FALSE))
TRUE
},
pos = "package:rsconnect"
)},
envir = as.environment("package:rsconnect")
)
new_fun = getFromNamespace("performPackratSnapshot", pos="package:rsconnect")
rsconnect::deployApp(appDir="./inst/shinyconnect/",appName ="shinyapp_name",logLevel = "verbose",forceUpdate = TRUE)
The problem is one of nomenclature.
I have set up a repo in the CRAN sense. It works fine and is OK. When packrat references a local repo, it is referring to a local git-style repo.
This solves why findlocalrepoforpkg doesn't look like it will work - it is designed to work with a different kind of repo.
Feel free to also reach out to support#rstudio.com
I believe the local package code path is triggered in packrat because of the missing Repository: value line in the Description file of the package. You mentioned you added this line, could you try the case-sensitive version?
That said, RStudio Connect will not be able to install the package from the RepoAddress as you've specified it (hardcoded on the Windows share). We'd recommend hosting your repo over https from a server that both your Dev environment and RStudio Connect have access to. To make this type of repo setup much easier we just released RStudio Package Manager which you (and IT!) may find a compelling alternative to manually managing releases of your internal packages via drat.
Related
This bounty has ended. Answers to this question are eligible for a +100 reputation bounty. Bounty grace period ends in 4 hours.
antonio wants to draw more attention to this question:
Explain what is the proper workflow when you are building two packages, where one depends on the other. What is the correct way to run devtools::check() on the depending-on package?
I want to add a local package dependency using R devtools.
The suggested way to add packages to the package DESCRIPTION file is one of the two functions use_package() and use_dev_package(), from the usethis package. The latter adds a dependency on an in-development package. The function help shows the following prototype:
use_dev_package(package, type = "Imports", remote = NULL)
where remote is
a character string to specify the remote, e.g. ‘"gitlab::jimhester/covr"’, using any syntax supported by the remotes package.
The remotes vignette shows the following
# Local
Remotes: local::/pkgs/testthat
So the command should be along these lines:
use_dev_package(foopack, type = "Imports", remote = "local::<foopack>")
However, what should be the path to the foopack. An absolute one or relative to project dir? The root package directory or the R directory with the code, or perhaps the foopack.tar.gz build?
All attempts failed for me.
Needless to say that, beyond having the local dependency properly listed in the DESCRIPTION file, I need it to be seen by the devtools build & check functions.
Edit
As regards use_dev_package(), I found a solution:
if I use devtools::check(), then the dependency appears in the search path, and use_dev_package() does not complain any more (see answer below).
However, it is still unclear to me what arguments should I use to make a development check() for the main package, in particular when the package has a vignette.
Ideally, I should be able to pass the check with local dependencies by passing cran = FALSE, but this still gives
"Package required but not available".
It seems that I have to check the local dependencies before adding them to the description file.
devtools::check("path/to/foopack")
usethis::use_dev_package("foopack", remote ="local::path/to/foopack")
The paths can be relative or absolute, and even a single colon works.
It might be worth noting that, when I build the main package, I can use the ordinary:
devtools::build()
but, for a successful check, I need to use the remote argument:
devtools::check(remote = TRUE)
I can't see a rationale for restating what is in the DESCRIPTION file, but I do not have enough expertise to say it's a bug.
Let's see what the others say in this regard.
Edit
Unfortunately, it seems that the remote argument above does not apply to vignettes. So, if I add a vignette to the package, checks fail with local packages
Until an actual solution is found, all I can do is (sadly) to ignore vignette checks:
devtools::check(remote = TRUE, vignettes = FALSE)
Summary
When creating a package, I can list CRAN dependencies in the Depends list in the DESCRIPTION file. This documentation outlines how to list Bitbucket dependencies, eg. Remotes: bitbucket::sulab/mygene.r#default.
However, I don't know how to do this when authentication is needed to access the repository.
Attempt
I've tried putting the following code into the main packagename.R file. The function contents work fine as a snippet at the top of a standalone script:
.onLoad <- function(libname, pkgname) {
otherPackageVersion <- "1.0"
if (suppressWarnings(suppressPackageStartupMessages(require("otherPackageName", quietly = TRUE, character.only = TRUE)))) {
if (installed.packages()[installed.packages()[,"Package"] == "otherPackageName", "Version"] != otherPackageVersion) {
remove.packages("otherPackage")
devtools::install_bitbucket(sprintf("bitbucketUser/otherPackageName#%s", otherPackageVersion), auth_token = Sys.getenv("BITBUCKET_PAT"))
}
} else {
devtools::install_bitbucket(sprintf("bitbucketUser/otherPackageName#%s", otherPackageVersion), auth_token = Sys.getenv("BITBUCKET_PAT"))
}
}
but R CMD check fails saying it cannot be loaded after hanging for a while:
checking whether package ‘packageName’ can be installed ... ERROR
Installation failed.
Further Detail
The version of devtools I have loaded is 1.12.0.9000 (see this Github thread) which I installed using devtools::install_github("hadley/devtools#1220"). This allows me to install private Bitbucket R packages using an App Password stored in an environment variable, rather than committing my username/password in plaintext.
This will not be possible until this (a pull request using Bitbucket PATs) is merged into the devtools package.
EDIT: Checking in on this many years later, it's sorted for me by the current version of devtools (2.4.3) using a BitBucket App Password with Repo Read/Write and Project Read/Write permissions.
I want to setup local repository for R package, I'd like the repository works like sonatype nexus(it can proxy the central repository, and cache the artifacts after downloading the artifact from central repository).
Currently nexus does not support R repository format, so it doesn't suite what I needed to do.
Is that any existing solution for creating this repository? I don't want to create a CRAN mirror, which is too heavy for me.
First, you'll want to make sure you have the following path and its directories in your system: "/R/src/contrib". If you don't have this path and these directories, you'll need to create them. All of your R packages files will be stored in the "contrib" directory.
Once you've added package files to the "contrib" directory, you can use the setRepositories function from the utils package to create the repository. I'd recommend adding the following code to your .Rprofile for a local repository:
utils::setRepositories(ind = 0, addURLs = c(WORK = "file://<your higher directories>/R"))
After editing your .Rprofile, restart R.
ind = 0 will indicate that you only want the local repository. Additional repositories can be included in the addURLs = option and are comma separated within the character vector.
Next, create the repository index with the following code:
tools::write_PACKAGES("/<your higher directories>/R/src/contrib", verbose = TRUE)
This will generate the PACKAGE files that serve as the repository index.
To see what packages are in your repository, run the following code and take a look at the resulting data frame: my_packages <- available.packages()
At this point, you can install packages from the repo without referencing the entire path to the package installation file. For example, to install the dplyr package, you could run the following:
install.packages("dplyr", dependencies = TRUE)
If you want to take it a step further and manage a changing repository, you could install and use the miniCRAN package. Otherwise, you'll need to execute the write_PACKAGES function whenever your repository changes.
After installing the miniCRAN package, you can execute the following code to create the miniCRAN repo:
my_packages <- available.packages()
miniCRAN::makeRepo(
pkgs = my_packages[,1,
path = "/<your higher directories>/R",
repos = getOption("repos"),
type = "source",
Rversion = R.version,
download = TRUE,
writePACKAGES = TRUE,
quiet = FALSE
)
You only need to execute the code above once for each repo.
Then, check to make sure each miniCRAN repo has been created. You only need to do this once for each repo:
pkgAvail(
repos = getOption("repos"),
type = "source",
Rversion = R.version,
quiet = FALSE
)
Whenever new package files are placed into the local repo you can update the local repo's index as follows:
miniCRAN::updateRepoIndex("/<your higher directories>/R/")
Finally, as an optional step to see if the new package is in the index, create a data frame of the available packages and search the data frame:
my_packages <- available.packages(repos = "file://<your higher directories>/R/")
This approach has worked pretty well for me, but perhaps others have comments and suggestions that could improve upon it.
I want to use packrat on a Windows 7 machine with no internet connection.
I have downloaded all binary packages from http://cran.r-project.org/bin/windows/contrib/3.1/ into the local folder C:/xyz/CRAN_3_1.
The problem is now that
packrat::init(options=list(local.repos="C:/xyz/CRAN_3_1"))
throws a bunch of warnings and errors like
Warning: unable to access index for repository http://cran.rstudio/bin/...
Warning: unable to access index for repository http://cran.rstudio/src/...
Fetching sources for Rcpp (0.11.4) ... Failed
Package Rcpp not available in repository or locally
As it seems packrat tries to find
the binary version of Rcpp on CRAN (fails since there is no internet connection)
the source of Rcpp on CRAN (fails since there is no internet connection)
the local source of the package (fails since I only have the binaries)
What I don't understand is why packrat does not also search for the local binary package...
Question 1: I could download the source CRAN repository to get around this problem. But I would like to know from you guys whether there is an easier solution to this, i.e., whether it is possible to make packrat accept a local binary repo.
Question 2: When I create my own package myPackage with packrat enabled, will the myPackage-specific local packrat library also be included in the package? That is, assume that I give the binary myPackage zip File to one of my colleagues who does not have one of the packages that myPackage depends on (let's say Rcpp). Will Rcpp be included in myPackage when I use packrat? Or does my colleague have to install Rcpp himself?
I managed to hack around this problem. Please bear in mind that I have never used packrat before and that I do not know its "proper" behaviour. But my impression is that the hack works.
Here is how I did it:
Open your project, load packrat via library(packrat)
type fixInNamespace("snapshotImpl",ns="packrat") - a window opens - copy its content into the clipboard
Go to /yourProjDir/ and create a file snapshotImplFix.R
Copy the clipboard's content into this file ...
... but change the first line to
snapshotImplFix=function (project, available = NULL, lib.loc = libDir(project),
dry.run = FALSE, ignore.stale = FALSE, prompt = interactive(),
auto.snapshot = FALSE, verbose = TRUE, fallback.ok = FALSE,
snapshot.sources = FALSE)
Note snapshot.sources = FALSE! Save and close the file.
Create /yourProjDir/.Rprofile and add
setHook(packageEvent("packrat","onLoad"),function(...) {
source("./snapshotImplFix.R");
tmpfun=get("snapshotImpl",envir=asNamespace("packrat"));
environment(snapshotImplFix)=environment(tmpfun);
utils::assignInNamespace(x="snapshotImpl",value=snapshotImplFix,ns="packrat");})
Points 2-6 fix the problem with the snapshot.sources argument being TRUE by default (I did not find a better way to change that...)
Finally, we have to tell packrat to take our local repository. It's important that you have the right folder structure. Therefore I moved the repo from C:/xyz/CRAN_3_1 to C:/xyz/CRAN_3_1/bin/windows/contrib/3.1. Do not forget to run library(tools);write_PACKAGES("C:/xyz/CRAN_3_1/bin/windows/contrib/3.1"); if you also have to move your files.
Open yourProjDir/.Rprofile again and add at the end
local({r=getOption("repos");r["CRAN"]="file:///C:/xyz/CRAN_3_1";r["CRANextra"]=r["CRAN"];options(repos=r)})
Note the 3 / right after file! Save and exit file.
Close the project and re-open.
Now you can execute packrat::init() and it should run without errors.
It would be great if someone with more experience regarding packrat could give his/her input so that I can be sure that this hack works. Any pointers to proper solutions are highly appreciated, of course.
I'm distributing jobs over a cluster and I'd rather not go to each machine and manually install the right packages. The job controller runs scripts as nobody, so I have to specify uncontroversial writeable paths for the installations. I actually had this working solution:
`%ni%` = Negate(`%in%`) ### "not in"
.libPaths("/tmp/") ### for local (remote non super user) install of packages
if ("xxx" %ni% installed.packages()) {install.packages("xxx", repos = "http://cran.r-project.org", lib="/tmp/")}
# ... and more
library(xxx)
# ... and more
It worked at first, but a week later I've got a strange problem.
> library(xxx)
Error in library(xxx) : there is no package called 'xxx'
xxx (and other packages) is in the manifest of installed.packages(), .libPaths is reporting /tmp/ on the path, and ls shows a folder for the package in /tmp/. Reinstalling with install.packages throws an error, as does remove.package, update.package, and find.package.
Two questions:
Is there a different way that I ought to have managed the remote install?
Any ideas what has caused my problem with the failure to load the package?
Please save me from having to implement a kludge like
locdir <- paste("/tmp/", as.integer(runif(1, 1, 100000)), sep='')
system(paste("mkdir", locdir))
.libPaths(locdir)
install.packages("xxx", repos = "http://cran.r-project.org", lib=locdir)
library(xxx)
You might need option character.only = TRUE, although it is weird that your code worked before but not anymore. Anyway, try this function:
packageLoad<-function(libName){
# try to load the package
if (!require(libName,character.only = TRUE)){
# if package is not available, install it
install.packages(libName,dep=TRUE,
repos="http://cran.r-project.org",lib="/tmp/",destdir="/tmp/")
# try again
if(!require(libName,character.only = TRUE))
stop(paste("Package ",libName,"
not found and its installation failed."))
}
}