I added the package Knet with Pkg.add("Knet") and noticed that several packages were installed including CUDA. However, after the installation finished when I try:
using CUDA
it says that this package is not found but that it is available from a registry. It seems that this package is a requirement for Knet and it is installed but then one cannot access it right away. Do you know what is happening behind scenes? Thanks.
The underlying mechanism is a bit complex, and is described in detail here.
But the general logic is as follows: you can use (with using or import) the packages that you have explicitly installed. However, such packages might depend on other packages. Julia will automatically decide what other packages are needed to be installed, but they will be not visible in your project unless you explicitly install them.
In fact, typically, on one computer you will have hundreds of packages installed in one place (to avoid having to download and precompile them each time), but each individual project will have access only to packages that you explicitly specify you want to use in this project. The information what packages should be visible in an individual project is typically contained in the Project.toml file as is described here.
You can find more information how to manage projects in Julia here.
Related
In Julia 1.5.3 when i try to add MLPlots package with this command "] add MLPlots" I get the below error :
ERROR: The following package names could not be resolved:
* MLPlots (not found in project, manifest or registry)
Fredrik's comment should really be the answer here, so I'll expand on it a little bit:
When you do ] add SomePackage, Julia's package manager will search the General Registry to look for the package, and then install it. As you will see when looking at the M entry in the general registry, MLPlots is not registered, hence the "not found in ... registry" error you're seeing.
Now it should be noted that packages don't have to be installed in the general registry to be installed - the general registry is just the default registry, and largely a convenience tool in the Julia ecosystem which provides the default location to check for packages and enforces certain standards on registered packages to ensure things play nicely with each other. That being said, nothing prevents you from
Adding a different registry: as noted in the Pkg.jl docs introduction, "it allows multiple registries managed by different parties to interact seamlessly. In particular, this includes private registries which can live behind corporate firewalls. One example of this is the JuliaPro registry, which offers a curated subset of the General registry;
Just adding a package that's not registered anywhere, either from a local path or using a GitHub (or equivalent service, e.g. GitLab) link.
While the second would in principle be an option in this case, at this point the other problem mentioned in Fredrik's comment becomes pertinent:
(#v1.6) pkg> activate --temp
Activating new environment at `/tmp/jl_pKGogE/Project.toml`
(jl_pKGogE) pkg> add https://github.com/JuliaML/MLPlots.jl
Cloning git-repo `https://github.com/JuliaML/MLPlots.jl`
Updating git-repo `https://github.com/JuliaML/MLPlots.jl`
ERROR: could not find project file in package at `https://github.com/JuliaML/MLPlots.jl` maybe `subdir` needs to be specified
Since v1.0, Julia requires packages to come with a "project" file (Project.toml), which specifies the packages dependencies including (at least as best practice, and for packages in the general registry) compatibility bounds. As the package you're looking at was last worked on 5 years ago, it is not in a format that Julia's current package manager can deal with.
Looking at the package itself I would also say that this isn't necessarily a problem - not trying to knock Tom here who's done exceptional and pioneering work in the Julia plotting ecosystem, but this specific package seems to be a bit of a proof-of-concept which doesn't provide an awful lot of functionality. Maybe if you post on the Julia Discourse a bit of a more open-ended request for help with whatever problem you're trying to solve people can point you towards the current state-of-the-art packages in the ecosystem.
I'll just add here that the central package for standard machine learning in Julia at the moment is probably MLJ.jl, which includes a roc_curve function that could give you the basis for the ROCAnalysis plot you see in the MLPlots readme.
I'm trying to use a linux server with R installed. Apparently the R system library has old versions of non-base packages installed like dplyr and testthat.
Because i don't have permission to edit the system library, i'm unable to update the packages.
My plan is to only use a user library, so I can controll the package versions myself. However i'm unable to remove the "/usr/lib64/R/library" folder from .libPaths(). I tried changing the environment variables R_LIBS_SITE and R_LIBS with the .Renviron and .Rprofile files to a different folder, but the /usr/lib64/R/library folder will always be present. Removing it with the command .libPaths(.libPaths()[1:2]) doesn't work either.
Is there a way to remove the system library from .libPaths(), so I'm not depending on the update policy of the server admin?
You can't remove the system library, because that's where the base packages live. They can't be installed anywhere else, and R won't work without them.
Best would be for you to get your sysadmin to update the system library. Those obsolete packages probably contain bugs.
If you can't do that, then run update.packages(instlib = "local") to install all the latest versions in the library named "local". (Substitute your own local lib name, of course.) This requires all your users to specify .libPaths("local") when they start, and some will likely forget, so it's not as good.
It might be easiest for you to just install a full copy of R in your own account. Then you'll have control of things, and anyone using your copy will get your library.
(There's a new release (3.5.3) coming in ten days; you might wait for that, or install one of the betas or RCs, which should be available now, then update again when the final release arrives.)
For me, it works to use
.libPaths(.libPaths()[2:1])
This will still search the system library, but only after it searches my personal library, so if I have a newer version, it uses that. Note: I used .libPaths()[2:1] not .libPaths()[1:2]
How do you best pin package versions in R?
Rejected strategy 1: Pin to CRAN source tar.gzs
Doesn't work if you want to pin it at the latest version since CRAN does not put the tip version in the archive (duh)
Rejected strategy 2: Use devtools
Don't want to, because it takes ages to compile and adds lots of stuff I don't want to use
Rejected strategy 3: Vendor
Would rather avoid having to copy all source
To provide a little bit more information on packrat, which I use for this purpose. From the website.
R package dependencies can be frustrating. Have you ever had to use
trial-and-error to figure out what R packages you need to install to
make someone else’s code work–and then been left with those packages
globally installed forever, because now you’re not sure whether you
need them? Have you ever updated a package to get code in one of your
projects to work, only to find that the updated package makes code in
another project stop working?
We built packrat to solve these problems. Use packrat to make your R
projects more:
Isolated: Installing a new or updated package for one project won’t
break your other projects, and vice versa. That’s because packrat
gives each project its own private package library. Portable: Easily
transport your projects from one computer to another, even across
different platforms. Packrat makes it easy to install the packages
your project depends on. Reproducible: Packrat records the exact
package versions you depend on, and ensures those exact versions are
the ones that get installed wherever you go.
Packrat stores the version of the packages you use in the packrat.lock file, and then downloads that version from CRAN whenever you packrat::restore(). It is much lighter weight than devtools, but can still take some time to re-download all of the packages (depending on the packages you are using).
If you prefer to store all of the sources in a zip file, you can use packrat::snapshot() to pull down the sources / update the packrat.lock and then packrat::bundle() to "bundle" everything up. The aim for this is to make projects / research reproducible and portable over time by storing the package versions and dependencies used on the original design (along with the source code so that the OS dependency on a binary is avoided).
There is much more information on the website I linked to, and you can see current activity on the git repo. I have encountered a few cases that work in a less-than-ideal way (packages not on CRAN have some issues at times), but the git repo still seems to be pretty active with issues/patches which is encouraging.
The development of RStudio and the packages devtools and roxygen2 has made R package creation pretty easy. I use GitHub for version control and devtools allows others to easily install directly from my account.
As my package gradually changes with each version, I'm wondering if I should be maintaining .zip files (or other format) of my past stable builds, in case anyone would ever want to use a previous version.
It's easy to download a .zip of an R package directly from GitHub, but I'm wondering if I should add this to the same GitHub directory (e.g. https://github.com/myaccount/mypackage/previous_versions/mypackage_0.1.zip) without messing up somebody's installation via install_github("myaccount/mypackage").
So, the main Qs are:
Should I keep an old package version at all?
Should I keep old package versions in a sub-folder of my GitHub R package directory?
Should I save .zip files downloaded from GitHub as my old version, or produce a Source or Binary file during the package build itself (i.e. in RStudio)?
Is this a superfluous activity if one isn't yet willing to publish to CRAN?!
When you think your package is at a good solid place, you should tag a release. This archives the branch at that point in time and stores the zip file with the source code, and the tar.gz file.
I tend to mark my CRAN packages as a release each time I release it to CRAN (for example, see https://github.com/nutterb/pixiedust/releases) and with some intemediary tags that I consider noteworthy.
Another good strategy for managing changes in between tagged releases is to maintain a development branch below your main branch. That way your development changes won't pollute or break anything being used by those pulling from your main branch. It makes you free to experiment in the dev branch while always having a clean, working copy to push to and restore from.
1. Should I keep an old package version at all?
It's subjective, but I'd definitely say "yes" unless there's a space constraint, which is probably unlikely.
This serves 2 purposes. One is for your own convenience, such as if you want to make sure that you always have a quick way to test the results of older versions versus a newer version.
The other is that people often need older versions of packages, such as if someone wants to use your package but they're using an older version of R on a server where the policies prevent an update to R. Perhaps a newer version of your package includes a new dependency which only works with a package that depends on a certain version of R or higher.
Of course, packages can always be installed without the compressed or binary files, but it's a nice convenience.
2. Should I keep old package versions in a sub-folder of my GitHub R package directory?
I would put it in a trunk or special subfolder that won't be automatically downloaded when someone tries to install_github or clone your master branch. Having a separate branch is a good idea.
3. Should I save .zip files downloaded from GitHub as my old version, or produce a Source or Binary file during the package build itself (i.e. in RStudio)?
As the package author you're in a position to know if these differ significantly and which if either is better, but by default I'd recommend the RStudio build because I assume (if you're like me) that you're less likely to include unnecessary files this way.
4. Is this a superfluous activity if one isn't yet willing to publish to CRAN?!
No, not necessarily. If people rely on your package then it really doesn't matter if it's on CRAN or not. In fact, not being on CRAN may be a reason to be more proactive like this to ensure that your users will always have access to the needed version of your package.
In order to be able to compare two versions of a package, I need to able to choose which version of the package that I load. R's package system is set to by default to overwrite existing packages, so that you always have the latest version. How do I override this behaviour?
My thoughts so far are:
I could get the package sources, edit the descriptions to give different names and build, in effect, two different packages. I'd rather be able to work directly with the binaries though, as it is much less hassle.
I don't necessarily need to have both versions of the packages loaded at the same time (just installed somewhere at the same time). I could perhaps mess about with Sys.getenv('R_HOME') to change the place where R installs the packages, and then .libpaths() to change the place where R looks for them. This seems hacky though, so does anyone have any better ideas?
You could selectively alter the library path. For complete transparency, keep both out of your usual path and then do
library(foo, lib.loc="~/dev/foo/v1") ## loads v1
and
library(foo, lib.loc="~/dev/foo/v2") ## loads v2
The same works for install.packages(), of course. All these commands have a number of arguments, so the hooks you aim for may already be present. So don't look at changing R_HOME, rather look at help(install.packages) (assuming you install from source).
But AFAIK you cannot load the same package twice under the same name.
Many years have passed since the accepted answer which is of course still valid. It might however be worthwhile to mention a few new options that arised in the meanwhile:
Managing multiple versions of packages
For managing multiple versions of packages on a project (directory) level, the packrat tool can be useful: https://rstudio.github.io/packrat/. In short
Packrat enhances your project directory by storing your package dependencies inside it, rather than relying on your personal R library that is shared across all of your other R sessions.
This basically means that each of your projects can have its own "private library", isolated from the user and system libraries. If you are using RStudio, packrat is very neatly integrated and easy to use.
Installing custom package versions
In terms of installing a custom version of a package, there are many ways, perhaps the most convenient may be using the devtools package, example:
devtools::install_version("ggplot2", version = "0.9.1")
Alternatively, as suggested by Richie, there is now a more lightweight package called remotes that is a result of the decomposition of devtools into smaller packages, with very similar usage:
remotes::install_version("ggplot2", version = "0.9.1")
More info on the topic can be found:
https://support.rstudio.com/hc/en-us/articles/219949047-Installing-older-versions-of-packages
I worked with R for a longtime now and it's only today that I thought about this. The idea came from the fact that I started dabbling with Python and the first step I had to make was to manage what they (pythonistas) call "Virtual environments". They even have dedicated tools for this seemingly important task. I informed myself more about this aspect and why they take it so seriously. I finally realized that this is a neat and important way to manage different projects with conflicting dependencies. I wanted to know why R doesn't have this feature and found that actually the concept of "environments" exists in R but not introduced to newbies like in Python. So you need to check the documentation about this and it will solve your issue.
Sorry for rambling but I thought it would help.