I have a bit confusion with regards binaries storage in Nexus. If I have only one blob store, and have multiple repos that use it, does duplication happens for binaries that have the same digest/hash?
For example: a docker image alpine:latest, is pushed to both repo1 and repo2 in Nexus which use the same blob store, is this image stored twice?
The image is stored twice if you push it to two seperate repositories. This is true of any format.
Source: NXRM Experience and Sonatype Contributor
Related
In a near future I will start using Artifactory in my project. I have been reading about local and remote repositories and I am a bit confused of their practical use. In general as far as I understand
Local repositories are for pushing and pulling artifacts. They have no connection to a remote repository (i.e. npm repo at https://www.npmjs.com/)
Remote repositories are for pulling and caching artifacts on demand. It works only one way, it is not possible to push artifacts.
If I am right up to this point, then practically it means you only need a remote repository for npm if you do not develop npm modules but only use them to build your application. In contrast, if you need to both pull and push Docker container images, you need to have one local repository for pushing&pulling custom images and one remote repository for pulling official images.
Question #1
I am confused because our Artifactory admin created a local npm repository for our project. When I discussed the topic with him he told me that I need to first get packages from the internet to my PC and push them to Artifactory server. This does not make any sense to me because I have seen some remote repositories on the same server and what we need is only to pull packages from npm. Is there a point that I miss?
Question #2
Are artifacts at remote repository cache saved until intentionally deleted? Is there a default retention policy (i.e. delete packages older than 6 months)? I ask this because it is important to keep packages until a meteor hits the servers (for archiving policy of the company).
Question #3
We will need to get official Docker images and customize them for CI. It would be a bit hard to maintain one local repo for pulling&pushing custom images and one remote repo for pulling official images. Let's say I need to pull official Ubuntu latest, modify it, push and finally pull the custom image back. In this case it should be pulled using remote repository, pushed to local repo and pulled again from local repo. Is it possible to use virtual repositories to do this seamlessly as one repo?
Question #1 This does not make any sense to me because I have seen some remote repositories on the same server and what we need is only to pull packages from npm. Is there a point that I miss?
Generally, you would want to use a remote repository for this. You would then point your client to this remote repository and JFrog Artifactory would grab them from the remote site and cache them locally, as needed.
In some very secure environments, corporate policies do not even allow this (they may not even be connected to the internet) and instead manually download, vet, and then upload those third-party libraries to a local repository. I don't think that is your case and they may just not understand their intended usages.
Question #2 Are artifacts at remote repository cache saved until intentionally deleted? Is there a default retention policy?
They will not be deleted unless you actively configure it to do so.
For some repo types there are built-in retention mechanisms like the number of snapshots or maximum tags but not for all of them and even in those that have it, they must be actively turned on. Different organizations have different policies for how long artifacts must be maintained. There are a lot of ways to cleanup those old artifacts but ultimately it will depend on your own requirements.
Question #3 Is it possible to use virtual repositories to do this seamlessly as one repo?
A virtual repository will let you aggregate your local and remote sites and appear as a single source. So you can do something like:
docker pull myarturl/docker/someimage:sometag
... docker build ...
docker push myarturl/docker/someimage:sometag-my-modified-version
docker pull myarturl/docker/someimage:sometag-my-modified-version
It is also security-aware so if the user only has access to the local stuff and not the remote stuff, they will only be able to access the local stuff even though they are using the virtual repository that contains both of them.
That said, I don't see why it would be any harder to explicitly use different repositories:
docker pull myarturl/docker-remote/someimage:sometag
... docker build ...
docker push myarturl/docker-local/someimage:sometag-my-modified-version
docker pull myarturl/docker-local/someimage:sometag-my-modified-version
This also has the added advantage that you know they can only pull your modified version of the image and not the remote (though you can also accomplish that by creating the correct permissions).
Artifactory is using storage based on checksum; So if i need to upload the same artifact in 2 artifactory repos; The artifact shall be physically stored only once to optimize footprint.
Is this applicable to any type of repo: especially generic and docker ?
in other words, if i have 2 registries configured in my artifactory, will image common to several charts be stored only once?
Brs
Yes. It's stored only once for best efficiency and control, regardless of the repository type.
See the official documentation on how it actually works.
Is there some way to link Conan packages over local repositories like Bintray (conan-center) ?
I read Manipulating Artifacts but I just found Copy and Move features.
Regards.
Artifactory does not support linking artifacts between repositories.
As you mentioned, you can copy artifacts between repositories. Keep in mind that this is a "cheap copy" - since Artifactory is using a checksum based storage, the bytes are not duplicated and instead Artifactory just creates another pointer in the database.
Another option for having the same artifact in multiple repositories is using virtual repositories (will be supported in the future for Conan). You can keep the artifact in a local repository and include this local repository in multiple virtual repositories. This will result with being able to access the same artifact from multiple virtual repositories.
We have an Artifactory solution deployed and I am trying to figure out if it can meet my use case. The normal use case is that artifacts are deleted within a week or so and can normally fit in X GB of local storage, but we'd like to be able to:
Keep some artifacts around much longer, and since they are accessed infrequently, store them in AWS S3.
Sometimes artifacts aren't able to be cleaned up in time, so we'd like to burst to the cloud when local storage is overflowed.
I was thinking I could do the following:
Local repository of X GB
Repo pointing to S3
Virtual repo in front of both of these
Setup a plugin to move artifacts from local->S3 via our policies
However, I can't figure out what a Filestore is in Artifactory, and how you'd have two Repositories backed by different filestores.
Anyone have pointers to documentation or anything that can help? The docs I can find are rather slim on the high level details of filestores and repositories.
The Artifactory binary provider does not support configuring multiple storage backends, so it is impossible to use S3 and NFS in parallel. The main reason for this limitation is that Artifactory has a checksum based storage which stores each binary only once and keeps pointers from all relevant repositories. For that reason Artifactory does not manage separate storage per repository.
For archiving purposes, one of the possible solutions is setting up another Artifactory instance which will take care of archiving. This instance can be connected to an S3 storage backend.
You can use replication to synchronize between the two instances (without syncing deletes). You can have a repository(s) in your master Artifactory which contains artifacts which should be archived, those artifacts will be replicated to the archive Artifactory and later on can be deleted from the master.
You can use a user plugin to decide which artifacts should be moved to the archive repository.
I want to move a set of artifacts from one Nexus to other (download and later upload). I just can download the artifacts one by one, ¿Is there any way to download the entire folder? ¿Is there any other kind of operation like export/import?
Thanks!
EDIT:
I have access to the nexus repository (sonatype-work\nexus\storage) in the user folder. I have got from there all the artifacts. I didn't find any way to do it from the web client.
Nexus stores the artifacts on disk in standard Maven 2/3 repository layout, so you can just directly copy the artifacts from one storage directory to the other using whatever means you like.
After you're finished, schedule a repair index task against the destination repository so that searching for artifacts from the web UI will work. Note that your builds will work immediately after you copy the artifacts, indexes are not used by maven builds.