Is there some way to link Conan packages over local repositories like Bintray (conan-center) ?
I read Manipulating Artifacts but I just found Copy and Move features.
Regards.
Artifactory does not support linking artifacts between repositories.
As you mentioned, you can copy artifacts between repositories. Keep in mind that this is a "cheap copy" - since Artifactory is using a checksum based storage, the bytes are not duplicated and instead Artifactory just creates another pointer in the database.
Another option for having the same artifact in multiple repositories is using virtual repositories (will be supported in the future for Conan). You can keep the artifact in a local repository and include this local repository in multiple virtual repositories. This will result with being able to access the same artifact from multiple virtual repositories.
Related
We have our builds pointing to a remote repo in artifactory for installing python packages. We do see the requested packages being cached.
Now we want to get the list of all the packages that are cached along with version information. How do we get this info.?
Tried the REST API for artifact version search, but it requires a group and an artifact id which we don't have for public packages.
Note: Not looking for all the available versions of packages, rather only the version that is cached for each package.
Instead of querying the remote repository you can query its cache.
The cache can be accessed by adding -cache to the repository key. For example, if you have a remote repository named pip, you can query a repository named pip-cache instead. This will include only the cached artifacts.
See the following article in JFrog's knowledge base: Remote repository how it works, pull replication with other Artifactory server and the ability to use only the cached artifacts
Some other useful and relevant articles:
How Remote Repository Metadata works
Remote Repositories documentation
In a near future I will start using Artifactory in my project. I have been reading about local and remote repositories and I am a bit confused of their practical use. In general as far as I understand
Local repositories are for pushing and pulling artifacts. They have no connection to a remote repository (i.e. npm repo at https://www.npmjs.com/)
Remote repositories are for pulling and caching artifacts on demand. It works only one way, it is not possible to push artifacts.
If I am right up to this point, then practically it means you only need a remote repository for npm if you do not develop npm modules but only use them to build your application. In contrast, if you need to both pull and push Docker container images, you need to have one local repository for pushing&pulling custom images and one remote repository for pulling official images.
Question #1
I am confused because our Artifactory admin created a local npm repository for our project. When I discussed the topic with him he told me that I need to first get packages from the internet to my PC and push them to Artifactory server. This does not make any sense to me because I have seen some remote repositories on the same server and what we need is only to pull packages from npm. Is there a point that I miss?
Question #2
Are artifacts at remote repository cache saved until intentionally deleted? Is there a default retention policy (i.e. delete packages older than 6 months)? I ask this because it is important to keep packages until a meteor hits the servers (for archiving policy of the company).
Question #3
We will need to get official Docker images and customize them for CI. It would be a bit hard to maintain one local repo for pulling&pushing custom images and one remote repo for pulling official images. Let's say I need to pull official Ubuntu latest, modify it, push and finally pull the custom image back. In this case it should be pulled using remote repository, pushed to local repo and pulled again from local repo. Is it possible to use virtual repositories to do this seamlessly as one repo?
Question #1 This does not make any sense to me because I have seen some remote repositories on the same server and what we need is only to pull packages from npm. Is there a point that I miss?
Generally, you would want to use a remote repository for this. You would then point your client to this remote repository and JFrog Artifactory would grab them from the remote site and cache them locally, as needed.
In some very secure environments, corporate policies do not even allow this (they may not even be connected to the internet) and instead manually download, vet, and then upload those third-party libraries to a local repository. I don't think that is your case and they may just not understand their intended usages.
Question #2 Are artifacts at remote repository cache saved until intentionally deleted? Is there a default retention policy?
They will not be deleted unless you actively configure it to do so.
For some repo types there are built-in retention mechanisms like the number of snapshots or maximum tags but not for all of them and even in those that have it, they must be actively turned on. Different organizations have different policies for how long artifacts must be maintained. There are a lot of ways to cleanup those old artifacts but ultimately it will depend on your own requirements.
Question #3 Is it possible to use virtual repositories to do this seamlessly as one repo?
A virtual repository will let you aggregate your local and remote sites and appear as a single source. So you can do something like:
docker pull myarturl/docker/someimage:sometag
... docker build ...
docker push myarturl/docker/someimage:sometag-my-modified-version
docker pull myarturl/docker/someimage:sometag-my-modified-version
It is also security-aware so if the user only has access to the local stuff and not the remote stuff, they will only be able to access the local stuff even though they are using the virtual repository that contains both of them.
That said, I don't see why it would be any harder to explicitly use different repositories:
docker pull myarturl/docker-remote/someimage:sometag
... docker build ...
docker push myarturl/docker-local/someimage:sometag-my-modified-version
docker pull myarturl/docker-local/someimage:sometag-my-modified-version
This also has the added advantage that you know they can only pull your modified version of the image and not the remote (though you can also accomplish that by creating the correct permissions).
After a fresh install of Artifactory, I've noticed a huge number of files I don't need in artifactory/libs-release dir. From Artifact Repository Browser, these artifacts are listed under the libs-release folder cf image below
Is it possible to install Artifactory without all these artifacts ?
Can I safely remove them ?
Those are not stored in your Artifactory instance. The repository libs-release is a virtual repository that contains both local and remote repositories. This will show an aggregate view of three things:
The items in the Artifactory local repositories (which are stored in your instance)
The items cached in your Artifactory remote repositories (which are stored in your instance)
The items available in the Artifactory remote repositories (but not stored in your instance)
In your case, the items you are seeing fall under #3 and are not actually stored in your Artifactory instance, as such you cannot delete them.
I have a remote Maven-repository where I would like to copy a subset of the artifacts to a local repository on the same Artifactory server.
But the GUI does not display the copy-option when I right-click on the artifact and the copy-method in the REST API is only supported for local repository.
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-CopyItem
What is the best thing to do ? Do I need to download the artifacts and "republish" them to the local repository ? Is there not smarter way to copy the artifacts from a remote to a local repository ?
You can use this method as a workaround:
Retrieve artifact from remote repository: GET {remoteRepoKey}/{filePath}. You don't have to save the file in your local machine. As by-product, the download process will copy the item from remoteRepoKey to remoteRepoKey-cache.
Move item from remoteRepoKey-cache to your local repository: POST /api/move/{remoteRepoKey-cache}/{filePath}?to=/{localRepoKey}/{filePath}
It is not perfect, but it'll spare an upload to Artifactory and also the issues of storing a file in filesystem.
Further reading:
What is a Remote repository and how does it work?
Remote Repositories
My blog: Artifactory as a Caching Mechanism for Package Managers
Is there a script or any other automated process for migration of artifacts into JFrog? We are currently working on this and need more information to carry out this process. Please help us in achieving this. Thanks in advance.
If you have an existing artifact repository, JFrog Artifactory supports acting as an artifact proxy while you are in the process of migrating to Artifactory.
I would recommend the following:
Create a local repository in artifactory
Create a remote repository in artifactory which points to your current artifact repository.
Create a virtual repository in artifactory which contains both the local and remote repositories.
Iterate on all your projects to have them publish to the local artifactory repository and pull from the virtual repository.
The advantage to this workflow is that you can port things over piece by piece, rather than trying to do it all at once. If you point a dependency at artifactory that hasn't been ported there yet, artifactory will proxy it for you. When the dependency is ported over, it will be transparent to its users.
When you have moved everything to your local Artifactory repository, then you can remove the remote repository from your virtual repository.
The relevant documentation is available here: https://www.jfrog.com/confluence/display/RTF/Configuring+Repositories
For an Enterprise account, I'd suppose S3 storage and a significant number of artifacts, so there will be no easy and automated way to do it. It also highly dependent on the storage implementation of choice in the on-prem solution. If you plan to use S3 storage, JFrog can help to perform S3 replication. In other scenarios, the solution will be different. I suggest contacting the support.