Filter property based searches in Artifactory - artifactory

I'm looking to use the Artifactory property search
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-ArtifactSearch%28QuickSearch%29
Currently this will pull json listing any artifact that matches my properties.
"results" : [
{
"uri": "http://localhost:8080/artifactory/api/storage/libs-release-local/org/acme/lib/ver/lib-ver.pom"
},{
"uri": "http://localhost:8080/artifactory/api/storage/libs-release-local/org/acme/lib/ver2/lib-ver2.pom"
}
]
I need to be able to filter the artifacts I get back as i'm only interested in a certain classifier. The GAVC Search has this with &c=classifier
I can do it in code if this isn't possible via the interface
Any help appreciated

Since the release of AQL in Artifactory 3.5, it's now the official and the preferred way to find artifacts.
Here's an example similar to your needs:
items.find
(
{
"$and":[
{"#license":{"$eq":"GPL"}},
{"#version":{"$match":"1.1.*"}},
{"name":{"$match":"*.jar"}}
]
}
)
To run the query in Artifactory, copy the query to a file and name it aql.query
Run the following command from the directory that contains the aql.query file
curl -X POST -uUSER:PASSWORD 'http://HOST:PORT/artifactory/api/search/aql' -Taql.query
Don't forget to replace the templates (USER, PASSWORD,HOST and PORT) to real values.
In the example
The first two criteria are used to filter items by properties.
The third criteria filters items by the artifact name (in our case the artifact name should end with .jar)
For more details on how to write AQL query are in AQL
Old answer
Currently you can't combine the property search with GAVC search.
So you have two options:
Executing one of them (whichever gives you more precise results) and then filter the JSON list on the client by a script
Writing an execution user plugin that will execute the search by using the Searches service and then filter the results on the server side.
Of course, the later is preferable.

Related

Create document set in Sharepoint with Graph API in a subfolder

I already implemented the creation of a document set at library root level. For this I used the following link: Is it possible to create a project documentset using graph API?
I follow the following steps :
1- Retrieve the document library's Drive Id:
GET https://graph.microsoft.com/v1.0/sites/${siteId}/lists/${listId}?$expand=drive
2- Create the folder:
POST https://graph.microsoft.com/v1.0/drives/${library.drive.id}/root/children
The body of the request is the following
{
"name": ${folderName},
"folder": {},
}
3- Get the folder's SharePoint item id:
GET https://graph.microsoft.com/v1.0/sites/${siteId}/drives/${library.drive.id}/items/${folder.id}?expand=sharepointids
4- Update the item in the Document Library so that it updates to the desired Document Set:
PATCH https://graph.microsoft.com/v1.0/sites/${siteId}/lists/${listId}/items/${sharepointIds.listItemId}
I do send the following body to the patch request:
{
"contentType": {
"id": "content-type-id-of-the-document-set"
},
"fields": {}
}
I'm looking now how to create a document set in a specific folder in Sharepoint.
For example, i want to create the following folder structure.
The documents folder is at the library root and I want to create a document set named billing.
documents
|_ billing
|_ 2021
|_11
|_01
|_ document1.pdf
|_ document2.pdf
|_ document3.pdf
|_02
...
|_03
...
|_04
|_10
...
thanks
I'm doing something similar but I'm a little behind you, haven't yet created the Document Set (almost there!), but may I respectfully challenge your approach?
I'm not sure it's a good idea to mix Folders and Document Sets, mainly because a Folder breaks the metadata flow, you could achieve the same results just using Document Sets.
I am assuming that your 'day' in the data structure above is your Document Set (containing document1.pdf, etc.) You might want to consider creating a 'Billing' Document Set and either specifically add a Date field to the Document Set metadata or, perhaps better still, just use the standard Created On metadata and then create Views suitably filtered/grouped/sorted on that date.
This way you can also create filtered views for 'Client' or 'Invoice' or 'Financial Year' or whatever.
As soon as your documents exist in a folder, you can no longer filter/sort/group etc., the document library based on metadata.
FURTHER INFORMATION
I am personally structuring my Sales document library thus:
Name: Opportunity; Content Type: Document Set; Metadata: Client Name, Client Address, Client Contact
Name: Proposal; Content Type: Document; Metadata: Proposal ID, Version
Name: Quote; Content Type: Document; Metadata: Quote ID, Version
Etc...
This way the basic SharePoint view is a list of Opportunities (Document Sets), inside which are Proposals, Quotes etc., but I can also filter the view to just show Proposals (i.e. filter by Content Type), or search for a specific Proposal ID, or group by Client Name, then sort chronologically, or by Proposal ID etc.
I'm just saying that you get a lot more flexibility if you avoid using Folders entirely.
p.s. I've been researching for days now how to create Document Sets with graph, it never occurred to me that it might be a two-step process i.e. create the folder, then patch its content type. Many thanks for your post!!
Just re-read your post and my assumption that the 'day' would be your document set was incorrect. In this case, there would be no benefit having a Document Set containing Folders because the moment a Folder exists in the Document Set, metadata flow stops, and the only reason (well, the main reason*) to use Document Sets in preference to Folders is that metadata flow.
*Document Sets also allow you to automatically create a set of documents based on defined templates.

Is there a way to fetch list of files belonging to a particular file format from a folder in a zip in Artifactory using REST/AQL?

We have a use case to download or at least list all .xml files from a folder inside a zip in Artifactory from a python script. Is it possible with REST or using AQL ? I am looking for a search query to get all xmls from a folder in zip so that I can use "archive entries download" thing from REST API to make download request for each file individually. Please help me with relevant information about the endpoint or query that I can use to achieve my task(list all xml in a folder in zip). Also please let me know of any other efficient approach to accomplish the task.
You can use an AQL query to get a list of files from a given path inside an archive.
As an example, lets take the my-archive.zip archive deployed in a repository names myrepo inside folder1 folder
The follow AQL query can be used to find all the xml files within this archive under the path level-1/level-2/level-3:
archive.entries.find( {
"archive.item.path":{"$eq":"folder1"},
"archive.item.name":{"$eq":"my-archive.zip"},
"archive.entry.path":{"$eq":"level-1/level-2/level-3"},
"archive.entry.name":{"$match":"*.xml"}
})
You can run this query using the AQL REST API method and get the following result:
{
"results" : [ {
"entry.name" : "dummy.xml",
"entry.path" : "level-1/level-2/level-3"
} ],
"range" : {
"start_pos" : 0,
"end_pos" : 1,
"total" : 1
}
}
Here is an example of calling this REST API using curl (the query itself is stored in a file called aql.json):
curl -XPOST -uadmin:password -H"content-type: text/plain" -d#aql.json http://localhost:8081/artifactory/api/search/aql

How can I add 'path' to my Artifactory AQL query?

I'm using insomnia to make calls to the Artifactory API.
I have the following query, which works really well:
items.find({"repo":{"$eq":"my-repository-virt"}}, {"$and":[{"#my.fileType":{"$match": "jar"}},{"#my.otherType":{"$match": "type2"}},{"#prodVersion":{"$match": "false"}}]})
But I have a problem in that there are duplicate files in some sub-folders with the same properties/filename that I would like to exclude.
I would like to add path to this query, but I can never get any results returned.
The repository is a virtual repository that links to 3 other real repositories.
One of my colleagues can call the following query with the command line tool and get the expected results:
jfrog rt search my-repo-snapshots/myproject/subfolder/jars/*.jar
I have tried adding the path parameter to my query, I've tried removing everything except the repo and the path, like this:
items.find({"repo":{"$eq":"my-repo-snapshots"}},{"path" : "my-repo-snapshots/myproject/subfolder/jars/*.jar"})
I've tried with just the path, with variations on the path, including/excluding the repo name, using the virtual repo, the actual repo, but I always get a successful search with 0 results returned.
How can I build this query to search the virtual repo, along a certain path, and including certain properties?
EDIT:
I've also tried:
items.find({"repo":{"$eq":"my-repo-snapshots"}},{"path" : {"$match":"my-repo-snapshots/myproject/subfolder/jars/*.jar"}})
Both with the repo in the path and without, I still get 0 results.
OK I figured it out.
The path part needs to be added in with the {"$and": ...} section where the properties are included. Like so:
items.find({"repo":{"$eq":"my-repository-virt"}},
{"$and":[
{"path":{"$match":"path/to/relevant/folders/*"}},
{"#my.fileType":{"$match": "jar"}},
{"#my.otherType":{"$match": "type2"}},
{"#prodVersion":{"$match": "false"}}
]})
The easier fix would have been:
items.find({"repo":{"$eq":"my-repo-snapshots"}},{"path" : {"$eq":"my-repo-snapshots/myproject/subfolder/jars"}, {"name" : {"$match":"*.jar"}})
So the problem with your initial attempt, is that the "path" should match the folder and the "name" should match the filename

Artifactory AQL delete empty folders

How do I delete empty folders(folders without any content) by using Artifactory AQL?
I have the current AQL query to find files that are older than 12w and never downloaded, which I will delete by an script.
items.find(
{
"repo":{"$eq":"libs-release-local"},
"stat.downloads":{"$eq":null},
"created":{"$before" : "12w"},
}
)
This leaves me with empty folders, how do I specify an AQL query that finds all empty folders?
From Artifactory Query Language documentation: if type is not specified in the query, the default type searched for is file.
By adding a type to the query you can control the result type: file, folder or both.
For example:
items.find(
{
"repo": {"$eq":"libs-release-local"},
"stat.downloads": {"$eq":null},
"created": {"$before" : "12w"},
"type": {"$eq":"any"}
}
)
If you are not married to the idea of using AQL, note that there is an Empty Folder Clean-up plugin by JFrog.

Aggregating a list of http log paths in kibana

I have a nginx->fluentd->elasticsearch->kibana stack up and running. Trying to figure if I can do something like a "terms" panel but with a path string component from logs. Using a terms panel directly on that results in top used words from paths, e.g. for drupal it shows "node" as the most popular, which is quite useless without actual node id.
Is that something that is possible to do with elasticsearch?
Update: Here's a sample of my logs:
"path": "/node/123"
"path": "/node/456"
"path": "/user/create"
If I add a "terms" panel for "path" field, I get columns for "node", "user", "create", which make no statistical sense. What I need is a terms panel that aggregates on unique field values, not unique word parts of the field.
You need to configure Elasticsearch's mapping for setting your "path" field as a "not_analyzed" one. The default setting is "analyzed" and by default, ES parses the string fields and divide them in multiple tokens when possible, which is probably what happened in your case. See this related question.
As for how to configure Elasticsearch's mapping, I am also still digging, having a similar problem myself with multi-token strings I want to be able to sort on. It seems like there would be a put mapping API or the possibility of using config files, see here.

Resources