I'm using the below json filespec for a search query with Jfrog CLI but it's ignoring the "include" part of the spec. There's no documentation on using include within a filespec, just straight AQL examples.
{
"files": [{
"aql": {
"items.find":{
"$and" : [
{"created" : {"$before" : "1y"}},
{"repo" : {"$match":"docker-local"}},
{"stat.downloads":{"$eq":null}},
{"name" : { "$nmatch" : "sha256*" }}
]
}
},
"sortBy":["created"],
"include":["path","name","created_by"]
}]
}
I'm pretty sure it's ignoring anything not in the "aql" object. I added a bunch of random entries like "foo":"bar" under include and sortBy and no error was thrown, nor was it acknowledged in anyway.
My question is, where does the include fit into the spec.json file?
Related
I need to filter an array in my AWS Step Functions state. This seems like something I should easily be able to achieve with JsonPath but I am struggling for some reason.
The state I want to process looks like this:
{
"items": [
{
"id": "A"
},
{
"id": "B"
},
{
"id": "C"
}
]
}
I want to filter this array by removing entries for which id is not in a specified whitelist.
To do this, I define a Pass state in the following way:
"ApplyFilter": {
"Type": "Pass",
"ResultPath": "$.items",
"InputPath": "$.items.[?(#.id in ['A'])]",
"Next": "MapDeployments"
}
This makes use of the JsonPath in operator.
Unfortunately when I execute the state machine I receive an error:
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'ApplyFilter' (entered at the event id #8). Invalid path '$.items.[?(#.id in ['A'])]' : com.jayway.jsonpath.InvalidPathException: com.jayway.jsonpath.InvalidPathException: Space not allowed in path"
}
However, I don't understand what is incorrect with the syntax. When I test here everything works correctly.
What is wrong with what I have done? Is there another way of achieving this sort of filter using JsonPath?
According to the official AWS docs for Step Functions,
The following in paths are not supported # .. , : ? *
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-paths.html
I am trying to query which build number(s) produced artifacts from build foo with artifact property vcs.Revision=aabbccddee123456.
In Artifactory 5.1.3.
I was trying like this so far:
curl -u user:apikey -i -X POST https://artifactory.foobar.com/artifactory/api/search/aql -H "content-type:text/plain" -T query.json
query.json:
builds.find(
{
"module.artifact.item.repo":"snapshot-local",
"name":"foo",
"module.artifact.item.#vcs.Revision":"aabbccddee123456"
}
)
However, none of these 3 lines seem individually correct:
builds.find({"module.artifact.item.repo":"snapshot-local"})
returns nothing,
builds.find({"name":"foo"})
returns the same empty response,
builds.find({"module.artifact.item.#vcs.Revision":"aabbccddee123456"}) also returns this:
{
"results" : [ ],
"range" : {
"start_pos" : 0,
"end_pos" : 0,
"total" : 0
}
}
What am I doing wrong here? I do see in the webapp the builds I published with this name, and with the correct artifact properties.
Here's a working solution that will give build numbers (since giving admin rights to query builds is not a solution for us):
query.json:
items.find(
{
"repo":"snapshot-local",
"artifact.module.build.name":"foo",
"artifact.item.#vcs.Revision":"aabbccddee123456"
}
).include("artifact.module.build.number")
This returns a list of all the artifacts that were built with the relevant properties, with the build number attached, e.g:
{
"results" : [ {
"repo" : "snapshot-local",
"path" : "foo/42",
"name" : "a.out",
"type" : "file",
"size" : 123456789,
"created" : "2018-07-05T12:34:56.789+09:00",
"created_by" : "jenkins",
"modified" : "2018-07-05T12:34:56.789+09:00",
"modified_by" : "jenkins",
"updated" : "2018-07-05T12:34:56.789+09:00",
"artifacts" : [ {
"modules" : [ {
"builds" : [ {
"build.number" : "42"
} ]
} ]
} ]
},
[SNIP]
}
],
"range" : {
"start_pos" : 0,
"end_pos" : 30,
"total" : 30
}
}
I can then parse this to extract build.number.
Certain AQL queries requires a user with admin permissions.
To ensure that non-privileged users do not gain access to information without the right permissions, users without admin privileges have the following restrictions:
The primary domain in the query may only be item.
The following three fields must be included in the include directive: name, repo, and path.
In your case, you are using the build domain in the query which requires admin permissions
I am using Artifactory 4.8.0 (OSS)
I have deployed artifacts to 2 repos: libs-devel and libs-release-candidates.
When I execute
$ curl -u denham:password -X GET https://artifactory.server/artifactory/api/search/artifact?name=MyPackage&repos=libs-devel
The response I receive is:
"results" : [ {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-devel/com/acme/MyPackage/17.10.1-SNAPSHOT/MyPackage-17.10.1-20170908.092803-1.pom"
}, {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-devel/com/acme/MyPackage/17.10.1-SNAPSHOT/MyPackage-17.10.1-20170908.092803-1.war"
}, {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-release-candidates/com/acme/MyPackage/17.10.1-24/MyPackage-17.10.1-24.pom"
}, {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-release-candidates/com/acme/MyPackage/17.10.1-24/MyPackage-17.10.1-24.war"
}, {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-devel/com/acme/MyPackage/17.9.3-SNAPSHOT/MyPackage-17.9.3-20170907.105908-1.pom"
}, {
"uri" : "https://artifactory.server/artifactory/api/storage/libs-devel/com/acme/MyPackage/17.9.3-SNAPSHOT/MyPackage-17.9.3-20170907.105908-1.war"
} ]
}
I would expect that the results would be limited to the the libs-devel repo, as specified in the url.
Have I done something wrong? Is this a bug?
Here's the API reference (for version 4) that I followed:
https://www.jfrog.com/confluence/display/RTF4X/Artifactory+REST+API#ArtifactoryRESTAPI-ArtifactSearch(QuickSearch)
Thanks in advance.
Yes. This is correct.
You will be able to limit the search query to specific repositories by adding the 'repos' parameter on this rest call.
We (JFrog) are not familiar with a bug on this subject and it is working on our side (Artifactory OSS - same version: 4.8.0).
We will be interested to know if the same is working for you using the UI quick search.
If using curl, the full URL needs to be encapsulated in quotes. Otherwise the & is left off the request.
I tried using Postman and it works as expected.
Bested by quote marks. :-(
According to below link, Artifactory AQL allows "Displaying of specific fields" via REST API by returning only fields of interest.
https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language#ArtifactoryQueryLanguage-DisplayingSpecificFields
It doesn't work if I provide a list of fields, see below
Not Work - Bad request (400)
items.find(...).include("name", "repo")
Works
items.find(...).include("*")
Can anyone advise
Thanks, Jag
I suspect that the problem is related to encoding during the REST call, therefore I suggest to upload the query as a file Here is a working example:
Save the following query to file, lets call it aql.query
items.find
(
{
"repo": {"$match":"*"}
}
)
.include("name","repo")
Run the following curl command from the same directory that contains the aql.query file and don't forget to replace the templates in the command with your user name, password, host and port.
curl -X POST -uuser:password 'http://host:port/artifactory/api/search/aql' -Taql.query
In the result you will get:
{
"results" :
[
{
"repo" : "ext-snapshot-local",
"name" : "maven-metadata.xml"
},{
"repo" : "ext-snapshot-local",
"name" : "multi-3.0.0-20150705.195404-1.pom"
},{
.
.
.
}
],
"range" :
{
"start_pos" : 0,
"end_pos" : 46,
"total" : 46
}
}
As you can see that the result contains only the "item repo" and the "item name" fields.
Had the same issue. Spent quite a bit of time trying to figure this out. Couldn't find an answer online.
With a bad request(400), I printed the response text: "For permissions reasons AQL demands the following fields: repo, path and name."
This solution worked for me -
at a minimum: have repo, path, name.
ie... items.find(...).include("name", "repo", "path", "created_by")
** UPDATE **
Thanks to Alfred Fuller for pointing out that I need to create a manual index for this query.
Unfortunately, using the JSON API, from a .NET application, there does not appear to be an officially supported way of doing so. In fact, there does not officially appear to be a way to do this at all from an app outside of App Engine, which is strange since the Cloud Datastore API was designed to allow access to the Datastore outside of App Engine.
The closest hack I could find was to POST the index definition using RPC to http://appengine.google.com/api/datastore/index/add. Can someone give me the raw spec for how to do this exactly (i.e. URL parameters, what exactly should the body look like, etc), perhaps using Fiddler to inspect the call made by appcfg.cmd?
** ORIGINAL QUESTION **
According to the docs, "a query can combine equality (EQUAL) filters for different properties, along with one or more inequality filters on a single property".
However, this query fails:
{
"query": {
"kinds": [
{
"name": "CodeProse.Pogo.Tests.TestPerson"
}
],
"filter": {
"compositeFilter": {
"operator": "and",
"filters": [
{
"propertyFilter": {
"operator": "equal",
"property": {
"name": "DepartmentCode"
},
"value": {
"integerValue": "123"
}
}
},
{
"propertyFilter": {
"operator": "greaterThan",
"property": {
"name": "HourlyRate"
},
"value": {
"doubleValue": 50
}
}
},
{
"propertyFilter": {
"operator": "lessThan",
"property": {
"name": "HourlyRate"
},
"value": {
"doubleValue": 100
}
}
}
]
}
}
}
}
with the following response:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "FAILED_PRECONDITION",
"message": "no matching index found.",
"locationType": "header",
"location": "If-Match"
}
],
"code": 412,
"message": "no matching index found."
}
}
The JSON API does not yet support local index generation, but we've documented a process that you can follow to generate the xml definition of the index at https://developers.google.com/datastore/docs/tools/indexconfig#Datastore_Manual_index_configuration
Please give this a shot and let us know if it doesn't work.
This is a temporary solution that we hope to replace with automatic local index generation as soon as we can.
The error "no matching index found." indicates that an index needs to be added for the query to work. See the auto index generation documentation.
In this case you need an index with the properties DepartmentCode and HourlyRate (in that order).
For gcloud-node I fixed it with those 3 links:
https://github.com/GoogleCloudPlatform/gcloud-node/issues/369
https://github.com/GoogleCloudPlatform/gcloud-node/blob/master/system-test/data/index.yaml
and most important link:
https://cloud.google.com/appengine/docs/python/config/indexconfig#Python_About_index_yaml to write your index.yaml file
As explained in the last link, an index is what allows complex queries to run faster by storing the result set of the queries in an index. When you get no matching index found it means that you tried to run a complex query involving order or filter. So to make your query work, you need to create your index on the google datastore indexes by creating a config file manually to define your indexes that represent the query you are trying to run. Here is how you fix:
create an index.yaml file in a folder named for example indexes in your app directory by following the directives for the python conf file: https://cloud.google.com/appengine/docs/python/config/indexconfig#Python_About_index_yaml or get inspiration from the gcloud-node tests in https://github.com/GoogleCloudPlatform/gcloud-node/blob/master/system-test/data/index.yaml
create the indexes from the config file with this command:
gcloud preview datastore create-indexes indexes/index.yaml
see https://cloud.google.com/sdk/gcloud/reference/preview/datastore/create-indexes
wait for the indexes to serve on your developer console in Cloud Datastore/Indexes, the interface should display "serving" once the index is built
once it is serving your query should work
For example for this query:
var q = ds.createQuery('project')
.filter('tags =', category)
.order('-date');
index.yaml looks like:
indexes:
- kind: project
ancestor: no
properties:
- name: tags
- name: date
direction: desc
Try not to order the result. After removing orderby(), it worked for me.