What does the below error mean in xquery :-
SVC-CODEPOINT: (err:FOCH0001) xdmp:unquote("{ "info": { "title": "Inventory...") -- Codepoint not legal
Getting the above error while running xquery file where I have defined a mapping file inside a variable like -
let $mapping := xdmp:unquote('{
"info": {
"title": "Inventory"
},
"InvTable": [
{
"prefix": "pdw",
"default": true
}
],
"ViewNames": {
"InvTable": {
"$ref": "pdw/DAU"
}
}
}')
It's basically saying you have an invalid character. A codepoint is the underlying numerical value for a character and there's something in your string that isn't legal to become JSON, probably. I suggest you simplify what you're unquoting until you find where that's coming from.
Related
I want to know the value of 'pageName' where the value of 'createdTs' is '333' in JSON like below, how should I use Jsonpath?
[
{
"Test1":{
"pageName":"Apple"
},
"Test1-1":{
"createdTs":"333"
}
},
{
"Test2":{
"pageName":"Tomato"
},
"Test2-1":{
"createdTs":"555"
}
}
]
.Test1-1[?(#.createdTs == 333)] would look like this:
[
{
"createdTs" : "333"
}
]
The parsing result I want is this.
[
{
"Test1":{
"pageName":"Apple"
},
"Test1-1":{
"createdTs":"333"
}
}
]
if you are using Jayway JSONPath a Java DSL for reading JSON documents, then you can use the these jsonpath expressions.
$[?(#.['Test1-1'].createdTs == 333)]
OR using in filter operator
$[?(333 in #..createdTs)]
OR using contains filter operator
$[?(#..createdTs contains 333)]
I'm having trouble validating a schema in Postman using tv4 inside the tests tab - it is always returning a true test, no matter what I feed it. I am at a complete loss and could really use a hand - here is my example JSON Response, and my tests:
I've tried a ton of variations from every Stack Overflow/tutorial I could find and nothing will work - it always returns true.
//Test Example
var jsonData = JSON.parse(responseBody);
const schema = {
"required" : ["categories"],
"properties": {
"categories": {
"required" : ["aStringOne", "aStringTwo", "aStringThree" ],
"type": "array",
"properties" : {
"aStringOne": {"type": "string" },
"aStringTwo": {"type": "null" },
"aStringThree": {"type": "boolean" }
}
}
}
};
pm.test('Schema is present and accurate', () => {
var result=tv4.validateMultiple(jsonData, schema);
console.log(result);
pm.expect(result.valid).to.be.true;
});
//Response Example
{
"categories": [
{
"aStringOne": "31000",
"aStringTwo": "Yarp",
"aStringThree": "More Yarp Indeed"
}
]
}
This should return false, as all three properties are strings but its passing. I'm willing to use a different validator or another technique as long as I can export it as a postman collection to use with newman in my CI/CD process. I look forward to any help you can give.
I would suggest moving away from using tv4 in Postman, the project isn't actively supported and Postman now includes a better (in my opinion), more actively maintained option called Ajv.
The syntax is slightly different but hopefully, this gives you an idea of how it could work for you.
I've mocked out your data and just added everything into the Tests tab - If you change the jsonData variable to pm.response.json() it will run against the actual response body.
var jsonData = {
"categories": [
{
"aStringOne": "31000",
"aStringTwo": "Yarp",
"aStringThree": "More Yarp Indeed"
}
]
}
var Ajv = require('ajv'),
ajv = new Ajv({logger: console, allErrors: true}),
schema = {
"type": "object",
"required": [ "categories"],
"properties": {
"categories": {
"type": "array",
"items": {
"type": "object",
"required": [ "aStringOne", "aStringTwo", "aStringThree" ],
"properties": {
"aStringOne": { "type": "string" },
"aStringTwo": { "type": "integer"},
"aStringThree": { "type": "boolean"},
}
}
}
}
}
pm.test('Schema is valid', function() {
pm.expect(ajv.validate(schema, jsonData), JSON.stringify(ajv.errors)).to.be.true
});
This is an example of it failing, I've included the allErrors flag so that it will return all the errors rather than just the first one it sees. In the pm.expect() method, I've added JSON.stringify(ajv.errors) so you can see the error in the Test Result tab. It's a little bit messy and could be tidied up but all the error information is there.
Setting the properties to string show the validation passing:
If one of the required Keys is not there, it will also error for this too:
Working with schemas is quite difficult and it's not easy to both create them (nested arrays and objects are tricky) and ensure they are doing what you want to do.
There are occasions where I thought something should fail and it passed the validation test. It just takes a bit of learning/practising and once you understand the schema structures, they can become extremely useful.
I have an index in ElasticSearch with two fields of date type (metricsTime & arrivalTime). A sample document is quoted below. In Kibana, I created a scripted field delay for the difference between those two fields. My painless script is:
doc['arrivalTime'].value - doc['metricsTime'].value
However, I got the following error message when navigating to Kibana's Discover tab: class_cast_exception: Cannot apply [-] operation to types [org.joda.time.MutableDateTime] and [org.joda.time.MutableDateTime].
This looks same as the error mentioned in https://discuss.elastic.co/t/problem-in-difference-between-two-dates/121655. But the answer in that page suggests that my script is correct. Could you please help?
Thanks!
{
"_index": "events",
"_type": "_doc",
"_id": "HLV274_1537682400000",
"_version": 1,
"_score": null,
"_source": {
"metricsTime": 1537682400000,
"box": "HLV274",
"arrivalTime": 1539930920347
},
"fields": {
"metricsTime": [
"2018-09-23T06:00:00.000Z"
],
"arrivalTime": [
"2018-10-19T06:35:20.347Z"
]
},
"sort": [
1539930920347
]
}
Check the list of Lucene Expressions to check what expressions are available for date field and how you could use them
Just for sake of simplicity, check the below query. I have created two fields metricsTime and arrivalTime in a sample index I've created.
Sample Document
POST mydateindex/mydocs/1
{
"metricsTime": "2018-09-23T06:00:00.000Z",
"arrivalTime": "2018-10-19T06:35:20.347Z"
}
Query using painless script
POST mydateindex/_search
{ "query": {
"bool": {
"must": {
"match_all": {
}
},
"filter": {
"bool" : {
"must" : {
"script" : {
"script" : {
"inline" : "doc['arrivalTime'].date.dayOfYear - doc['metricsTime'].date.dayOfYear > params.difference",
"lang" : "painless",
"params": {
"difference": 2
}
}
}
}
}
}
}
}
}
Note the below line in the query
"inline" : "doc['arrivalTime'].date.dayOfYear - doc['metricsTime'].date.dayOfYear > params.difference"
Now if you change the value of difference from 2 to 26 (which is one more than the difference in the dates) then you see that the above query would not return the document.
But nevertheless, I have mentioned the query as an example as how using scripting you can compare two different and please do refer to the link I've shared.
I have a set of timestamps formatted as seconds since the epoch. I'd like to insert to ElasticSearch as epoch_seconds but when querying would like to see the output as a pretty date, e.g. strict_date_optional_time.
My below mapping preserves the format that the input came in - is there any way to normalize the output to just one format via the mapping api?
Current Mapping:
PUT example
{
"mappings": {
"time": {
"properties": {
"time_stamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_second"
}
}
}
}
}
Example docs
POST example/time
{
"time_stamp": "2018-03-18T00:00:00.000Z"
}
POST example/time
{
"time_stamp": "1521389162" // Would like this to output as: 2018-03-18T16:05:50.000Z
}
GET example/_search output:
{
"total": 2,
"max_score": 1,
"hits": [
{
"_source": {
"time_stamp": "1521389162", // Stayed as epoch_second
}
},
{
"_source": {
"time_stamp": "2018-03-18T00:00:00.000Z"
}
}
]
}
Elasticsearch differentiates between the _source and the so called stored fields. The first one is supposed to represent your input.
If you actually use stored fields (by specifying store=true in your mapping) then specify multiple date formats this is easy: (emphasis mine)
Multiple formats can be specified by separating them with || as a separator. Each format will be tried in turn until a matching format is found. The first format will be used to convert the milliseconds-since-the-epoch value back into a string.
I have tested this with elasticsearch 5.6.4 and it works fine:
PUT /test -d '{ "mappings": {"doc": { "properties": {"post_date": {
"type":"date",
"format":"basic_date_time||epoch_millis",
"store":true
} } } } }'
PUT /test/doc/2 -d '{
"user" : "test1",
"post_date" : "20150101T121030.000+01:00"
}'
PUT /test/doc/1 -d '{
"user" : "test2",
"post_date" : 1525167490500
}'
Note how two different input-formats will result in the same format when using GET /test/_search?stored_fields=post_date&pretty=1
{
"hits" : [
{
"_index" : "test",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"fields" : {
"post_date" : [
"20150101T111030.000Z"
]
}
},
{
"_index" : "test",
"_type" : "doc",
"_id" : "1",
"_score" : 1.0,
"fields" : {
"post_date" : [
"20180501T093810.500Z"
]
}
}
]
}
If you want to change the input (in _source) you're not so lucky, the mapping-transform feature has been removed:
This was deprecated in 2.0.0 because it made debugging very difficult. As of now there really isn’t a feature to use in its place other than transforming the document in the client application.
If, instead of changing the stored data you are interested in formatting the output, have a look at this answer to Format date in elasticsearch query (during retrieval)
I want to have definitions section better generated and organized and for me this would mean to not allow depth building up in definitions, but, each class involved in the structure tree should have its own entry in the definitions section and referenced via $ref. This means that for each definition I would only have a list of properties that would either be of primitive types (string, boolean, etc.) or would be a $ref to another definition entry for another custom class. You can also see this as depth 1 definition, close to how classes are originally defined in C#.
To illustrate this via a trivial example:
JSchemaGenerator schemaGenerator = new JSchemaGenerator();
schemaGenerator = new JSchemaGenerator()
{
DefaultRequired = Newtonsoft.Json.Required.DisallowNull,
SchemaIdGenerationHandling = SchemaIdGenerationHandling.TypeName,
SchemaLocationHandling = SchemaLocationHandling.Definitions,
SchemaReferenceHandling = SchemaReferenceHandling.Objects,
};
JSchema schema = schemaGenerator.Generate(typeof(Setting));
Renders:
{
"id": "Setting",
"definitions": {
"SubSetting": {
"id": "SubSetting",
"type": "object",
"properties": {
"SubSubSetting": {
"id": "SubSubSetting",
"type": "object",
"properties": {
"String": {
"type": "string"
}
}
}
}
},
"SubSubSetting": {
"$ref": "SubSubSetting"
}
},
"type": "object",
"properties": {
"SubSetting": {
"$ref": "SubSetting"
},
"SubSubSetting": {
"$ref": "SubSubSetting"
}
}
}
Thus, SubSubSetting definition is placed inline SubSetting definition and later we have SubSubSetting defined as reference to that inline definition. That's what I want to avoid as for complex data structures it becomes really obscure and I want to provide the schema even as part of a living, auto-generated documentation based on data annotations and JsonProperty.
How can I accomplish this using JSchemaGenerator?
Maybe I shouldn't do this, but as a second very short question: Are those $ref syntactically correct? Shouldn't they look like "#/definitions/SubSetting"?
The latest version of Json.NET Schema (3.0.3) has been updated to fix this issue. SubSubSetting will contain the full definition and not just a $ref.
https://github.com/JamesNK/Newtonsoft.Json.Schema/releases/tag/3.0.3