I have a query that creates a vertex and an edge "Created". The edge has a property "on" which is unix datetime long. When I execute a query with the following segment in the Azure Cosmos DB terminal, it works as expected - an object is returned with an "On" property that is a number.
.project('Id', 'CreatedOn')
.by('id')
.by(
select('createdEdge')
.by('on')
)
When I execute this query from my application code using Gremlin.NET, it fails with the error:
JSON type not supported.
I see in the source code that the deserialization logic of Gremlin.NET does not seem to handle any number types. Is this really the case? Is there no way to use long, float, int property types?
Gremlin.NET does not seem to handle any number types. Is this really the case? Is there no way to use long, float, int property types?
Gremlin.NET does of course support serialization of numerical types. TinkerPop however has its own serialization formats, one of which is the JSON based GraphSON format that is also supported by Cosmos DB.
GraphSON basically serializes everything as an object in JSON that consists of a type key and the value. So, an integer will be serialized like this:
{
"#type" : "g:Int32",
"#value" : 100
}
This type identifier was added with GraphSON 2 and only used for really all types in GraphSON 3. GraphSON 2 still serialized some types without this type identifier. The GraphSON 2 spec describes this like this:
With GraphSON 2.0, there are essentially two type formats:
A non-typed value which is assumed the type implied by JSON. These non-types are limited to String, Boolean, Map and Collection.
All other values are typed by way of a "complex object" that defines a #typeId and #value.
As you can see, numerical types are not listed as types that don't have the type identifier.
The Cosmos DB docs mention that they only support GraphSON 2, but it looks like they serialize numerical types without this type identifier. This used to work with Gremlin.NET versions < 3.5.0 as that still had the logic to deserialize numerical types without a type identifier. This was simply a leftover from the GraphSON 1 format which didn't have these type identifiers.
Gremlin.NET 3.5.0 came with major changes to the JSON serialization, mostly due to the switch from Newtonsoft.JSON to System.Text.Json, and this logic was removed as part of that change.
So, it looks like you need to stay on Gremlin.NET 3.4 until Cosmos DB fixes their serialization.
Note there is a work-around as discussed here: https://www.mail-archive.com/dev#tinkerpop.apache.org/msg22532.html
In short, you can create an custom reader derived from GraphSON2Reader:
public class CustomGraphSON2Reader : GraphSON2Reader
{
public override dynamic ToObject(JsonElement graphSon) =>
graphSon.ValueKind switch
{
// numbers
JsonValueKind.Number when graphSon.TryGetInt32(out var intValue) => intValue,
JsonValueKind.Number when graphSon.TryGetInt64(out var longValue) => longValue,
JsonValueKind.Number when graphSon.TryGetDecimal(out var decimalValue) => decimalValue,
_ => base.ToObject(graphSon)
};
}
and then pass it into your client:
var client = new GremlinClient(server, new GraphSON2MessageSerializer(new CustomGraphSON2Reader()));
Related
I am actually looking for a way to validate elements in json. I thought there is a way to list them out to strictly avoid accepting wrong elements. For instance instead of "gender": "male" as illustrated below, someone could send "sex": "male" and I am trying to avoid it.
I have a data field (column) called Profile
profile = {'name': 'Payne', 'gender': 'male', 'favourites': [{'drinks': 'soda'}, {'colour': 'blue'}, {'game': 'scrabble'}], 'dob': '1962'}
I am using a third party API to populate the database using HttpClient.
My response is returning JSON and I want to make some decisions with it and store it in the database but I need to validate it in conformity with what is expected strictly.
About validation:
If you know how to do it with arrays, you can still decode it and validate it as an array, then encode it again. Symfony has a validator service but I do not know exactly how to correctly use it in all cases.
The official Symfony documentation for the Validator Service and how to use it can be found in this link and it's anchor:
https://symfony.com/doc/current/validation.html
https://symfony.com/doc/current/validation.html#using-the-validator-service
Some info about JSON in PHP:
The most typical use for json, even for updating and downloading content to and from a DataBase, is the use of json_encode() and json_decode(), which will basically help you make your arrays more "portable" in different use cases.
For example, in MySQL, which accepts the JSON type, you can insert arrays into a column by encoding them with JSON.
If you want to declare a JSON type variable in Symfony, you can do it as in this example:
/**
* #ORM\Column(type="json", ...)
*/
private $yourColumn;
Both json_encode() and json_decode() are available since PHP v5.2.0
As an example of a DB, the JSON type was added to MySQL in version 5.7.8 (https://dev.mysql.com/doc/refman/5.7/en/json.html)
You should take a look at these links:
https://www.php.net/manual/es/function.json-encode.php
https://www.php.net/manual/es/function.json-decode.php
https://www.w3schools.com/js/js_json_php.asp
https://dev.mysql.com/doc/refman/8.0/en/
We have an existing CosmosDb collection that we use with the SQL API. We want to start using the Graph API against this collection. Using the Microsoft.Azure.Graphs NUGET version 0.3.1-preview and performing the simple Gremlin query g.V() throws an Invalid Cast Exception:
Invalid cast from 'System.String' to 'Newtonsoft.Json.Linq.JObject'.
After lots of trial and error, we narrowed it down to a property array of string values:
{
"Roles": [ "user", "admin" ]
}
Converting this property to an object or removing this property fixes the issue, of course that is not a viable solution.
Is this a bug on the Graph SDK, on the Cosmos Db engine, on Gremlin, or simply not doable in Graph in general?
How do we solve this issue?
I've been working with JSON.net for a while. I have written both custom converters and custom contract resolvers (generally from modifying examples on S.O. and the Newtonsoft website), and they work fine.
The challenge is, other than examples, I see little explanation as to when I should use one or the other (or both) for processing. Through my own experience, I've basically determined that contract resolvers are simpler, so if I can do what I need with them, I go that way; otherwise, I use custom JsonConverters. But, I further know both are sometimes used together, so the concepts get further opaque.
Questions:
Is there a source that distinguishes when to user one vs. the other? I find the Newtonsoft documentation unclear as to how the two are differentiated or when to use one or the other.
What is the pipeline of ordering between the two?
Great question. I haven't seen a clear piece of documentation that says when you should prefer to write a custom ContractResolver or a custom JsonConverter to solve a particular type of problem. They really do different things, but there is some overlap between what kinds of problems can be solved by each. I've written a fair number of each while answering questions on StackOverflow, so the picture has become a little more clear to me over time. Below is my take on it.
ContractResolver
A contract resolver is always used by Json.Net, and governs serialization / deserialization behavior at a broad level. If there is not a custom resolver provided in the settings, then the DefaultContractResolver is used. The resolver is responsible for determining:
what contract each type has (i.e. is it a primitive, array/list, dictionary, dynamic, JObject, plain old object, etc.);
what properties are on the type (if any) and what are their names, types and accessibility;
what attributes have been applied (e.g. [JsonProperty], [JsonIgnore], [JsonConverter], etc.), and
how those attributes should affect the (de)serialization of each property (or class).
Generally speaking, if you want to customize some aspect of serialization or deserialization across a wide range of classes, you will probably need to use a ContractResolver to do it. Here are some examples of things you can customize using a ContractResolver:
Change the contract used for a type
Serialize all Dictionaries as an Array of Key/Value Pairs
Serialize ListItems as a regular object instead of string
Change the casing of property names when serializing
Use camel case for all property names
Camel case all property names except dictionaries
Programmatically apply attributes to properties without having to modify the classes (particularly useful if you don't control the source of said classes)
Globally use a JsonConverter on a class without the attribute
Remap properties to different names defined at runtime
Allow deserializing to public properties with non-public setters
Programmatically unapply (ignore) attributes that are applied to certain classes
Optionally turn off the JsonIgnore attribute at runtime
Make properties which are marked as required (for SOAP) not required for JSON
Conditionally serialize properties
Ignore read-only properties across all classes
Skip serializing properties that throw exceptions
Introduce custom attributes and apply some custom behavior based on those attributes
Encrypt specially marked string properties in any class
Selectively escape HTML in strings during deserialization
JsonConverter
In contrast to a ContractResolver, the focus of a JsonConverter is more narrow: it is really intended to handle serialization or deserialization for a single type or a small subset of related types. Also, it works at a lower level than a resolver does. When a converter is given responsibility for a type, it has complete control over how the JSON is read or written for that type: it directly uses JsonReader and JsonWriter classes to do its job. In other words, it can change the shape of the JSON for that type. At the same time, a converter is decoupled from the "big picture" and does not have access to contextual information such as the parent of the object being (de)serialized or the property attributes that were used with it. Here are some examples of problems you can solve with a JsonConverter:
Handle object instantiation issues on deserialization
Deserialize to an interface, using information in the JSON to decide which concrete class to instantiate
Deserialize JSON that is sometimes a single object and sometimes an array of objects
Deserialize JSON that can either be an array or a nested array
Skip unwanted items when deserializing from an array of mixed types
Deserialize to an object that lacks a default constructor
Change how values are formatted or interpretted
Serialize decimal values as localized strings
Convert decimal.MinValue to an empty string and back (for use with a legacy system)
Serialize dates with multiple different formats
Ignore UTC offsets when deserializing dates
Make Json.Net call ToString() when serializing a type
Translate between differing JSON and object structures
Deserialize a nested array of mixed values into a list of items
Deserialize an array of objects with varying names
Serialize/deserialize a custom dictionary with complex keys
Serialize a custom IEnumerable collection as a dictionary
Flatten a nested JSON structure into a simpler object structure
Expand a simple object structure into a more complicated JSON structure
Serialize a list of objects as a list of IDs only
Deserialize a JSON list of objects containing GUIDs to a list of GUIDs
Work around issues (de)serializing specific .NET types
Serializing System.Net.IPAddress throws an exception
Problems deserializing Microsoft.Xna.Framework.Rectangle
I have made a gridx grid that uses a JsonRest Memory store from the dojo framework
http://dojotoolkit.org/reference-guide/1.10/dojo/store/JsonRest.html
the issue is I do not know how to pull out the sort parameter from the query string.
The url being formatted from the JsonRest call is
/admin/sales?sort(+DealershipName)
using the following statement gives me a null error
String sort = Request.QueryString["sort"].ToString();
Looking at the debugger I see the following (I need more rep to post images :( )
ok I can see that the following variables hold this value.
Request.QueryString = {sort(+DealershipName)}
type : System.Collections.Specialized.NameValueCollection
{System.Web.HttpValueCollection}
but the array is null.
I'm thinking I can do two thing. Parse the string myself or overload the dojo JsonRest Memory store. Parsing the string seems easier but if anyone has any idea or knows any libraries that can help me out. I would greatly appreciate it.
dojo/store/JsonRest has a sortParam property that you can set to the name of a standard query parameter to use instead of sort(...) (which it uses by default to avoid colliding with any standard query parameters).
For example, adding sortParam: 'sort' to the properties passed to the JsonRest constructor will result in the query string including sort=+DealershipName instead.
http://dojotoolkit.org/reference-guide/1.10/dojo/store/JsonRest.html#sorting
If the + also presents a problem, you can also override ascendingPrefix to be an empty string (''). Note that descending sort will still be indicated by a leading - (controllable via descendingPrefix).
I am not sure if this is a JSON.Net issue or a SolrNet issue, which is why I am tagging it with both.
I am using SolrNet and currently handling all page interactivity with JavaScript from an MVC call, by returning a JsonNetResult encoding of the object returned by solr.Query(). I am now experimenting with Faceting, but am not seeing the Facet info in my results.
I've got an MVC Controller method like the one below. (JsonNetResult is similar to the stock JsonResult, but encodes JSON using JSON.Net, described here.)
public JsonNetResult Index(string keywords)
{
JsonNetResult jsonNetResult = new JsonNetResult();
var documents = solr.Query(new SolrQuery(keywords), new QueryOptions
{
Rows = 10,
Facet = new FacetParameters
{
Queries = new[] {new SolrFacetFieldQuery("system")}
}
});
jsonNetResult.Formatting = Formatting.Indented;
jsonNetResult.Data = documents;
return jsonNetResult;
}
I was expecting to see the faceting information encoded into JSON in the JsonNetResult, but all it contains is the array of hashes of the documents matching my query. Am I missing something in how SolrNet response objects work, or do I really need to parse through the object myself, and create something that JSON.Net can use to encode all of the information related to my query?
FYI, I have tried using a standard JsonResult in MVC, and the results are the same. Also, the reason I am using SolrNet and not just calling Solr directly and asking for JSON is because we do not want to expose the Solr search engine web interface directly to the user.
Since Solr can respond with JSON, if you want to return JSON directly to the view you'd be incurring some unnecessary overhead by having SolrNet deserialize a XML response, then serialize it to JSON. Instead, use SolrNet's components to skip the response parser. A couple of pointers to do this:
SolrQueryExecuter.GetAllParameters()
Low-level SolrNet
I am not sure if this is the best answer, but I have since been experimenting and found that if I change my original line from:
jsonNetResult.Data = documents;
to:
jsonNetResult.Data = new { Documents = documents, FacetFields = documents.FacetFields };
The data is at least serialized by JSON.Net. I guess I still don't understand the format of the object being returned by SolrNet's Query() method, since it seems like those properties (like FacetFields) should be serialized without me having to explicitly name them like I'm now doing above?