I'm using Gremlin 3.0.2 together with Titan 1.0.0.
The request I send to the Gremlin Server will return a list of nodes and their properties. Effectively, it's a list of items like the following:
[coverurl:[https://lh3.googleusercontent.com/RYb-duneinq8ClWVLVKknkIx1jOKm64LjreziFApEnkKME8j9tHNDRi9NMA6PK4PTXO7], appname:[Slack], pkgid:[com.Slack]]
In one case, a request will return 38 items like the one above and everything is fine. In another case, the list would contain 56 of these items and I get the following exception:
WARN org.apache.tinkerpop.gremlin.driver.MessageSerializer - Response [PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.
ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler - Could not process the response
io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.kryo.KryoException: Buffer too small: capacity: 0, required: 1
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
This problem has been discussed here last year. However, for different versions of Titan and for writing data to Titan, instead of reading like it is the case here.
I don't see any programmatic way to adapt the buffer size of the (De)serializer so what is the preferred way to deal with this problem? Also, setting some limit (which?, where?) to some higher value can only be a temporary solution since I never know how much data a request will return.
Anyways - The amount of data I receive is fairly small (probably a little more than 8500 Bytes). I'm surprised that this exception is thrown at all?
Titan 1.0 is based on TinkerPop 3.0.1... are you building Titan on your own?
TINKERPOP-817 introduces a fix that allows a bufferSize parameter to be configured. As Stephen mentioned in the comments:
the kryo buffer size was defaulted to 4096 and would those throw that
"Buffer too small" exception
The fix went into TinkerPop 3.0.2 and is documented here.
In order to use this, you'll need to upgrade your Titan Server to run with TinkerPop 3.0.2, and it would be best to recompile from source after modifying the tinkerpop.version in the Titan pom.xml. Find the Titan build directions here. Alternately, you could consider building the titan11 branch for the latest available fixes and TinkerPop 3.1.1 (Hadoop 2 support!).
Next, you will need to configure the bufferSize on the appropriate serializer in the gremlin-server.yaml configuration. I do not think you cannot fix this problem with a client configuration only.
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { bufferSize: 8192, useMapperFromGraph: graph }} # application/vnd.gremlin-v1.0+gryo
Since you have a Java client and you're expecting to work directly with the Vertex objects, perhaps you could consider doing a direct connection to Titan and avoid this serialization completely.
Related
I've sent a gremlin statement which fails with the following error:
{"code":"InternalFailureException","requestId":"d924c667-2c48-4a7e-a288-6f23bac6283a","detailedMessage":"null: .property(Cardinality, , String)"} (599)
I've enabled audit logs on the cluster side but there aren't any indications for any errors there although I see the request.
Are there any technics to debug such problems with AWS Neptune?
Since the gremlin is built dynamically and builds complicated graphs with thousands of operations, I'm looking for some way to understand better where the error is.
In one case it turned out the payload was too big and in another, the gremlin bytecode failed although it worked great on the local Tinkerpop Server.
But with the generic internal failure exception it is very hard to pinpoint the issues.
I'm using the Node.js gremlin package (3.5.0).
Thanks for the clarifications. As a side note, Neptune is not yet at the 3.5.0 TinkerPop level so there is always the possibility of mismatches between client and server. The audit log only shows the query received. It does not reflect the results of running the query. When I am debugging issues like this one, I often create a text version of the query (the node client has a translator that can do that for you) and use the Neptune REST API to first of all check the /explain plan - you can also generate that from the Neptune notebooks using %%gremlin explain. If nothing obvious shows up in the explain plan I usually try to run the text version of the query just to see if anything shows up - to rule out the client driver. I also sometimes run my query against a TinkerGraph just to see if the issue is as simple as Neptune not returning the best error message it could. If the text query should work, changing the Node.js Gremlin client driver to 3.4.11 is worth trying.
UPDATED to add a link to the documentation on Gremlin translators.
https://tinkerpop.apache.org/docs/current/reference/#translators
As mentioned in the docs on performing flow upgrades, all you need to do is basically shut down the node, replace JAR, and start the node back up. When I do this, when my upgraded flow is run the next time, I get the following error:
net.corda.core.contracts.TransactionVerificationException$ContractConstraintRejection: Contract constraints failed for com.company.project.contract.MyContract, transaction: ABCDEFG
And the flow does not complete as a result. What am I doing wrong?
As my experience it seem like Corda flow upgrade not update network parameter (state still belong to old hash, old contract). Then when replace with new contract it will be contract constraint.
So I think you have 3 way to manage this
For local network bootstrap, update network parameter before doing flow upgrade (I use network-bootstrap.jar for copy new contract to cordapp folder, then it will append new contract hash immediately)
For Corda network, you must contact network operator for update new hash.
Use SignatureConstraint of Corda4 (they claim that it's upgrade easier but I didn't try yet)
Hope this help
Recently, when I wanted to sign something with a certificate outside node, I got the below exceptions:
Caused by: java.lang.IllegalStateException: Expected exactly 1 of
{nodeSerializationEnv, globalSerializationEnv,
contextSerializationEnv, inheritableContextSerializationEnv} but got:
{}
https://github.com/corda/corda/blob/671a9d232cf1f29dbce4432bc91096ffd098a91c/core/src/main/kotlin/net/corda/core/serialization/internal/SerializationEnvironment.kt#L91
On debugging, I found it's first serialised and then signed. So, I had to set up the serialisation context to get it serialised and signed.
I have a limited understanding of why is it required. I understand that different contexts are required for P2P and RPC calls, but I'm not entirely sure can someone please fill me in with some background.
The internal library you are using to sign the certificate requires the certificate to be serialised first. In turn, this requires you to specify a serialisation context. A serialisation contexts defines how serialisation is performed in various situations, such as P2P, client-side RPC, server-side RPC, storage and checkpointing.
Note that these serialisation contexts are set for you automatically when running a node or a suite of node tests. You are only encountering this issue because you are using an internal library outside the context where it is expected to be used.
In your case, you should probably use globalSerializationEnv, which is the serialisation environment used for mock nodes and nodes created using the node driver. nodeSerializationEnv is used by the node itself, and contextSerializationEnv and inheritableContextSerializationEnv are used for various platform tests.
For educational purposes, it can be helpful to look at how the node sets up its serialisation framwork when it starts up (see https://github.com/corda/corda/blob/release-V3/node/src/main/kotlin/net/corda/node/internal/Node.kt):
nodeSerializationEnv = SerializationEnvironmentImpl(
SerializationFactoryImpl().apply {
registerScheme(KryoServerSerializationScheme())
registerScheme(AMQPServerSerializationScheme(cordappLoader.cordapps))
},
p2pContext = AMQP_P2P_CONTEXT.withClassLoader(classloader),
rpcServerContext = KRYO_RPC_SERVER_CONTEXT.withClassLoader(classloader),
storageContext = AMQP_STORAGE_CONTEXT.withClassLoader(classloader),
checkpointContext = KRYO_CHECKPOINT_CONTEXT.withClassLoader(classloader))
I used for > 1.5 years to use trackEvent in AI nodejs package with a property with various payload text.
until now, I was able to send large payload (sent many time >53k length string)
suddenly, starting on 18th of october 2018, all string longer than 1024 chars are truncated.
I looked into github repo and or new release and I can't understand if expected or not ...
AI "version": "1.0.6"
I suspect that version 1.0.6 has this new limitation ?
Yes, this is new in 1.0.6 and appears to be a bug.
Context:
The Application Insights SDK for Node had always intended to truncate custom dimensions to 1024 chars, but this logic was broken in some cases. As part of the changes in 1.0.6 to support nested objects in custom dimensions, the truncation logic was fixed.
We perform this truncation because your telemetry has a chance of being dropped entirely by the Application Insights backend if custom dimensions are longer than the limit specified in the schema. However, the limit in the schema is 8192 rather than 1024.
I've opened a bug to track fixing this: https://github.com/Microsoft/ApplicationInsights-node.js/issues/444
I have a StoredProcedure that returns a simple table containing several records:
DECLARE #STEPS_TABLE AS TABLE (OrchestrationID uniqueidentifier, [Message] nvarchar(1000));
-- LOADING THE VALUES HERE
SELECT * FROM #STEPS_TABLE As Step FOR XML AUTO, XMLDATA, ELEMENTS
I used the SQL Transport Schema Generation Wizard to create my schema and could configure the port correctly. If I use this schema on my orchestration, it works perfectly. BizTalk starts one instance of the orchestration everytime the #STEPS_TABLE has more than 1 record.
Reading Microsoft technical documentation, they recommend to get several messages in one call and then use the XML pipeline to disassemble the multi-row BizTalk message into a single-row BizTalk message.
I haven't used the XML pipeline before, so I tried the provided steps but couldn't get it to work.
Could somebody provide me a link to a "how to" (didn't find anything until now, after several hours of searching) or give me some hints to succeed.
Thanks in advance.
... some hours later I could figure it out myself. So if anybody comes across the same issue as me, here you have some guidelines to make it work on your environment.
At the end I followed a different walkthrough from Microsoft and avoided the pipeline recommendation altogether. The documentation I found is called "Disassembling Result Sets Using the SQL Adapter" and does exactly what i was looking for. You can just follow the whole walkthrough from Microsoft but avoid the creation of the send port and make some little adjustment on the receive port.
After following the technical document you will end up with two schemas, I will call them message and envelope (contains several messages) for the sake of this excercise. In your orchestration you can create a receiving port that maps to the message and then when you configure it as a SQL Port and you link it to your stored procedure (or select statement), you only have to change the Document Root Element Name to the envelope root name; the XML Receive pipeline (provided by default in BizTalk 2006) will do the magic of disassembling the messages contained in the envelope and instantiating an orchestration for each message.
The Microsoft "Disassembling Result Sets Using the SQL Adapter" walkthrough can be found under:
http://msdn.microsoft.com/en-us/library/aa562098(v=bts.20).aspx
Mission accomplished :)