tinkerpop3 Gremlin java client/cluster with org.apache.tinkerpop.gremlin.structure.Graph integration - gremlin

As known from tinkerpop doc, we can use code to connect remote graph db
cluster = Cluster.build(addr).serializer(Serializers.DEFAULT_RESULT_SERIALIZER)
.credentials(username, password)
.enableSsl(true)
.port(port).create();
client = cluster.connect();
client.submit("g.E().drop()").all().join();
client.submit("g.V().drop()").all().join();
client.submit("g.addV('person').property('id', '1').property('name', 'pli').property('age', 31)").all().join();
but there is another interface from org.apache.tinkerpop.gremlin.structure.Graph with code
Graph graph = EmptyGraph.instance();
GraphTraversalSource g = graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
g.E().drop();
g.V().drop();
g.addV("web").property("id", "2").properties("name", "github");
Can I integrate these two part for writing query to graph database? instead of use gremlin string literal for query?
Thanks in advance!
Update 1 with one new try with following code
client = cluster.connect().alias(new HashMap<String, String>());
Graph graph = EmptyGraph.instance();
GraphTraversalSource g = graph.traversal().withRemote(DriverRemoteConnection.using(cluster, "g"));
GraphTraversal traversal = g.E().drop().V().drop().addV("web").property("id", "2").properties("name", "github");
client.submit(traversal).all().join();
And it will hang at last client.submit in one native function ->
private native int poll0(long var1, int var3, int[] var4, int[] var5, int[] var6, long var7);
I guess it may need some specific configuration for alias hashmap ?
Update 2
Thanks for you clearly answers. I have a try on traversal this weekend with following code but still got hang when traversal.next/traversal.hasNext()
private void prepareEnvironment() {
this.cluster = Cluster.build(this.config.getEndpoint())
.serializer(Serializers.DEFAULT_RESULT_SERIALIZER)
.enableSsl(true)
.port(Integer.valueOf(this.config.getPort()))
.credentials(this.config.getUsername(), this.config.getPassword())
.create();
// this.client = this.cluster.connect(Constants.GREMLIN_ALIAS, true);
// this.client = this.cluster.connect();
this.graph = EmptyGraph.instance();
this.graphSource = this.graph.traversal().withRemote(DriverRemoteConnection.using(cluster));
}
#Before
public void setup() {
this.prepareEnvironment();
// client.submit("g.V().drop()").all().join();
GraphTraversal graphTraversal = this.graphSource.V().drop();
Object o = graphTraversal.next();
// while (graphTraversal.hasNext()) {
// }
}
I capture some log of console, and It seems it loops on http response here.
<submit from client> -> 23:05:38.559 [main] DEBUG org.apache.tinkerpop.gremlin.driver.Client - Submitted RequestMessage{, requestId=6dd55134-2266-4b23-a311-9d25b5b51dc0, op='eval', processor='', args={batchSize=64, gremlin=g.V().drop()}} to - Connection{host=Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin}}
<submit from traversal> -> 23:07:54.823 [main] DEBUG org.apache.tinkerpop.gremlin.driver.Client - Submitted RequestMessage{, requestId=1a7f550e-eb7f-497d-8b4b-1fbf40beb99e, op='bytecode', processor='traversal', args={gremlin=[[], [V(), drop()]], aliases={g=g}}} to - Connection{host=Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin}}
<submit from traversal log>
23:16:16.708 [main] DEBUG org.apache.tinkerpop.gremlin.driver.ConnectionPool - Borrowing connection from pool on Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin} - timeout in 3000 MILLISECONDS
23:16:16.708 [main] DEBUG org.apache.tinkerpop.gremlin.driver.ConnectionPool - Return least used Connection{host=Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin}, isDead=false, borrowed=1, pending=0} on Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin}
23:16:16.724 [main] DEBUG org.apache.tinkerpop.gremlin.driver.Client - Submitted RequestMessage{, requestId=c50cef01-270e-469a-8423-838fe67565d4, op='bytecode', processor='traversal', args={gremlin=[[], [V(), drop()]], aliases={g=g}}} to - Connection{host=Host{address=pli-gremlin-test.gremlin.cosmosdb.azure.com/104.45.144.73:443, hostUri=wss://pli-gremlin-test.gremlin.cosmosdb.azure.com:443/gremlin}}
23:16:16.833 [gremlin-driver-loop-1] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameEncoder - Encoding WebSocket Frame opCode=2 length=360
23:16:20.432 [gremlin-driver-loop-1] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=8
23:16:20.447 [gremlin-driver-loop-1] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame length=23
23:16:45.197 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=10
23:16:45.197 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame length=0
23:16:45.197 [gremlin-driver-loop-2] DEBUG org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler - Received response from keep-alive request
23:17:15.191 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=10
23:17:15.191 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame length=0
23:17:15.191 [gremlin-driver-loop-2] DEBUG org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler - Received response from keep-alive request
23:17:45.215 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=10
23:17:45.215 [gremlin-driver-loop-2] DEBUG io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder - Decoding WebSocket Frame length=0
23:17:45.215 [gremlin-driver-loop-2] DEBUG org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler - Received response from keep-alive request

Can I integrate these two part for writing query to graph database? instead of use gremlin string literal for query?
That's precisely what withRemote() does. It compiles the traversals you write into Gremlin bytecode, sends that to the server and then returns the results.
Or is there any way to convert Graph/GraphTraversalSource to gremlin query String literal?
you can do that too, but it isn't really recommended. Prefer bytecode over strings, but if you absolutely had to do it AND if your traversal does not include lambdas you can do:
gremlin> translator = org.apache.tinkerpop.gremlin.groovy.jsr223.GroovyTranslator.of("g")
==>translator[g:gremlin-groovy]
gremlin> translator.translate(g.V().out('knows').has('name','josh').asAdmin().getBytecode())
==>g.V().out("knows").has("name","josh")
GroovyTranslator is in the gremlin-groovy module.

Related

Spring cloud Stream - kafka - Null Acknowledgement Header

I want to manually Commit the offset using spring cloud stream - only when the message processing is successful.
Here is my code - application.yml & Handler Class
public void process(Message<?> message) {
System.out.println(message.getPayload());
Acknowledgment acknowledgment = message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment.class);
if (acknowledgment != null) {
System.out.println("Acknowledgment provided");
acknowledgment.acknowledge();
}
}
---------------------------------------------------------------------------------
spring:
application:
name: springCloud
cloud:
stream:
default-binder: kafka
kafka:
bindings:
myChannel:
consumer:
autoCommitOffset: false
But my Acknowledgement object is null as in the header object 'kafka_acknowledgement' itself is NOT present.
How to get the acknowledgment object?
My requirement is to commit the offset ONLY if the processing is successful, if the processing fails I do NOT want to pop the message from the channel so that it can be read later.
Will the above code be sufficient to achieve this?
What version are you using?
In 3.1, autoCommitOffset was deprecated in favor of setting the ackMode (to manual in this case); however, it looks like autoCommitOffset is now completely ignored rather than deprecated.
When using yaml file, please use the property 'auto-commit-offset'.

Sending Compiled bytecode of gremlin query to remote server

Example Code(JAVA):
Cluster cluster = Cluster.open(yml.getFile());
DriverRemoteConnection driver = DriverRemoteConnection.using(cluster, "graph_traversal");
GraphTraversalSource allGraph = AnonymousTraversalSource.traversal().withRemote(driver);
//Compile Script
GremlinScriptEngine engine = new GremlinGroovyScriptEngine();
String script = "graph_traversal.V().outE().inV().path().unfold().dedup().group().by{\"category\"}";
SimpleBindings bind = new SimpleBindings();
GraphTraversal compiled = (GraphTraversal)engine.eval(script, bind);
//Send bytecode to remote server
CompletableFuture<RemoteTraversal<?, Object>> result = driver.submitAsync(compiled.asAdmin().getBytecode());
result.get(); // Exception
I'm trying to send a gremlin bytecode to remote server through driver.
But the codes occurs an exception when the script includes 'lamda'.
The exception message is as following.
Exception:
io.netty.handler.codec.EncoderException: org.apache.tinkerpop.gremlin.driver.exception.ResponseException: An error occurred during serialization of this request [RequestMessage{, requestId=84d5d022-1b08-41a6-b57f-8fdc3b5b6c65, op='bytecode', processor='traversal', args={gremlin=[..., dedup(), unfold(), dedup(), group(), by(Script1$_run_closure1#78b612c6)]], aliases={g=graph_traversal}}}] - it could not be sent to the server - Reason: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IllegalArgumentException: Class is not registered: java.lang.reflect.InvocationHandler
Note: To register this class use: kryo.register(java.lang.reflect.InvocationHandler.class);
if the script doesn't contain the lambda, it won't make any exception.
How can I resolve this?
Thank you.
Solved:
By using Lambda.Methods.

How to response with a success JSON format after completing a transaction in corda

Hi everyone i am working on a project in which i need to send a response in JSON format to the CLI that the Transaction have completed let me give you an example.Consider that i have stated a flow Start ExampleFlow pojo: {iouValue: 7}, otherParty: "O=PartyB,L=London,C=GB" and the result will be Starting
Generating transaction based on new IOU.
Verifying contract constraints.
Signing transaction with our private key.
Gathering the counter party's signature.
Collecting signatures from counterparties.
Verifying collected signatures.
Obtaining notary signature and recording transaction.
Broadcasting transaction to participants
Done
Flow completed with result: SignedTransaction(id=F95406D901209BA77396C1A4D375585C6E051414EE22BE441FC02E5AE147A050)
but what i want is that their should be a JSON format result not all of it but something like this
{response: success }
i just want some success response in JSON format
i am using IOU project
thanks
You can achieve that by establishing an RPC connection with your node; call the flow, then return the JSON object.
There are a couple of approaches that you can follow, and I recommend that you go through the samples repository https://github.com/corda/samples to explore them:
Create a webserver (SpringBoot application) that server REST API's that call your flows and return a JSON object: https://github.com/corda/samples/tree/release-V4/spring-webserver
Create a simple Java app that establishes an RPC connection with your node and serves as a client to call a certain method/flow: https://github.com/corda/samples/blob/release-V4/cordapp-example/clients/src/main/java/com/example/server/JavaClientRpc.java
If you follow the webserver sample, you can add a method to your controller that does something like:
#GetMapping(value = "/my-api", produces = MediaType.APPLICATION_JSON_VALUE)
private ResponseEntity<YourObject> getSomething() {
// Some code that calls your flow and returns YourObject.
return ResponseEntity.ok().body(YourObject);
}
so i got the answer what u need to do is add this dependency in client build.gradle
cordaCompile "net.corda:corda-jackson:$corda_release_version"
after that you just need to implement this code snip
String json = "";
try {
ObjectMapper mapper = JacksonSupport.createNonRpcMapper();
json = mapper.writeValueAsString(results);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return json;
result can be any datatype you want to convert to json

Spring data redis stream receiver completes prematurely

I am using Spring data Redis to consume from a Redis stream ,using the reactive stream receiver to listen over a consumer group works ,but have observed that the Flux stream closes prematurely sometimes and doesn't listen to new messages any more and the flux terminates prematurely .
Code
StreamReceiverOptions<String, MapRecord<String, String, String>> options = StreamReceiverOptions.builder()
.build();
StreamReceiver.create(reactiveConnFactory, options)
.receiveAutoAck("CONSUMER_GRP", "CONSUMER_ID_1"), StreamOffset.create(
"CONSUMER_STREAM",
ReadOffset.lastConsumed()))
.doOnNext(msg -> LOG.info("Got [{}] message from stream", msg))
.flatMap(msg -> Mono.fromRunnable(() -> process("reactive", msg))
.subscribeOn(streamConsumerExecutor))
.onErrorResume(t -> Flux.empty())
.doOnCancel(() -> LOG.info("Consumer Stream was cancelled"))
.doOnComplete(() -> {
LOG.info("Consumer Stream Completed");
})
.doOnTerminate(() -> {
LOG.info("Consumer Stream terminated");
})
.subscribe();
After some time of reading messages from the stream get the log that the "consumer stream terminated"
version : 2.2.0.RELEASE
Is this a bug or am I missing anything ,Could any one help ?
UPDATE
Looks like redis commands are timing out as I get a RedisCommandTimeoutException, is there a way to retry the streaming process on such errors rather than cancelling it . Also figured out it happens in the XREADGROUP operation though when running through the nodejs redis-cli issuing the same command worked fine?

exactly once delivery Is it possible through spring-cloud-stream-binder-kafka or spring-kafka which one to use

I am trying to achieve exactly once delivery using spring-cloud-stream-binder-kafka in a spring boot application.
The versions I am using are:
spring-cloud-stream-binder-kafka-core-1.2.1.RELEASE
spring-cloud-stream-binder-kafka-1.2.1.RELEASE
spring-cloud-stream-codec-1.2.2.RELEASE spring-kafka-1.1.6.RELEASE
spring-integration-kafka-2.1.0.RELEASE
spring-integration-core-4.3.10.RELEASE
zookeeper-3.4.8
Kafka version : 0.10.1.1
This is my configuration (cloud-config):
spring:
autoconfigure:
exclude: org.springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration
kafka:
consumer:
enable-auto-commit: false
cloud:
stream:
kafka:
binder:
brokers: "${BROKER_HOST:xyz-aws.local:9092}"
headers:
- X-B3-TraceId
- X-B3-SpanId
- X-B3-Sampled
- X-B3-ParentSpanId
- X-Span-Name
- X-Process-Id
zkNodes: "${ZOOKEEPER_HOST:120.211.316.261:2181,120.211.317.252:2181}"
bindings:
feed_platform_events_input:
consumer:
autoCommitOffset: false
binders:
xyzkafka:
type: kafka
bindings:
feed_platform_events_input:
binder: xyzkafka
destination: platform-events
group: br-platform-events
I have two main classes:
FeedSink Interface:
package au.com.xyz.proxy.interfaces;
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.messaging.MessageChannel;
public interface FeedSink {
String FEED_PLATFORM_EVENTS_INPUT = "feed_platform_events_input";
#Input(FeedSink.FEED_PLATFORM_EVENTS_INPUT)
MessageChannel feedlatformEventsInput();
}
EventConsumer
package au.com.xyz.proxy.consumer;
#Slf4j
#EnableBinding(FeedSink.class)
public class EventConsumer {
public static final String SUCCESS_MESSAGE =
"SEND-SUCCESS : Successfully sent message to platform.";
public static final String FAULT_MESSAGE = "SOAP-FAULT Code: {}, Description: {}";
public static final String CONNECT_ERROR_MESSAGE = "CONNECT-ERROR Error Details: {}";
public static final String EMPTY_NOTIFICATION_ERROR_MESSAGE =
"EMPTY-NOTIFICATION-ERROR Empty Event Received from platform";
#Autowired
private CapPointService service;
#StreamListener(FeedSink.FEED_PLATFORM_EVENTS_INPUT)
/**
* method associated with stream to process message.
*/
public void message(final #Payload EventNotification eventNotification,
final #Header(KafkaHeaders.ACKNOWLEDGMENT) Acknowledgment acknowledgment) {
String caseMilestone = "UNKNOWN";
if (!ObjectUtils.isEmpty(eventNotification)) {
SysMessage sysMessage = processPayload(eventNotification);
caseMilestone = sysMessage.getCaseMilestone();
try {
ClientResponse response = service.sendPayload(sysMessage);
if (response.hasFault()) {
Fault faultDetails = response.getFaultDetails();
log.error(FAULT_MESSAGE, faultDetails.getCode(), faultDetails.getDescription());
} else {
log.info(SUCCESS_MESSAGE);
}
acknowledgment.acknowledge();
} catch (Exception e) {
log.error(CONNECT_ERROR_MESSAGE, e.getMessage());
}
} else {
log.error(EMPTY_NOTIFICATION_ERROR_MESSAGE);
acknowledgment.acknowledge();
}
}
private SysMessage processPayload(final EventNotification eventNotification) {
Gson gson = new Gson();
String jsonString = gson.toJson(eventNotification.getData());
log.info("Consumed message for platform events with payload : {} ", jsonString);
SysMessage sysMessage = gson.fromJson(jsonString, SysMessage.class);
return sysMessage;
}
}
I have set the autocommit property for Kafka and spring container as false.
if you see in the EventConsumer class I have used Acknowledge in cases where I service.sendPayload is successful and there are no Exceptions. And I want container to move the offset and poll for next records.
What I have observed is:
Scenario 1 - In case where the Exception is thrown and there are no new messages published on kafka. There is no retry to process the message and it seems there is no activity. Even if the underlying issue is resolved. The issue I am referring to is down stream server unavailability. Is there a way to retry the processing n times and then give up. Note this is retry of processing or repoll from the last committed offset. This is not about Kafka instance not available.
If I restart the service (EC2 instance) then the processing happens from the offset where the last successful Acknowledge was done.
Scenario 2 - In case where Exception happened and then a subsequent message is pushed to kafka. I see the new message is processed and the offset moved. It means I lost the message which was not acknowledged. So the question is if I have handled the Acknowledge. How do I control to read from last commit not just the latest message and process it. I am assuming there is internally a poll happening and it did not take into account or did not know about the last message not being acknowledged. I don't think there are multiple threads reading from kafka. I dont know how the #Input and #StreamListener annotations are controlled. I assume the thread is controlled by property consumer.concurrency which controls the thread and by default it is set to 1.
So I have done research and found a lot of links but unfortunately none of them answers my specific questions.
I looked at (https://github.com/spring-cloud/spring-cloud-stream/issues/575)
which has a comment from Marius (https://stackoverflow.com/users/809122/marius-bogoevici):
Do note that Kafka does not provide individual message acking, which
means that acknowledgment translates into updating the latest consumed
offset to the offset of the acked message (per topic/partition). That
means that if you're acking messages from the same topic partition out
of order, a message can 'ack' all the messages before it.
not sure if it is the issue with order when there is one thread.
Apologies for long post, but I wanted to provide enough information. The main thing is I am trying to avoid losing messages when consuming from kafka and I am trying to see if spring-cloud-stream-binder-kafka can do the job or I have to look at alternatives.
Update 6th July 2018
I saw this post https://github.com/spring-projects/spring-kafka/issues/431
Is this a better approach to my problem? I can try latest version of spring-kafka
#KafkaListener(id = "qux", topics = "annotated4", containerFactory = "kafkaManualAckListenerContainerFactory",
containerGroup = "quxGroup")
public void listen4(#Payload String foo, Acknowledgment ack, Consumer<?, ?> consumer) {
Will this help in controlling the offset to be set to where the last
successfully processed record? How can I do that from the listen
method. consumer.seekToEnd(); and then how will listen method reset to get the that record?
Does putting the Consumer in the signature provide support to get
handle to consumer? Or I need to do anything more?
Should I use Acknowledge or consumer.commitSyncy()
What is the significance of containerFactory. do I have to define it
as a bean.
Do I need #EnableKafka and #Configuration for above approach to work?
Bearing in mind the application is a Spring Boot application.
By Adding Consumer to listen method I don't need to implement
ConsumerAware Interface?
Last but not least, Is it possible to provide some example of above approach if it is feasible.
Update 12 July 2018
Thanks Gary (https://stackoverflow.com/users/1240763/gary-russell) for providing the tip of using maxAttempts. I have used that approach. And I am able to achieve exactly once delivery and preserve the order of the message.
My updated cloud-config:
spring:
autoconfigure:
exclude: org.springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration
kafka:
consumer:
enable-auto-commit: false
cloud:
stream:
kafka:
binder:
brokers: "${BROKER_HOST:xyz-aws.local:9092}"
headers:
- X-B3-TraceId
- X-B3-SpanId
- X-B3-Sampled
- X-B3-ParentSpanId
- X-Span-Name
- X-Process-Id
zkNodes: "${ZOOKEEPER_HOST:120.211.316.261:2181,120.211.317.252:2181}"
bindings:
feed_platform_events_input:
consumer:
autoCommitOffset: false
binders:
xyzkafka:
type: kafka
bindings:
feed_platform_events_input:
binder: xyzkafka
destination: platform-events
group: br-platform-events
consumer:
maxAttempts: 2147483647
backOffInitialInterval: 1000
backOffMaxInterval: 300000
backOffMultiplier: 2.0
Event Consumer remains the same as my initial implementation. Except for rethrowing the error for the container to know the processing has failed. If you just catch it then there is no way container knows the message processing has failures. By doing acknoweldgement.acknowledge you are just controlling the offset commit. In order for retry to happen you must throw the exception. Don't forget to set the kafka client autocommit property and spring (container level) autocommitOffset property to false. Thats it.
As explained by Marius, Kafka only maintains an offset in the log. If you process the next message, and update the offset; the failed message is lost.
You can send the failed message to a dead-letter topic (set enableDlq to true).
Recent versions of Spring Kafka (2.1.x) have special error handlers ContainerStoppingErrorHandler which stops the container when an exception occurs and SeekToCurrentErrorHandler which will cause the failed message to be redelivered.

Resources