I am attempting to use the AWS DaxClient to connect to a localstack setup running DynamoDB.
I start localstack from the docker-compose file in their github repo here.
Then I try creating a Dax Client in Java code and pointing it to that
public class PlainJavaClass {
static AmazonDynamoDB daxClient;
public static void main(String[] args) throws Exception {
AmazonDaxClientBuilder daxClientBuilder = AmazonDaxClientBuilder.standard();
daxClientBuilder.withRegion("us-east-1").withEndpointConfiguration("localhost:4566");
daxClient = daxClientBuilder.build();
}
}
However I'm getting the following exception:
12:36:53.713 [main] WARN com.amazon.dax.client.dynamodbv2.ClusterDaxClient - exception starting up cluster client: java.io.IOException: failed to configure cluster endpoints from hosts: [localhost:4566]
java.io.IOException: failed to configure cluster endpoints from hosts: [localhost:4566]
at com.amazon.dax.client.cluster.Source$AutoconfSource.pull(Source.java:127)
at com.amazon.dax.client.cluster.Source$AutoconfSource.update(Source.java:59)
at com.amazon.dax.client.cluster.Source$AutoconfSource.refresh(Source.java:50)
at com.amazon.dax.client.cluster.Cluster.refresh(Cluster.java:426)
at com.amazon.dax.client.cluster.Cluster.refresh(Cluster.java:409)
at com.amazon.dax.client.cluster.Cluster.startup(Cluster.java:330)
at com.amazon.dax.client.cluster.Cluster.startup(Cluster.java:263)
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.<init>(ClusterDaxClient.java:148)
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.<init>(ClusterDaxClient.java:119)
at com.amazon.dax.client.dynamodbv2.AmazonDaxClientBuilder.build(AmazonDaxClientBuilder.java:34)
at com.example.crudspringbootdynamodb.PlainJavaClass.createDaxClient(PlainJavaClass.java:30)
at com.example.crudspringbootdynamodb.PlainJavaClass.main(PlainJavaClass.java:50)
Caused by: com.amazon.cbor.EndOfStreamException: null
at com.amazon.cbor.CborInputStream.readObject(CborInputStream.java:1340)
at com.amazon.dax.client.exceptions.DaxServiceException.pickException(DaxServiceException.java:44)
at com.amazon.dax.client.generated.DaxClientStubs.handleResponse(DaxClientStubs.java:963)
at com.amazon.dax.client.generated.DaxClientStubs.endpoints_455855874_1(DaxClientStubs.java:479)
at com.amazon.dax.client.dynamodbv2.DaxClient.endpoints(DaxClient.java:2375)
at com.amazon.dax.client.cluster.Source$AutoconfSource.pullFrom(Source.java:137)
at com.amazon.dax.client.cluster.Source$AutoconfSource.pull(Source.java:105)
... 11 common frames omitted
Suppressed: com.amazon.cbor.EndOfStreamException: null
... 18 common frames omitted
I know I'm able to connect to localstack using a normal DynamoDb client created like this:
AmazonDynamoDBClientBuilder
.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:4566/", "us-east-1"))
.build();
So not really sure where to go next. Is it possible to use DAX client to connect to DynamoDB localstack?
No, localstack does not support it yet. There is an open issue about it on github (https://github.com/localstack/localstack/issues/3707).
Related
I'm trying to run some tests in my consumer application using Spring Cloud Contract's Stub Runner.
I have noticed that when the stubsMode property is set to LOCAL.
#AutoConfigureStubRunner(
stubsMode = StubRunnerProperties.StubsMode.LOCAL,
ids = "com.example:spring-cloud-contract-producer:+:stubs:8090")
my build is successful because an embedded Wiremock instance boots up and listens in that port.
However, if I change the stubsMode property to CLASSPATH, my build fails because the test cannot establish a connection at that port.
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8090/validate/prime-number": Connect to localhost:8090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to localhost:8090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
According to the docs, this should only affect how the stubs are downloaded:
StubRunnerProperties.StubsMode.CLASSPATH (default value) - will pick
stubs from the classpath
What am I doing wrong here? Thanks beforehand!
If you turn on the classpath mode then, as the name suggests, you need to have the dependencies on your classpath. The jar with the stubs needs to contain a predefined structure described in the docs. In general, in your case, it would contain a META-INF/com.example/spring-cloud-contract-producer/mappings folder with WireMock stubs inside it. If you don't have the producer stubs on your classpath then the classpath mode will not work.
I have the following Corda service. It starts up a Jetty Server
#CordaService
class IOUService(private val serviceHub: AppServiceHub): SingletonSerializeAsToken() {
init {
val port = serviceHub.myInfo.addresses.first().port - 1002
log.println("IOUService init was called...")
log.println("Port: $port")
val jettyServer = JettyServer()
jettyServer.start(port)
}
}
My problem is how to release the stared Jetty port when running integration tests. Here are two example test (basically the same test twice to illustrate the problem):
#Test
fun `node test`() = withDriver {
val (partyAHandle, partyBHandle) = startNodes(bankA, bankB)
assertEquals(bankB.name, partyAHandle.resolveName(bankB.name))
assertEquals(bankA.name, partyBHandle.resolveName(bankA.name))
}
#Test
fun `node test2`() = withDriver {
val (partyAHandle, partyBHandle) = startNodes(bankA, bankB)
assertEquals(bankB.name, partyAHandle.resolveName(bankB.name))
assertEquals(bankA.name, partyBHandle.resolveName(bankA.name))
}
The first test will start up 3 nodes: one notary, BankA and BankB nodes with the following details:
Notary:
Advertised P2P messaging addresses : localhost:10000
RPC connection address : localhost:10001
RPC admin connection address : localhost:10002
Jetty Port: 8998
BankA:
Advertised P2P messaging addresses : localhost:10004
RPC connection address : localhost:10005
RPC admin connection address : localhost:10006
Jetty Port: 9002
BankB:
Advertised P2P messaging addresses : localhost:10008
RPC connection address : localhost:10009
RPC admin connection address : localhost:10010
Jetty Port: 9006
Unfortunately the second test will fails since the Jetty ports are still bound:
[ERROR] 14:22:04,825 [driver-pool-thread-0] internal.Node.installCordaServices - Corda service com.example.flows.IOUService failed to instantiate. Reason was: Address already in use [errorCode=1pryyp4, moreInformationAt=https://errors.corda.net/OS/4.1/1pryyp4]
java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:9006
How to register a shutdown hook during integration testing in order to shut down the Jetty servers?
The example code can be found here:
https://github.com/altfatterz/learning-corda
A proper lifecycle for corda services is currently being looked at. Hopefully, you can do this in the future.
For now, there is not a simple way to do this from within the node.
I am working on kafka . I have created kafka producer on my server . I want to get data from kafkaproducer to my local system in r.
I have tried following code in R:
library(rkafka)
consumer1<-rkafka.createConsumer("ipaddress:9092","mytest")
consumer11 <- rkafka.read(consumer1)
It throws following error:
[1] "Java-Object{com.musigma.consumer.MuConsumer#3349e9bb}"
Unable to connect to zookeeper server
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within
timeout: 100000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:156)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:114)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:65)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:67)
at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100)
at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala)
at com.musigma.consumer.MuConsumer.CreateConsumer(MuConsumer.java:99)
java.lang.NullPointerException
at com.musigma.consumer.MuConsumer.startConsumer(MuConsumer.java:133)
My zookeeper is running on the ipaddress successfully.
The first parameter is Zookeeper, which runs on port 2181
You've given it Kafka port
Source - https://github.com/cran/rkafkajars/blob/master/java/com/musigma/consumer/MuConsumer.java#L87
Note: Looks like that library isn't maintained and using Zookeeper to connect with a consumer is practically deprecated, so maybe try finding another library
As a test developer, I am trying to use localstack to mock SQS for Integration Test.
Docker-compose:
localstack:
image: localstack/localstack:0.8.7
ports:
- "4567-4583:4567-4583"
- "9898:${PORT_WEB_UI-8080}"
environment:
- SERVICES=sqs
- DOCKER_HOST=unix:///var/run/docker.sock
- HOSTNAME=localstack
- HOSTNAME_EXTERNAL=192.168.99.101
- DEFAULT_REGION=us-east-1
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
After spinning up the localstack SQS: Able to connect and create queue and able to retrieve it via AWS CLI. Localstack Dashboard also displays the queue created.
$ aws --endpoint-url=http://192.168.99.101:4576 --region=us-west-1 sqs create-queue --queue-name myqueue
{
"QueueUrl": "http://192.168.99.101:4576/queue/myqueue"
}
The application is using com.amazon.sqs.javamessaging.SQSConnectionFactory to connect to SQS. It also uses com.amazonaws.auth.DefaultAWSCredentialsProviderChain for the AWS credentials
1) If I give "-e AWS_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=foobar -e AWS_SECRET_ACCESS_KEY=foobar" while bringing up the application, I am getting
HTTPStatusCode: 403 AmazonErrorCode: InvalidClientTokenId
com.amazonaws.services.sqs.model.AmazonSQSException: The security token included in the request is invalid
2) If I give the ACCESS_KEY and SECRET_KEY of the AWS SQS, I am getting
HTTPStatusCode: 400 AmazonErrorCode: AWS.SimpleQueueService.NonExistentQueue
com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version.
Below is the application code. The first 2 log messages are printing the connection and session it obtained. The error is coming from "Queue publisherQueue = sqsSession.createQueue(sqsName);"
sqsConnection = (Connection) context.getBean("outputSQSConnection");
LOGGER.info("SQS connection Obtained " + sqsConnection);
sqsSession = sqsConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);
LOGGER.info("SQS Session Created " + sqsSession);
Queue publisherQueue = sqsSession.createQueue(sqsName);
I tried both "http://localstack:4576/queue/myqueue" "http://192.168.99.101:4576/queue/myqueue". The results are same.
Can you please help me to resolve the issue?
I ran into a similar issue couple of weeks back . Looking at your config, I think you should be just able to use localhost. In my case, we had services calling localstack also running in docker and we ended up creating a docker network to communicate between containers.
I was able figure out a solution by looking at the Localstack tests. The important thing to note here is that you need to set Endpoint configuration correctly.
private AmazonSQSClient getLocalStackConfiguredClient() {
AmazonSQSClientBuilder clientBuilder = AmazonSQSClientBuilder.standard();
clientBuilder.withEndpointConfiguration(getEndpointConfiguration(configuration.getLocalStackConfiguration()
.getSqsEndpoint(), awsRegion));
return (AmazonSQSClient)clientBuilder.build();
}
private AwsClientBuilder.EndpointConfiguration getEndpointConfiguration(String endpoint, Regions awsRegion) {
return new AwsClientBuilder.EndpointConfiguration(endpoint, awsRegion.getName());
}
Hopefully this helps you to resolve the issues.
I have the problem in Corda regarding performing IOU from Party A to Party B.
Below is configuration detail:
3 node.conf [Party A, Party B, and Notary ].
Hosting application in AWS, So in node config file instead of "localhost", I gave the IP of the machines. I gave the same IP for Notary & Party A, different for Party B.
Network Bootstrapping was successful and moved the newly created node folders respective EC2 instances and started run nodes.
But when performed the IOU from Party A to Party B it's not working. Please suggest how to resolve the issue.
I see the following error in the node logs:
E 11:34:47+0000 [main] internal.Node.run - Exception during node startup {}
java.net.BindException: Cannot assign requested address: bind
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_161]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:1.8.0_161]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:1.8.0_161]
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:1.8.0_161]
at io.netty.channe
Reference: https://docs.corda.net/tutorial-cordapp.html#running-nodes-across-machines
I reach node communication on different hosts by the following way.
First of all I deploy node with node.conf file which contains
"p2pAddress" : "host:10012",
"rpcSettings" : {
"address" : "host:10014",
"adminAddress" : "host:10013"
}
Then after node deployed I change host of rpcSettings to localhost
"rpcSettings" : {
"address" : "localhost:10014",
"adminAddress" : "localhost:10013"
}
Such way looks strange, however after this manipulation nodes started to communicate
This is related with NodeInfo file which is generated at node deploy and it should contains the host for rpc. After that rpc needs localhost for interaction. I think it might be a bug, but works fine in that way.
When using rpcSettings in Corda V3.1 the address and adminAddress need to be using 0.0.0.0.
rpcSettings {
address="0.0.0.0:10003"
adminAddress="0.0.0.0:10103"
}
These endpoints are not advertised externally so the local ip is solely a binding for Corda.
This should solve the following exception on starting your cordapp when using public ip or DNS:
E 21:28:56+0000 [main] internal.Node.run - Exception during node
startup {} io.netty.channel.unix.Errors$NativeIoException: bind(..)
failed: Cannot assign requested address