I have setup a cluster within kubernetes using jgroups and the cluster appears to form correctly, each node has a local ip and a public ip, when I connect to one of the nodes using the public ip all is fine but the list of available nodes that is returned to the client (wildfly instance) contains the local ips of the nodes rather than their public ones, I have defined the connector with the public ip
<connectors>
<connector name="netty-connector">tcp://{public ip}:61616</connector>
</connectors>
and then configured the broadcast as
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<broadcast-period>5000</broadcast-period>
<jgroups-file>jgroups-file_ping.xml</jgroups-file>
<jgroups-channel>activemq_broadcast_channel</jgroups-channel>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
and then configured the discvery as
<discovery-groups>
<discovery-group name="my-discovery-group">
<jgroups-file>jgroups-file_ping.xml</jgroups-file>
<jgroups-channel>activemq_broadcast_channel</jgroups-channel>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
and finally the cluster as
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
Whenever I force a node to shutdown the client reconnects but fails and reports the local ip of the node, I was under the impression that the connector defined in the broker was used to broadcast to other members of the cluster but it uses the local ip is that correct?
Wildfly runs and send and receives messages but every few minutes I get the following log
14:27:31,463 WARN [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) AMQ122015: Can not connect to XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?trustStorePassword=****&port=61616&sslEnabled=true&host=x-x-x-x&trustStorePath=client-ts], discoveryConfiguration=null, username=username, password=****, JNDI_NAME=java:/RemoteJmsXA] on auto-generated resource recovery: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119007: Cannot connect to server(s). Tried with all available servers.]at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:797)
at org.apache.activemq.artemis.service.extensions.xa.recovery.ActiveMQXAResourceWrapper.connect(ActiveMQXAResourceWrapper.java:311)
at org.apache.activemq.artemis.service.extensions.xa.recovery.ActiveMQXAResourceWrapper.getDelegate(ActiveMQXAResourceWrapper.java:239)
at org.apache.activemq.artemis.service.extensions.xa.recovery.ActiveMQXAResourceWrapper.recover(ActiveMQXAResourceWrapper.java:69)
at org.apache.activemq.artemis.service.extensions.xa.ActiveMQXAResourceWrapperImpl.recover(ActiveMQXAResourceWrapperImpl.java:106)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:634)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:226)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:171)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:770)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:382)
This is the expected behavior as you are connecting through a load balancer. You can work around that by setting useTopologyForLoadBalancing=false and specifying servers explicitly in your connection URL.
When using WildFly, the connection factory or pooled connection factory must be configured with the attribute use-topology-for-load-balancing set to false. This is how to set this from the CLI (replace remote-artemis with your actual name):
/subsystem=messaging-activemq/pooled-connection-factory=remote-artemis:write-attribute(name=use-topology-for-load-balancing, value=false)
Got it working eventually by creating a service per pod and putting public ip in the connector definition for each node
Related
I have created a knative service[gRPC server] in aks cluster, I have exposed the service using istio gateway private static IP,
After using the command kubectl get ksvc I have got an address sample-app.test.10.153.157.156.sslio.io
When I try to use this address in the python client, it throws error saying failed to connect addresses, but if I try to hit the service using
curl sample-app.test.10.153.157.156.sslio.io I am able to hit the service, I don't know what i am missing here.. please suggest..
GRPC uses HTTP/2. You may need to explicitly name your port h2c. I'm assuming that you've tested the container locally without Knative in the path and have been able to make a grpc call in that case.
I am using apache KafkaConsumer in my Scala app to talk to a Kafka server wherein the Kafka and Zookeeper services are running in a docker container on my VM (the scala app is also running on this VM). I have setup the KafkaConsumer's property "bootstrap.servers" to use 127.0.0.1:9092.
The KafkaConsumer does log, "Sending coordinator request for group queuemanager_testGroup to broker 127.0.0.1:9092". The problem appears to be that the Kafka client code is setting the coordinator values based on the response it receives which contains responseBody={error_code=0,coordinator={node_id=0,host=e7059f0f6580,port=9092}} , that is how it sets the host for future connections. Subsequently it complains that it is unable to resolve address: e7059f0f6580
The address e7059f0f6580 is the container ID of that docker container.
I have tested using telnet that my VM is not detecting this as a hostname.
What setting do I need to change such that the Kafka on my docker returns localhost/127.0.0.1 as the host in its response ? Or is there something else that I am missing / doing incorrectly ?
Update
advertised.host.name is deprecated, and --override should be avoided.
Add/edit advertised.listeners to be the format of
[PROTOCOL]://[EXTERNAL.HOST.NAME]:[PORT]
Also make sure that PORT is also listed in property for listeners
After investigating this problem for hours on end, found that there is a way to
set the hostname while starting up the Kafka server, as follows:
kafka-server-start.sh --override advertised.host.name=xxx (in my case: localhost)
I have recently started exploring kuberenetes and done with practical implementation of pods,services and replication Controller on google cloud. I have some doubts over service and network access .
First, Where is the service deployed which will work as load balancer for group of pods ?
Second, does the request to access an application running in pod using a service load balancer go through master or direct to minions nodes ?
A service proxy runs on each node on the cluster. From inside the cluster, when you make a request to a service IP, it is intercepted by the service proxy and routed to a pod matching the label selector for the service. If you have specified an external load balancer for your service, the load balancer will pick a node to send the request to, at which point it will be captured by the proxy and directed to an appropriate pod. If you are using public IPs, then your router will send the request to the node with the public IP where it will be captured by the proxy and directed to an appropriate pod.
If you followed by description, you can see that service requests do not go through the master. They bounce through a proxy running on the nodes.
As an aside, there is also a proxy running on the master, which you can use to reach nodes, services, pods, but this proxy isn't in the packet path for services that you create within the cluster.
I tried to configure MS DTC via our VPN. But when I try to open the connection it gives me the following error.
The MSDTC transaction manager was unable to push the transaction to the destination transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn't have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers. (Exception from HRESULT: 0x8004D02A)
When I check with the network team they told me that the firewall is already configured to allow DTC.
If I explain this further this communication is done via VPN. According to my network admin, although the client machine could see the NetBios name of the server but server cannot see the client's NetBios name, when this communication is done through a firewall/router. He is telling that, to start a DTC communication both machines should be able to see their NetBios names.
I tried with DTCPing (Same setup) and the dtc ping error is
03-04, 10:18:33.918-->RPC server:NGSVR received following information:
Network Name: NGSVR
Source Port: 49179
Partner LOG: WS-PCSPOS76036.log
Partner CID: EBA77A41-C9F9-4162-B7A2-E10404719072
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from WS-PCSPOS7
Network Name: NGSVR
Source Port: 49179
Hosting Machine:NGSVR
03-04, 10:18:33.996-->Trying to Reverse Bind to WS-PCSPOS7...
Test Guid:EBA77A41-C9F9-4162-B7A2-E10404719072
gethostbyname can not resolve WS-PCSPOS7
Error(0xB7) at nameping.cpp #43
-->gethostbyname failure
-->183(Cannot create a file when that file already exists.)
Can not resolve WS-PCSPOS7
Error(0x6BA) at ServerManager.cpp #453
-->RPC reverse BIND failed
-->1722(The RPC server is unavailable.)
Reverse Binding to WS-PCSPOS7 Failed
In GUID
Out GUID
Reverse BIND FAILED
Session Down
I have tried to open and do a transaction via non vpn setup and it was successful.
Can we configure MS DTC via VPN?
If it is possible any additional configuration should do to VPN?
The application i'm currently working on start to enter in a pre-release phase.
In this phase, the server-side application components are to be deployed on Amazon VMs while the client-side application remains on the user machine.
This applications connects to server using JNDI and RMI to call remote EJB methods. This works well on localhost and local network.
But, when trying to connect to Amazon host, the application hangs up on context.lookup method. that's to say a JNDI context can be obtained from this remote server, but no lookup can be performed on that context.
What can I do to obtain good diagnostic on the failure ?
Are there logs that can be generated for the RMI handshake/whatever ?
Is there any way to see on server side if query really drive its way through the internet to the server ?
Also notice I've already enabled public IP usage on my Glassfish server (using recommended Oracle procedure).
EDIT According to a fast TCP capture on server, it seems that server receives the client context query with in-lan client address, which it of course isn't aware of :
query is
[3/27/2012 11:05:22 AM:169]
GIOP.......(................NameService....._is_a...................
NEO................ª.......(IDL:omg.org/SendingContext/CodeBase:1.0.
...........n........172.27.63.145.ܺ....¯«Ë........e................
........... ................... ... ...........&...............(IDL:
omg.org/CosNaming/NamingContext:1.0.
reply is
[3/27/2012 11:05:22 AM:171]
GIOP.......2............NEO................0.......(IDL:omg.org/Send
ingContext/CodeBase:1.0............ô........46.137.114.67.'5....¯
«Ë........d........................... ................... .........
.....&...........!...|...............$....f............10.241.42.208
.'6.#........g..............g........default...................g....
...........+IDL:omg.org/CosNaming/NamingContextExt:1.0..............
.......10.241.42.208.'5...M¯«Ë.... ...d... S1AS-ORB............Root
POA....TNameService............................... .................
.. ... ...........&......
(as read using SmartSniff ASCII output).
The IP in query (172.27.63.145) is my IP in my company LAN. From what I understand of communication over itnernet, it should be my company LAN public IP, no ? How can I make Glassfish client udnerstand it should use that IP ?
Diagnostic has been clearly obtained : the client, which connects from a LAN to the server, sends its own internal network private address to server, which server can't forward any answer to. As a consequence, server doesn't answer, hence the hangup.