Cannot get WSO2 to work using membershipScheme=AWS - wso2-api-manager

Question

I'm running WSO2 AM 1.10.0 on to Amazon EC2 nodes of the same region/zone. One of them is a manager node, the other worker node.
When running them using membershipScheme=WKA, everything works ok. When trying membershipScheme=AWS, both servers register correctly to the cloud but do not interact with each other. I verified the AWS parameters in axis2.xml are correct, and opened ports 5700-5800 to ALL.
My WKA configuration is this:
Manager node (WKA)
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">5701</parameter>
<parameter name="localMemberHost">ec2-M</parameter>
<groupManagement enable="true">
<applicationDomain name="wso2.apim.domain"
description="APIM group"
agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
subDomain="worker"
port="5757"/>
</groupManagement>
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>5701</port>
</member>
<member>
<hostName>ec2-GW</hostName>
<port>5702</port>
</member>
</members>
Worker node (WKA)
<parameter name="membershipScheme">wka</parameter>
<parameter name="localMemberPort">5702</parameter>
<parameter name="localMemberHost">ec2-GW</parameter>
<groupManagement enable="true">
<applicationDomain name="wso2.apim.domain"
description="APIM group"
agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
subDomain="mgt"
port="5757"/>
</groupManagement>
<members>
<member>
<hostName>ec2-M</hostName>
<port>5701</port>
</member>
<member>
<hostName>127.0.0.1</hostName>
<port>5702</port>
</member>
</members>
In this mode everything works ok, and in manager log I see the gateway worker member joining the cluster:
Managing group application domain:wso2.apim.domain, sub-domain:mgt using agent class org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent
Cluster domain: wso2.am.domain
Using wka based membership management scheme
Added member: Host:127.0.0.1, Remote Host:null, Port: 5701, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
Added member: Host:ec2-GW, Remote Host:null, Port: 5702, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
Local member: [cf971938-ae01-42f7-84c1-563943ea1570] - Host:10.1.0.87, Remote Host:null, Port: 5701, HTTP:8281, HTTPS:8244, Domain: wso2.am.domain, Sub-domain:mgt, Active:true
Elected this member [cf971938-ae01-42f7-84c1-563943ea1570] as the Coordinator node
Added member: Host:10.1.0.87, Remote Host:null, Port: 5701, HTTP:8281, HTTPS:8244, Domain: wso2.am.domain, Sub-domain:mgt, Active:true
Application member Host:ec2-M, Remote Host:null, Port: 5757, HTTP:8281, HTTPS:8244, Domain: wso2.apim.domain, Sub-domain:mgt, Active:true joined application cluster
Member joined [a9268bd0-de59-47eb-bd90-69e6e3ef82c8]: ec2-GW/10.1.0.193:5702
Added member: Host:10.1.0.193, Remote Host:null, Port: 5702, HTTP:8280, HTTPS:8243, Domain: wso2.am.domain, Sub-domain:worker, Active:true
Now changing to AWS mode:
Manager and Worker node (AWS)
<parameter name="membershipScheme">aws</parameter>
<parameter name="accessKey">...</parameter>
<parameter name="secretKey">...</parameter>
<parameter name="securityGroup">...</parameter>
<parameter name="region">eu-west-1</parameter>
(left everything else untouched)
Managing group application domain:wso2.apim.domain, sub-domain:mgt using agent class org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent
Cluster domain: wso2.am.domain
Using aws based membership management scheme
Group management local member for domain [wso2.apim.domain],sub-domain [mgt] UUID: e8847f8c-b6c4-434d-9c3c-7eb8ceca59f2. Host:ec2-M, Remote Host:null, Port: 5757, HTTP:8281, HTTPS:8244, Domain: wso2.apim.domain, Sub-domain:mgt, Active:true
Local member: [2740f020-ca36-4e41-a94c-996d4c1a4fee] - Host:10.1.0.87, Remote Host:null, Port: 5701, HTTP:8281, HTTPS:8244, Domain: wso2.am.domain, Sub-domain:mgt, Active:true
Elected this member [2740f020-ca36-4e41-a94c-996d4c1a4fee] as the Coordinator node
Application member Host:ec2-M, Remote Host:null, Port: 5757, HTTP:8281, HTTPS:8244, Domain: wso2.apim.domain, Sub-domain:mgt, Active:true joined application cluster
As you can see from the log, there is no indication of the worker.
Furthermore, changes done in the manager are not consumed by the worker.
In the worker node:
Managing group application domain:wso2.apim.domain, sub-domain:worker using agent class org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent
Cluster domain: wso2.am.domain
Using aws based membership management scheme
Group management local member for domain [wso2.apim.domain],sub-domain [worker] UUID: d034e9b0-b101-4d78-af44-d3e4a28ba0fd. Host:ec2-GW, Remote Host:null, Port: 5757, HTTP:8280, HTTPS:8243, Domain: wso2.apim.domain, Sub-domain:worker, Active:true
Application member Host:ec2-GW, Remote Host:null, Port: 5757, HTTP:8280, HTTPS:8243, Domain: wso2.apim.domain, Sub-domain:worker, Active:true joined application cluster
Local member: [50961848-8d78-412f-b83f-01dc8c7ad3bf] - Host:10.1.0.193, Remote Host:null, Port: 5702, HTTP:8280, HTTPS:8243, Domain: wso2.am.domain, Sub-domain:worker, Active:true
Elected this member [50961848-8d78-412f-b83f-01dc8c7ad3bf] as the Coordinator node
I checked with telnet: ports 5757, 5701 and 5702 are accessible b/w the two nodes.
Any ideas why the two nodes are not communicating?
Follow up #1 based on comments
Added tag to both nodes
$ aws ec2 create-tags --resources ec2-M ec2-GW --tags Key=component,Value=gateway
Verifiy with describe-instances:
$ aws ec2 describe-instances
...
"Tags": [
{
"Value": "gateway",
"Key": "component"
}
]
...
Modified axis2.xml, for both ec2-M and ec2-GW:
<parameter name="localMemberPort">5701</parameter>
...
<parameter name="accessKey">...</parameter>
<parameter name="secretKey">...</parameter>
<parameter name="securityGroup">...</parameter>
<parameter name="region">eu-west-1</parameter>
<parameter name="tagKey">component</parameter> 
<parameter name="tagValue">gateway</parameter>
Still no communication b/w the nodes
Follow up #2 after further investigation
Using Hazelcast REST API, I noticed that in WKA mode, each member is registered in Hazelcast of both nodes
On manager, running http://ec2-M:5701/hazelcast/rest/cluster:
Members [2] {
Member [ec2-M]:5701 this
Member [ec2-GW]:5702
}
On worker, running http://ec2-GW:5702/hazelcast/rest/cluster:
Members [2] {
Member [ec2-M]:5701
Member [ec2-GW]:5702 this
}
While in AWS mode, each node see only itself.
On manager, running http://ec2-M:5701/hazelcast/rest/cluster:
Members [1] {
Member [ec2-M]:5701 this
}
On worker, running http://ec2-GW:5701/hazelcast/rest/cluster:
Members [1] {
Member [ec2-GW]:5701 this
}