Coherence: Simple custom AddressProvider not working - oracle-coherence

Question

I was trying to use address-provider for well-known-addresses configuration. But it is not working.
Here's my config file:
<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">
<cluster-config>
<unicast-listener>
<well-known-addresses>
<address-provider>
<class-name>xilu.scratch.coherence.MyAddressProvider</class-name>
</address-provider>
<!--
<socket-address id="1">
<address>127.0.0.1</address>
<port>8888</port>
</socket-address>
-->
</well-known-addresses>
<address>127.0.0.1</address>
<port>8888</port>
<port-auto-adjust system-property="tangosol.coherence.localport.adjust">
false
</port-auto-adjust>
</unicast-listener>
</cluster-config>
</coherence>
And here's my address provider:
package xilu.scratch.coherence;
import java.net.InetSocketAddress;
/**
* Created by xxilu on 5/18/16.
*/
public class MyAddressProvider implements com.tangosol.net.AddressProvider {
private String[] addressList = {
"127.0.0.1",
};
private int index = 0;
#Override
public InetSocketAddress getNextAddress() {
if(index >= 1) {
return null;
}
String addressText = addressList[index];
index++;
InetSocketAddress addr = new InetSocketAddress(addressText, 8888);
return addr;
}
#Override
public void accept() {
}
#Override
public void reject(Throwable throwable) {
}
}
It looks very trivial, but the cluster cannot be formed.
I get log like this:
/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/bin/java -Dtangosol.coherence.log.level=9 -Djava.net.preferIPv4Stack=true -Didea.launcher.port=7537 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 15.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/lib/tools.jar:/Users/xxilu/Projects/scratch/Coherence1/out/production/Coherence1:/Users/xxilu/Oracle/Middleware/Oracle_Home/coherence/lib/coherence.jar:/Users/xxilu/Oracle/Middleware/Oracle_Home/oracle_common/modules/com.fasterxml.jackson.core.jackson-core.jar:/Users/xxilu/Oracle/Middleware/Oracle_Home/oracle_common/modules/com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider.jar:/Users/xxilu/Oracle/Middleware/Oracle_Home/oracle_common/modules/com.fasterxml.jackson.core.jackson-databind.jar:/Users/xxilu/Oracle/Middleware/Oracle_Home/oracle_common/modules/com.fasterxml.jackson.core.jackson-annotations.jar:/Users/xxilu/Oracle/Middleware/Oracle_Home/oracle_common/modules/com.fasterxml.jackson.module.jackson-module-jaxb-annotations.jar:/Applications/IntelliJ IDEA 15.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain xilu.scratch.coherence.HelloWorld
2016-05-19 00:19:42.117/1.064 Oracle Coherence 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Loaded operational configuration from "jar:file:/Users/xxilu/Oracle/Middleware/Oracle_Home/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2016-05-19 00:19:42.198/1.136 Oracle Coherence 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Loaded operational overrides from "jar:file:/Users/xxilu/Oracle/Middleware/Oracle_Home/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2016-05-19 00:19:42.258/1.196 Oracle Coherence 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Loaded operational overrides from "file:/Users/xxilu/Projects/scratch/Coherence1/out/production/Coherence1/tangosol-coherence-override.xml"
2016-05-19 00:19:42.268/1.206 Oracle Coherence 12.2.1.0.0 <D5> (thread=pool-1-thread-1, member=n/a): Optional configuration override "cache-factory-config.xml" is not specified
2016-05-19 00:19:42.269/1.207 Oracle Coherence 12.2.1.0.0 <D5> (thread=pool-1-thread-1, member=n/a): Optional configuration override "cache-factory-builder-config.xml" is not specified
2016-05-19 00:19:42.269/1.208 Oracle Coherence 12.2.1.0.0 <D5> (thread=pool-1-thread-1, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
2016-05-19 00:19:42.270/1.208 Oracle Coherence 12.2.1.0.0 <D6> (thread=pool-1-thread-1, member=n/a): Loaded edition data from "jar:file:/Users/xxilu/Oracle/Middleware/Oracle_Home/coherence/lib/coherence.jar!/coherence-grid.xml"
Oracle Coherence Version 12.2.1.0.0 Build 60603
Grid Edition: Development mode
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
2016-05-19 00:19:42.798/1.736 Oracle Coherence GE 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Loaded FMW commons version: 12.2.1-0-0-SNAPSHOT b60603
2016-05-19 00:19:42.885/1.823 Oracle Coherence GE 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Loaded cache configuration from "file:/Users/xxilu/Projects/scratch/Coherence1/out/production/Coherence1/coherence-cache-config.xml"
2016-05-19 00:19:43.539/2.477 Oracle Coherence GE 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): The cluster name has not been configured, a value of "xxilu's cluster" has been automatically generated
2016-05-19 00:19:43.567/2.506 Oracle Coherence GE 12.2.1.0.0 <Info> (thread=pool-1-thread-1, member=n/a): Created cache factory com.tangosol.net.ExtensibleConfigurableCacheFactory
2016-05-19 00:19:44.020/2.958 Oracle Coherence GE 12.2.1.0.0 <Warning> (thread=pool-1-thread-1, member=n/a): Local address "127.0.0.1" is a loopback address; this cluster node will not connect to nodes located on different machines
2016-05-19 00:19:44.069/3.008 Oracle Coherence GE 12.2.1.0.0 <D4> (thread=pool-1-thread-1, member=n/a): TCMP bound to /127.0.0.1:8888 using SystemDatagramSocketProvider
And it stops there.
If I change the address provider to the socket-address style (as commented out in the config), things will work.

Answer 1

[I posted the question earlier for documentation purpose. I've spent 10 hours debugging this issue and I finally found the root cause].
After spending 10 hours, I found the root cause.
There's one very subtle detail about AddressProvider implementation. The document says getNextAddress should return a null to indicate all addresses are exhausted. But there's one important thing it didn't say: after returning the null, the next call to getNextAddress should look like the very first call to this object. In another word, if you view getNextAddress as an iterator, the iterator need to reset after returning the terminating null.
So changing MyAddressProvider from this:
#Override
public InetSocketAddress getNextAddress() {
if(index >= 1) {
return null;
}
}
To this:
#Override
public InetSocketAddress getNextAddress() {
if(index >= 1) {
index = 0;
return null;
}
}
Things will work.
Hopefully this can help people hitting similar issue.