HP Vertica - cannot connect and copy from different cluster - networking

Question

I'm trying to migrate a Vertica cluster.
The connect statement succeeds:
CONNECT TO VERTICA mydb USER myuser PASSWORD '' on 'private-ip',5433;
The copy does not:
COPY otherdb.whatever FROM VERTICA mydb.table.whatever;
ERROR 4534: Receive on v_otherdb_node0002: Message receipt from v_otherdb_node0005 failed [canceled]
ROLLBACK 4236: One or more nodes did not open a data connection to this node. This may indicate a network configuration problem. Check that the private interfaces used for communication among the cluster hosts reside in the same subnet and are returned first by host address lookup
/etc/hosts on both clusters are configured correctly so that hostname maps to private-ip across all machines. Telnet, ping, netcat connectivity is all good.
I have tried various options:
Use identical catalog and data paths in case a mismatch is not tolerated
Use the same db name and different db names - no difference
The 2 clusters have 6 machines each, and each cluster is healthy and all nodes are up and responsive.
Any suggestions on what else I can try?

Answer 1

The issue was that one of the clusters had been set to EXPORT ON a different subnet than the private subnet.
Doing netcat, I discovered that:
My new cluster was sending the old cluster it's correct private node IPs
The old cluster was responding with it's node public IPs, even though "select * from nodes;" was returning the private IPs
From this I ran:
select * from vs_subnets;
select database_name, export_subnet from databases;
It turns out somebody (long ago) had configured the old cluster to use the public subnet.
You can revert this by running:
ALTER DATABASE <my-db-name> EXPORT ON DEFAULT;
This makes the DB go back to its regular subnet.