I am currently in the process of creating 3 Neo4j High Availability servers. My business logic leaves one server as a dedicated master, while the other two machines are dedicated slaves. My slaves exist in an entirely different datacenter than my master.
What is the best method to establish a link between the two applications? I've been able to establish connections using OpenVPN, but am curious if that would be better than like SSH port forwarding? I'm not entirely sure how Zookeeper needs to communicate with each other node. A VPN connection only creates a one-way connection, where my master, for example, can create a connection with slave, but could not create one with its master. (I think?)
How should I do this? Thanks!
PS: My master is using an embedded instance of Neo4j, while the slaves are stand-alone instances (if this matters).
So your setup is not about availability as the slaves cannot become masters anyway?
Just about replication to the other datacenter?
You also need to take the neo4j coordinator (zookeeper) into account which is usually needed for all cluster participants.
My colleague suggested that you might get away with just putting the zookeeper (perhaps even just a single one as you don't need master election) directly besides your master server.
Then the ability to connect into the masters' VPN should be enough for the slaves to pull updates.
Related
I have two services that are deployed in the two different VMs. I want to call one service from other to get some data via APIs. That should need to have an ultra-low latency call.
For that how can I used the Chronicle-Network (https://github.com/OpenHFT/Chronicle-Network)?
Or Any other solution?
I would suggest using Chronicle-Queue for passing messages from one service to another and back again. This is much lower latency than using TCP and easier to work with, provided the two VMs are on the same machine. You also get a record of every message making testing/debugging easier.
https://github.com/OpenHFT/Chronicle-Queue#high-level-interface-for-readingwriting
If you have two different machines I suggest either moving one VMs or using a low latency network card like Solarflare or Mellanox.
If you have a network call between VMs, try to use Chronicle Network.
Another option is Aeron UDP unicast connection - https://github.com/real-logic/aeron
I'm in a high availability project which includes deployment of 2-node high availability cluster for hot replacement of services (applications) running on the cluster nodes. The applications have inbound and outbound tcp connections as well as process udp traffic (mainly for communicating with ntp server).
The problem is pretty standard until one needs to provide a hot migration of services to backup node with all the data stored in RAM. Applications are agnostic of backup mechanisms and it is highly undesirable to modify them.
As only approach to this problem, I've come off with a duplication approach assuming that both cluster nodes will run the same applications repeating calculations of each other. In case of failure the primary server the backup server will become a primary.
However, I have not found any ready solution for proxy which will have synchronous port mirroring. No existing proxy servers (haproxy, dante, 3proxy etc.) support such feature as far as I know. Have I missed something, or I should write a new one from scratch?
A rough sketch of the functionality can be found here:
p.s. I assume that it is possible to compare traffic from the two clones of the same application...
Long story short - I need to use networking between projects to have separate billing for them.
I'd like to reach all the VMs in different projects from a single point that I will use for provisioning systems (let's call it coordinator node).
It looks like VPC network peering is a perfect solution to this. But unfortunately one of the existing networks is "legacy". Here's what google docs state about legacy networks.
About legacy networks
Note: Legacy networks are not recommended. Many newer GCP features are not supported in legacy networks.
OK, naturally the question arises: how do you migrate out of legacy network? Documentation does not address this topic. Is it not possible?
I have a bunch of VMs, and I'd be able to shutdown them one by one:
shutdown
change something
restart
unfortunately it does not seem possible to change network even when VM is down?
EDIT:
it has been suggested to recreate VMs keeping the same disks. I would still need a way to bridge legacy network with new VPC network to make migration fluent. Any thoughts on how to do that using GCE toolset?
One possible solution - for each VM in the legacy network:
Get VM parameters (API get method)
Delete VM without deleting PD (persistent disk)
Create VM in the new VPC network using parameters from step 1 (and existing persistent disk)
This way stop-change-start is not so different from delete-recreate-with-changes. It's possible to write a script to fully automate this (migration of a whole network). I wouldn't be surprised if someone already did that.
UDPATE
https://github.com/googleinterns/vm-network-migration tool automates the above process, plus it supports migration of a whole Instance Group or Load Balancer, etc. Check it out.
Two computers in separate NAT networks, wants to connect with each other.
No port forwarding can be made.
We can connect between them using a intermediate server, however we dont want the entire communication to go through this server after the initial connectivity.
Is there any way that the intermediate server can perform the initial connection and then allow them to continue their communication directly somehow ?
Everything I know about computer networks say 'no', however I though I'll give this question a try here.
It can be done vía NAT traversal techniques. It isn't simple but many applications do it, e.g. Skype, where channeling all communication through a server would be innefective.
Consider the following situation:
You have an application you are tesing, but in order to test the networking functionality of said program, you are required to run multiple instances of it and have them communicate with one another.
Possible solutions are:
- Run software on individual machines connected by WAN or LAN.
- Run the software on virtual machines, all on the same computer.
I do not want to use either of these methods (the reasoning is irrelevant). I want to know if there is a way that I can reroute network transmissions from the test application (ideally in any programmming language) in a way such that I can run multiple instances of the same software on one computer, and have them behave as if they were the only instance running on that computer.
In other words, I want to be able to code the application so that each instance listens on the same "listening" port (since only one instance will be running on each computer when in production). Then, I want to know if I can reroute the network requests at a lower level then the application so that they do not interfere with eachother (clash over the same port number).
Essentially, I want to build a virtual environment which only redirects the network calls (whereas a virtual machine takes far more resources, and has way more involved). Is this possible, and how might I approach this problem?
Thank you!
UPDATE: This is a more accurate idea of what I want to accomplish:
Basically, I want to program another application which TRANSPARENTLY redirects bind requests to available ports, and manages which applications are bound where... So from the applications perspective, all the instances are bound to port 1000, but in reality, this other application is automatically managing which instance is bound where, and avoiding potential conflicts. I feel like this could be accomplished with Windows Hooks, but I'm not sure how you could implement this?
As far as I know, there is no sane way to multiplex the same port on the same network device. At the very minimum, you will need to choose on of the following:
Run each instance of your program on a different port
Create multiple virtual network interfaces
The first choice is easy and may be the one I would choose. The second one is more towards what you are looking for but it would be a true PITA to set up - you can look into VirtualBox and its host-only networks for inspiration. If you are writing things on linux you might look into pipes and chrooting but you'll be spending more time setting up this environment than writing your software.