Currently I am using one availability zone in my ec2 launch config. It is important that I don't get network partitions in my app, as rabbitmq does not handle network partitions well when clustering and HA is used (which I am using).
I am very fuzzy on the concept of network partitions. Would it be safe for me to use two availability zones?
The different Amazon EC2 Availability Zones are in different physical locations. While the connections between availability zones are quite good, it is still a WAN connection.
From the RabbitMQ docs
RabbitMQ clusters do not tolerate network partitions well. If you are thinking of clustering across a WAN, don't. You should use federation or the shovel instead
(emphasis mine)
https://www.rabbitmq.com/partitions.html
In short, a 1 minute or so interruption in connectivity will cause a network partition to be created. While this would be an unusual event for EC2, it can and sometimes will happen.
Related
I am using a Fabric 2.2 network with two organizations, one endorsing peer per organization and one ordering service.
I want to scale up my network with many clients and servers in order to test the performance of the system (latency and throughput), similarly to what it is done in this paper ieeexplore.ieee.org/iel7/69/4358933/08246573.pdf in section 6.1.2
I confuse maybe the terms. I am really new to Fabric.
What is the meaning of "clients" and "servers" in the paper? Are they ca-servers and ca-clients? In this case, how to configure a network with an increasing number of Fabric clients and servers?
We want to use zero tier to connect from one cloud machine to multiple remote machines. We do not want remote machines to access each other. What would be a good approach?
Use a single network and set rules based on tags to restrict access
Run multiple networks, each having cloud machine and a remote machine
Are there limits to
Number of members in zerotier network
Number of zerotier networks a machine can connect to at a time - tun interfaces, ip conflicts or performance impact
I would use a single network and use rules to prevent peering between the machines. For instance, you could set the 192.168.141.0/25 portion of the network to prevent peering, and allow only defined network paths between hosts.
Just a personal rant here: You don't want to do that. Really. You're going to make a headache for yourself when you have to scale horizontally (which you will if you're successful). I would STRONGLY recommend taking a mTLS approach to service authentication instead. Somewhat more work at the start, but a lot easier in the long run.
I'm in a high availability project which includes deployment of 2-node high availability cluster for hot replacement of services (applications) running on the cluster nodes. The applications have inbound and outbound tcp connections as well as process udp traffic (mainly for communicating with ntp server).
The problem is pretty standard until one needs to provide a hot migration of services to backup node with all the data stored in RAM. Applications are agnostic of backup mechanisms and it is highly undesirable to modify them.
As only approach to this problem, I've come off with a duplication approach assuming that both cluster nodes will run the same applications repeating calculations of each other. In case of failure the primary server the backup server will become a primary.
However, I have not found any ready solution for proxy which will have synchronous port mirroring. No existing proxy servers (haproxy, dante, 3proxy etc.) support such feature as far as I know. Have I missed something, or I should write a new one from scratch?
A rough sketch of the functionality can be found here:
p.s. I assume that it is possible to compare traffic from the two clones of the same application...
Amazon / AWS EC2 offers SR-IOV (Single Root I/O Virtualization) instances, which it dubs "enhanced networking" -- does Google offer this on Compute Engine?
Specifically, are any GCE instance types able to bypass the hypervisor and have direct access to a multi-queue NIC?
SRV-IOV support is needed to take advantage of Scylla DB's architecture?
HN Discussion: https://news.ycombinator.com/item?id=10262719
Currently Google Compute Engine does not offer SR-IOV. That said, SR-IOV is not strictly necessary to take advantage of Scylla's architecture.
GCE offers multi-queue networking and it is possible to directly user-mode assign the virtio-net queues using Intel's DPDK. This should allow our virtio-net NIC to work with Scylla, although at least at one point DPDK made certain qemu specific assumptions with respect to virtio-net (in particular it assumed Tx/Rx queue depths of 256 descriptors; the virtio-net NIC in GCE currently advertises 16,384 entry queues although this is likely to change in the near future).
For applications like Scylla this should offer superior network performance and better in-guest compute overhead over utilizing the kernel TCP/IP stack.
Additionally, for all GCE instances with >= 1 cores (i.e., not fractional core instances) we offer multi-Gbps throughput subject to fabric availability. Latency is likely to be lowest in zones with Haswell processors. We do not currently guarantee specific network characteristics, but we offer up to 2 Gbps/core of network throughput shared between the virtual NIC and any attached persistent disk volumes (Local SSD throughput does not count against this limit). Throughput wise this makes 8-vCPU and larger instances comparable to EC2 Enhanced Networking.
At the moment, nothing that we offer is similar to AWS' "enhanced networking".
You are more than welcome posting this as a Feature Request on our Compute Engine Issue tracker though, so we can look at implementing a similar feature.
I am currently in the process of creating 3 Neo4j High Availability servers. My business logic leaves one server as a dedicated master, while the other two machines are dedicated slaves. My slaves exist in an entirely different datacenter than my master.
What is the best method to establish a link between the two applications? I've been able to establish connections using OpenVPN, but am curious if that would be better than like SSH port forwarding? I'm not entirely sure how Zookeeper needs to communicate with each other node. A VPN connection only creates a one-way connection, where my master, for example, can create a connection with slave, but could not create one with its master. (I think?)
How should I do this? Thanks!
PS: My master is using an embedded instance of Neo4j, while the slaves are stand-alone instances (if this matters).
So your setup is not about availability as the slaves cannot become masters anyway?
Just about replication to the other datacenter?
You also need to take the neo4j coordinator (zookeeper) into account which is usually needed for all cluster participants.
My colleague suggested that you might get away with just putting the zookeeper (perhaps even just a single one as you don't need master election) directly besides your master server.
Then the ability to connect into the masters' VPN should be enough for the slaves to pull updates.