Octavia: Trying to delete immutable loadbalancer - openstack

Question

I have a loadbalancer (see status below) that I want to delete. I already deleted the instances in its pool. Full disclosure: This is on a Devstack which I rebooted, and where I recreated the lb-mgmt-network routing manually. I may have overlooked a detail after the reboot. The loadbalancer worked before the reboot.
The first step to delete the loadbalancer is to delete its pool members. This fails as follows:
$ alias olb='openstack loadbalancer'
$ olb member delete website-pool 08f55..
Load Balancer 1ff... is immutable and cannot be updated. (HTTP 409)
What can I do to make it mutable?
Below, see the loadbalancer's status after recreating the o-hm0 route and restarting the amphora. Its provisioning status is ERROR, but according to the API, this should enable me to delete it:
$ olb status show kubelb
{
"loadbalancer": {
"id": "1ff7682b-3989-444d-a1a8-6c91aac69c45",
"name": "kubelb",
"operating_status": "ONLINE",
"provisioning_status": "ERROR",
"listeners": [
{
"id": "d3c3eb7f-345f-4ded-a7f8-7d97e3af0fd4",
"name": "weblistener",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "9b0875e0-7d16-4ebc-9e8d-d1b90d4264a6",
"name": "website-pool",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"members": [
{
"id": "08f55bba-260a-4b83-ad6d-f9d6b44f0e2c",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "172.16.0.21",
"protocol_port": 80
},
{
"id": "f7665e90-dad0-480e-8ef4-65e0a042b9fa",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "172.16.0.22",
"protocol_port": 80
}
]
}
]
}
]
}
}

Answer 1

When you have a load balancer in ERROR state you have two options:
Delete the load balancer using the cascade delete option (--cascade on the cli).
Use the failover API to tell Octavia to repair the load balancer once your cloud is fixed.
In Octavia, operating status is a measured/observed status. If they don't go ONLINE it is likely that there is a network configuration issue with the lb-mgmt-net and the health heartbeat messages (UDP 5555) are not making it back to the health manager controller.
That said, devstack is not setup to work after a reboot. Specifically neutron and the network interfaces will be in an improper state. As you have found, you can manually reconfigure those and usually get things working again.

Answer 2

If I understand documentation and source code right, a loadbalancer in provisioning status ERROR can be deleted but not modified. Unfortunately, it can only be deleted after its pools and listeners have been deleted, which would modify the loadbalancer. Looks like a chicken and egg problem to me. I "solved" this by recreating the cloud from scratch. I guess I could also have cleaned up the database.
An analysis of the stack.sh log file revealed that a few additional steps were needed to make the Devstack cloud reboot-proof. To make Octavia ready:
Create /var/run/octavia, owned by the stack user
Ensure o-hm0 is up
Give o-hm0 the correct MAC and IP addresses, both found in the details of Neutron port octavia-health-manager-standalone-listen-port
add netfilter rules for traffic coming from o-hm0
At this point, I feel I can reboot Devstack and still have functioning load balancers. Strangely, all load balancers' operating_status (as well as their listeners' and pools' operating_status) is OFFLINE. However, that doesn't prevent them from working. I have not found out how to make that ONLINE.
In case anybody is interested, below is the script I use after rebooting Devstack. In addition, I also changed the Netplan configuration so that br-ex gets the server's IP address (further below).
restore-devstack script:
$ cat restore-devstack
source ~/devstack/openrc admin admin
if losetup -a | grep -q /opt/stack/data/stack-volumes
then echo loop devices are already set up
else
sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-default-backing-file
sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
echo restarting Cinder Volume service
sudo systemctl restart devstack#c-vol
fi
sudo lvs
openstack volume service list
echo
echo recreating /var/run/octavia
sudo mkdir /var/run/octavia
sudo chown stack /var/run/octavia
echo
echo setting up the o-hm0 interface
if ip l show o-hm0 | grep -q 'state DOWN'
then sudo ip l set o-hm0 up
else echo o-hm0 interface is not DOWN
fi
HEALTH_IP=$(openstack port show octavia-health-manager-standalone-listen-port -c fixed_ips -f yaml | grep ip_address | cut -d' ' -f3)
echo health monitor IP is $HEALTH_IP
if ip a show dev o-hm0 | grep -q $HEALTH_IP
then echo o-hm0 interface has IP address
else sudo ip a add ${HEALTH_IP}/24 dev o-hm0
fi
HEALTH_MAC=$(openstack port show octavia-health-manager-standalone-listen-port -c mac_address -f value)
echo health monitor MAC is $HEALTH_MAC
sudo ip link set dev o-hm0 address $HEALTH_MAC
echo o-hm0 MAC address set to $HEALTH_MAC
echo route to loadbalancer network:
ip r show 192.168.0.0/24
echo
echo fix netfilter for Octavia
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 20514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 10514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 5555 -j ACCEPT
echo fix netfilter for Magnum
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 443 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 9511 -j ACCEPT
Netplan config:
$ cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp1s0:
dhcp4: no
br-ex:
addresses: [192.168.1.200/24]
nameservers: { addresses: [192.168.1.16,1.1.1.1] }
gateway4: 192.168.1.1
version: 2