I am having trouble setting up a PXE VM. It is sending DHCP requests and the server is sending responses, but the VM does not appear to be processing the response. I am unsure as to the cause.
I did confirm physical machines are working just fine with the same DHCP and PXE settings and the DHCP requests and responses are the same as with the VM.
The DHCP server is provided by MaaS and is on the host.
Below is an image of the error.
The VM is created with: virt-install --name=maas-node-1 --connect=qemu:///system --ram=15360 --vcpus=8 --hvm --virt-type=kvm --pxe --boot network,hd --os-variant=ubuntu16.04 --graphics vnc --os-type=linux --accelerate --disk=/var/lib/libvirt/images/maas-node-1.qcow2,bus=virtio,format=qcow2,cache=none,sparse=true,size=60 --network=bridge:br0,model=virtio
The network are configured as:
auto br0
iface br0 inet static
address 192.168.10.2
network 192.168.10.0
broadcast 192.168.10.255
netmask 255.255.255.0
gateway 192.168.10.1
dns-nameservers 192.168.10.2
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
auto bond0
iface bond0 inet manual
mtu 1500
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-mode 0
bond-slaves none
post-up ifenslave bond0 eno1 eno2 eno3 eno4
pre-down ifenslave bond0 eno1 eno2 eno3 eno4
...
DHCP request is:
steel.maas.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 18:03:73:f8:ea:c9 (oui Unknown), length 257, xid 0xf97e014f, Flags [Broadcast] (0x8000)
Client-Ethernet-Address 18:03:73:f8:ea:c9 (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Client-ID Option 61, length 6: ieee1394 03:73:f8:ea:c9
Parameter-Request Option 55, length 3:
Default-Gateway, Subnet-Mask, Domain-Name-Server
DHCP Reply is:
steel.maas.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xf97e014f, Flags [Broadcast] (0x8000)
Your-IP steel.maas
Server-IP steel.maas
Client-Ethernet-Address 18:03:73:f8:ea:c9 (oui Unknown)
file "pxelinux.0"
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Offer
Server-ID Option 54, length 4: steel.maas
Lease-Time Option 51, length 4: 600
Subnet-Mask Option 1, length 4: 255.255.255.0
Default-Gateway Option 3, length 4: 192.168.10.1
Domain-Name-Server Option 6, length 4: steel.maas
The problem was that ARP was not being responded to due to a trait of using bond-mode 0 with no trunking on the switch. Switching to balance-tlb fixed the issues.
This helped narrow the problem down: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/785668
Related
Please tell me, how can I make it possible to transfer Trunk traffic at the user level in the Neutron Private Network?
Description
We have the following private, non-shared network without any Gateway:
I want to make sure that VMs can configure Trunk-VLAN connections to each other. At the same time, the creation of these connections was controlled at the OS level (the count of such connections and VMs is completely chaotic and random )
The problem is the following - if you make several VMs on, for example, Centos and try to set up the relationship of one VM to another through the Trunk port with ID, for example, 5, the Neutron network completely drop such traffic.
### Both VMs ###
[root#vlan-X centos]# modprobe bonding
[root#vlan-X centos]# modprobe 8021q
[root#vlan-X centos]# echo "8021q" > /etc/modules-load.d/8021q.conf
[root#vlan-X centos]# echo "bonding" > /etc/modules-load.d/bonding.conf
### VM1 ###
[root#vlan-1 centos]# vi /etc/sysconfig/network-scripts/ifcfg-eth1
NAME="eth1"
DEVICE="eth1"
ONBOOT="yes"
TYPE="Ethernet"
BOOTPROTO="none"
[root#vlan-1 centos]# vi /etc/sysconfig/network-scripts/ifcfg-eth1.5
ONBOOT=yes
VLAN=yes
DEVICE=eth1.5
BOOTPROTO=static
IPADDR=192.168.10.15
PREFIX=24
[root#vlan-1 centos]# systemctl restart network
### VM2 ###
[root#vlan-2 centos]# vi /etc/sysconfig/network-scripts/ifcfg-eth1
NAME="eth1"
DEVICE="eth1"
ONBOOT="yes"
TYPE="Ethernet"
BOOTPROTO="none"
[root#vlan-2 centos]# vi /etc/sysconfig/network-scripts/ifcfg-eth1.5
ONBOOT=yes
VLAN=yes
DEVICE=eth1.5
BOOTPROTO=static
IPADDR=192.168.10.16
PREFIX=24
[root#vlan-2 centos]# systemctl restart network
[root#vlan-2 centos]# ping 192.168.10.15
PING 192.168.10.15 (192.168.10.15) 56(84) bytes of data.
From 192.168.10.16 icmp_seq=1 Destination Host Unreachable
From 192.168.10.16 icmp_seq=2 Destination Host Unreachable
From 192.168.10.16 icmp_seq=3 Destination Host Unreachable
From 192.168.10.16 icmp_seq=4 Destination Host Unreachable
From 192.168.10.16 icmp_seq=5 Destination Host Unreachable
From 192.168.10.16 icmp_seq=6 Destination Host Unreachable
From 192.168.10.16 icmp_seq=7 Destination Host Unreachable
From 192.168.10.16 icmp_seq=8 Destination Host Unreachable
^C
--- 192.168.10.15 ping statistics ---
11 packets transmitted, 0 received, +8 errors, 100% packet loss, time 10001ms
pipe 4
At the same time, tcpdump looks like this - that is, VLAN was successfully accepted, but then even ARP is unsuccessful
[root#vlan-2 centos]# tcpdump -e -nvvvti eth1
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
fa:16:3e:8f:7f:df > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 5, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.10.15 tell 192.168.10.16, length 28
fa:16:3e:8f:7f:df > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 5, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.10.15 tell 192.168.10.16, length 28
fa:16:3e:8f:7f:df > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 5, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.10.15 tell 192.168.10.16, length 28
fa:16:3e:8f:7f:df > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 5, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.10.15 tell 192.168.10.16, length 28
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
At the same time, the same design on VirtualBox with an internal network works fine.
What I have tried to solve this problem:
I tried creating a network with option "--transparent-vlan" - nothing changed
I tried creating a network with option "--transparent-vlan --disable-port-security" - nothing changed
I tried to connect the "Trunk" option to Neutron and configure additional entities like "network trunk" and "subport" - nothing changed. I had a suspicion that this was all intended to set up communication between two VMs on different networks, and not many VMs on the same network.
In general, this option has one limitation - it only works with Linuxbridge
A worked configuration for me:
vi /etc/neutron/neutron.conf
...
[DEFAULT]
vlan_transparent = true
...
vi /etc/neutron/plugins/ml2/ml2_conf.ini
...
[ml2]
type_drivers = flat,vlan,vxlan,gre
tenant_network_types = vxlan
mechanism_drivers = linuxbridge
...
vi /etc/neutron/plugin.ini
...
[ml2]
type_drivers = flat,vlan,vxlan,gre
mechanism_drivers = linuxbridge
...
And only then I was able to create a network with option "vlan_transparent", but not through the CLI, but through the API:
$ curl -s -X POST http://internal.mystack.net:9696/v2.0/networks -H "X-Auth-Token: <TOKEN>" -H "Content-Type: application/json" -d '{"network": {"name": "test", "admin_state_up": true, "tenant_id": "56b0cfe82ef94b2b8a60c53d72921a8b", "vlan_transparent": true}}'
$ openstack network show test --debug
...
{"networks": [{"provider:physical_network": null, "ipv6_address_scope": null, "dns_domain": null, "revision_number": 4, "port_security_enabled": true, "provider:network_type": "vxlan", "id": "78af4991-1b50-4b8d-9299-3a5dfaf689a2", "router:external": false, "availability_zone_hints": [], "availability_zones": [], "ipv4_address_scope": null, "shared": false, "project_id": "56b0cfe82ef94b2b8a60c53d72921a8b", "status": "ACTIVE", "subnets": [], "private_dns_domain": "mcs.local.", "description": "", "tags": [], "updated_at": "2022-03-29T13:22:00Z", "provider:segmentation_id": 88, "name": "test", "admin_state_up": true, "tenant_id": "56b0cfe82ef94b2b8a60c53d72921a8b", "created_at": "2022-03-29T13:22:00Z", "mtu": 1400, "vlan_transparent": true}]}
That is, if you add any "openvswitch" or "l2population" to the configuration in the "mechanism_drivers", then all this will abruptly stop working
I didn't develop the experiment further. realized that it doesn't suit me
Hi I'm working on a project and I had a question involving ping commands and how they interface over network TUN ports.
Basically I'm sending out ping requests which are routed to my TUN port and the reply's are sent to the TUN port over the VPN. There are no other internet interfaces (i.e. no wifi/ethernet). Using wireshark and tcpdump I can see that the correct reply messages are seen on the TUN0 port but terminal does not see the replys and instead shows 100% drop rate. The issue seems to be that the TUN0 port is not properly linking back to the kernal? (total guess I'm quite new to IP routing).
The IP address of the TUN is 10.0.0.73 and I am pinging a computer with IP address 10.0.0.28
Bellow is a snippet from the tcpdump on TUN0 this is a request and reply that to my untrained eye should work:
23:08:52.257566 IP (tos 0x0, ttl 64, id 11185, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.0.73 > 10.0.0.28: ICMP echo request, id 24667, seq 2, length 64
23:09:11.508002 IP (tos 0x0, ttl 64, id 13315, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.28 > 10.0.0.73: ICMP echo reply, id 24667, seq 2, length 64
Based on other posts I checked my ip route list and the output is as such
pi#raspberrypi:~$ sudo ip route list
10.0.0.0/24 dev tun0 proto kernel scope link src 10.0.0.73
and the ifconfig is this:
pi#raspberrypi:~$ ifconfig tun0
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.0.0.73 P-t-P 10.0.0.73 Mask:255.255.255.0
...
Turns out the issue was that the replies were showing up in incorrect orders and greatly delayed, when I fixed the network connections this issue went away without changing any configurations in the iptables
I ran the raspbian image with the following command:
qemu-system-arm -kernel kernel-qemu-4.19.50-buster -cpu arm1176 -m 256 -M versatilepb -dtb versatile-pb.dtb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" -drive "file=2020-02-13-raspbian-buster-lite.img,index=0,media=disk,format=raw" -net user,hostfwd=tcp::5022-:22 -net nic -net user,smb=/dev/shm/
Booting the image completed successfully.
Withing guest machine I get the following routing table:
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.2.2 0.0.0.0 UG 202 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 202 0 0 eth0
Now when pinging the gateway at 10.0.2.2 works fine, but when pinging the host machine or the host gateway at 10.0.0.138 I get:
pi#raspberrypi:~$ ping 10.0.0.138
PING 10.0.0.138 (10.0.0.138) 56(84) bytes of data.
64 bytes from 10.0.0.138: icmp_seq=1 ttl=255 time=1.19 ms
64 bytes from 10.0.0.138: icmp_seq=1 ttl=255 time=1.23 ms (DUP!)
I verified that 10.0.0.138 isn't defined as broadcast address, and there are no IP duplications. Any idea how to debug from here? Thanks
As Peter Maydell suggested, merging the two options into one "-net user,smb=/dev/shm/,hostfwd=tcp::5022-:22" solved the case.
This is because QEMU creates a new 'user' network backend for each use of '-net user' on the command line, so in the original commandline there were two backends, each of which was responding to ping packets.
Case:
[ Subnet A , 192.168.2.0/24, Padavan firmware based internet gw ]
[ Subnet B , 192.168.1.0/24, Padavan firmware based internet gw ]
Host from subnet A (2.155) is connected via VPN (possible options: PPTP, OpenVPN, L2TP w/o ipsec) to subnet B, and receives address, saying 1.245/32
In subnet B exists host (1.10/32) which sends multicast datagramms to 224.0.0.50:9898 ; On router I see them with
tcpdump -i br0 -c 10 dst host 224.0.0.50 and port 9898 and multicast
13:46:54.345369 IP 192.168.1.10.4321 > 224.0.0.50.9898: UDP, length 135
I am looking for solutions, to receive/forward those broadcast messages, so they could be seen by hosts, connected via VPN
On router B, which is Padavan firmware based, I have, and limited to udpxy, igmproxy utilities, if needed.
On client host, I am debian based, and generally not limited in tools.
Datagrams are proprietary protocol, i.e. not a iptv or video stream.
Any ideas are welcomed.
[UPD] Additional info - per discussion in comments
That's a very specific hardware device, which is not very chatty in ethernet terms (saying max 1-2 datagramms in 5 seconds), thus for sure should be pretty forwardable. Unfortunately, It sends status updates purely via broadcasting. in Subnet A do exist similar device + control software. Thus I am looking for a way datagramms broadcasted to 224.0.0.50:9898 in subnet B to re-appear in subnet A. May be with help of some tool. May be smcroute, may be udpxy, maybe igmproxy
As I don't like to leave resolved questions unanswered, here is currently working solution
In subnet B I have installed openVPN server endpoint, configured as L2.
In subnet A, on a control host I have installed openvpn client, that connects to subnet B, assigned interface is tapz
20: tapz: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/ether 0a:da:be:96:78:d9 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.245/24 brd 192.168.1.255 scope global noprefixroute tapz
valid_lft forever preferred_lft forever
inet6 fe80::8da:beff:fe96:78d9/64 scope link
valid_lft forever preferred_lft forever
So now on a control host I have:
broadcasting from local device on physical ethernet enp5s0
sudo tcpdump -i enp5s0 -c 10 dst host 224.0.0.50 and port 9898 and multicast
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp5s0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:55:05.642963 IP lumi-gateway-v3_miio56591509.4321 > 224.0.0.50.9898: UDP,
length 136
and now I also receive broadcasts from remote network device on tapz
sudo tcpdump -i tapz -c 10 dst host 224.0.0.50 and port 9898 and multicast
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapz, link-type EN10MB (Ethernet), capture size 262144 bytes
13:53:49.141751 IP 192.168.1.10.4321 > 224.0.0.50.9898: UDP, length 135
So far that it what I was looking for I am getting necessary datagrams on a VPN client. OpenVPN on remote side can be also optimized on filter of information forwarded for multicasts.
For those who come here, with the same question.
When you will have necessary multicast on tap0,
you can create bridge from, saying, eth0 and tap0
For notes of everyone interested, who would came here.
ip link add br0 type bridge
ip link set tap0 master br0
ip link set eth0 master br0
POC - both multicasts on single interface
sudo tcpdump -i br0 dst host 224.0.0.50 and port 9898
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:09:51.823632 IP 192.168.1.10.4321 > 224.0.0.50.9898: UDP, length 135
21:09:55.045138 IP 192.168.2.214.4321 > 224.0.0.50.9898: UDP, length 136
I installed Openstack Ansible, Pike version. There is a separate network controller and on it one physical network interface. We created VLAN 139 that leads the traffic to gateway. Config file for that part looks like:
/etc/network/interfaces
...
auto eno1.139
iface eno1.139 inet manual
vlan-raw-device eno1
# OpenStack Networking VLAN bridge
auto br-vlan
iface br-vlan inet manual
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge_ports eno1.139
We created an external Openstack network using:
openstack network create --external --share --provider-physical-network vlan --provider-network-type vlan --provider-segment 139 provider1
and all the other steps (subnet, router, etc)
As per documentation, first test should be pinging default gateway from router namespace. When I try that it is not working:
root#infra1-neutron-agents-container-e800e983:/# ip netns exec qrouter-eb842b12-9a35-4a93-baa9-38cc73531d9f ping 139.25.25.193
When I do TCP dump on physical network interface of controller node I can see packets going out without any problem:
openstackadmin#clcontroller:~$ sudo tcpdump -i eno1 --immediate-mode -e -n | grep 139.25.25.193
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:30:09.182894 fa:16:3e:d4:b6:a1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 50: vlan 139, p 0, ethertype 802.1Q, vlan 139, p 0, ethertype ARP, Request who-has 139.25.25.193 tell 139.25.25.200, length 28
I see ARP request getting to gateway that has 139.25.25.193 and I am trying to ping:
hpadmin#hos-gw01:~$ sudo tcpdump -i any --immediate-mode -e -n | grep 139.25.25.193
[sudo] password for hpadmin:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:53:29.857281 B fa:16:3e:d4:b6:a1 ethertype 802.1Q (0x8100), length 62: vlan 139, p 0, ethertype 802.1Q, vlan 139, p 0, ethertype ARP, Request who-has 139.25.25.193 tell 139.25.25.200, length 38
15:53:29.857281 B fa:16:3e:d4:b6:a1 ethertype 802.1Q (0x8100), length 58: vlan 139, p 0, ethertype ARP, Request who-has 139.25.25.193 tell 139.25.25.200, length 38
but what is confusing is my gateway is not responding to those ARP requests.
If I try to do same thing from stand alone Linux machine connected to same network segment and same VLAN everything works perfect.
Any idea what the problem might be? Thanks in advance.
It seems that problem was that external OpenStack network was set up to be on VLAN 139. Once we changed it to be flat everything started working without any problems. I am still confused, though, why gateway did not sent ARP responses.