Packets drop at Open vSwitch - openstack

In our OpenStack environment, we did see huge packets lost. Then we found packets were dropped at Open vSwitch. Could someone give clue how to improve the situation?
[bscuser#compute-4 ~]$ sudo ovs-ofctl dump-ports br-int vhub97ae049-a2
OFPST_PORT reply (xid=0x4): 1 ports
port "vhub97ae049-a2": rx pkts=12472105, bytes=1647807101, drop=0, errs=0, frame=?, over=?, crc=?
tx pkts=83585797, bytes=77693614917, drop=4643534, errs=?, coll=?
[root#compute1 ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000c272e446ba49
n_tables:254, n_buffers:0
And n_buffers is 0, is it normal? I searched on web, it seems all the result are 256. But I don't know how to change it.
Thank you in advance.

Related

How to fix high latency and retransmission rate in Ubuntu 18.04

I installed Ubuntu 18.04 on Hyper-V Win Server 2016.
And network performance of the Ubuntu is bad: I'm hosting few sites (Apache + PHP) and sometime response time is > 10 seconds. Sometimes it is fast.
As I troubleshooted, I see this netstat results:
# netstat -s | egrep -i 'loss|retran'
3447700 segments retransmitted
226 times recovered from packet loss due to fast retransmit
Detected reordering 6 times using reno fast retransmit
TCPLostRetransmit: 79831
45 timeouts after reno fast retransmit
6247 timeouts in loss state
2056435 fast retransmits
107095 retransmits in slow start
TCPLossProbes: 220607
TCPLossProbeRecovery: 3753
TCPSynRetrans: 90564
What can be cause of such high "segments retransmitted" number? And how to fix it?
Few notes:
- VMQ is disabled for Ubuntu VM
- The host system Network adapter is Intel I210
- I disabled IPv6 both on host and in VM
Here is WireShark showing, that it takes ~7 seconds to connect (just initial connection) to my site Propovednik.com:
Sep 20: So far, the issue seems to be caused by OVH / SoYouStart bad network:
This command shows 20-30% packets loss:
sudo ping us.soyoustart.com -c 10 -i 0.2 -p 00 -s 1200 -l 5
The problem could be anywhere along the network, including the workstation where you work from. I suggest you check the network as retransmissions and packetloss means that either something is malfunctioning or misconfigured. If this is on a wireless network, you could be out of range of your router.
I am pinging the website you noted from my computer and there is no packetloss.

Serial communication between QEMU host and guest

I'm trying to set up a pair of serial ports between my QEMU host (Debian Jessie x86_64) and guest (also Debian Jessie, but on ARM). Everything except the serial port part works.
I'm really new to QEMU so there might be a better way but I've tested the following flags when running QEMU:
-chardev tty,id=mytty,path=/dev/pts/2 (/dev/pts/2 & 3 are up with socat)
-chardev pty,id=mypty QEMU opens a PTY but when I try to read or write from host get permission denied.
In either case I can't find the ports in my guest. /dev/pts is empty and in /dev there are only tty and ttyAMA3. So, my problem is setting up communication in general and I'm especially curious on where the ports are on my guest.
I found a solution to my own question. First the device tree was incomplete so I needed to add 3 additional uart ports. That's the reason I could not find my ports in the guest.
Second, I needed to tell QEMU to use on of the ports as stdio: -append ... console=ttyAMA3 and -serial mon:stdio. Then I'm able to, with -serial pty, link QEMUs ttyAMA* to pts/* on the host.

Preventing TCP SYN retry in netcat (for port knocking)

I'm trying to write the linux client script for a simple port knocking setup. My server has iptables configured to require a certain sequence of TCP SYN's to certain ports for opening up access. I'm able to successfully knock using telnet or manually invoking netcat (Ctrl-C right after running the command), but failing to build an automated knock script.
My attempt at an automated port knocking script consists simply of "nc -w 1 x.x.x.x 1234" commands, which connect to x.x.x.x port 1234 and timeout after one second. The problem, however, seems to be the kernel(?) doing automated SYN retries. Most of the time more than one SYN is being send during the 1 second nc tries to connect. I've checked this with tcpdump.
So, does anyone know how to prevent the SYN retries and make netcat simply send only one SYN per connection/knock attempt? Other solutions which do the job are also welcome.
Yeah, I checked that you may use nc too!:
$ nc -z example.net 1000 2000 3000; ssh example.net
The magic comes from (-z: zero-I/O mode)...
You may use nmap for port knocking (SYN). Just exec:
for p in 1000 2000 3000; do
nmap -Pn --max-retries 0 -p $p example.net;
done
try this (as root):
echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
or this:
int sc = 1;
setsockopt(sock, IPPROTO_TCP, TCP_SYNCNT, &sc, sizeof(sc));
You can't prevent the TCP/IP stack from doing what it is expressly designed to do.

Cannot connect to beaglebone.local

I need to know how to connect to a beaglebone (or beagleboard) with SSH when I plug it into a new network with an ethernet cable like this:
$ ssh root#beaglebone.local
So far I've only been able to access it like this, if I know the IP address:
$ ssh root#<ip_address>
But I don't always know the IP address of the board on new networks so I'm hoping to access it with with a name like: beaglebone.local.
Right now when I try to do this I get this error:
"ssh: Could not resolve hostname beaglebone.local: nodename nor servname provided, or not known"
I checked the hostname and hosts files, and added "127.0.0.1 beaglebone" to the hosts on the beaglebone, but not sure what else I can do?
# cat /etc/hostname
beaglebone
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 beaglebone
I had a similar issue running my beaglebone on Angstrom-Cloud9-IDE-GNOME-eglibc-ipk-v2012.05-beaglebone-2012.04.22.img.xz. In this distribution, "beaglebone.local" should appear on the network after the system boots.
About 50% of the time after reboot, "beaglebone.local" would not appear on the network (although the bone would be available by IP address). When this happened, "systemctl status avahi-daemon.service" showed that the avahi-daemon failed with "exit code 255". Interestingly, a subsequent "systemctl start avaihi-daemon.service" would always be successful and "beaglebone.local" would appear on the network.
Also "journalctl | grep avahi" returned a single message stating something like "Daemon already runnin gon PID NNN".
So, I "fixed" the problem by adding the line "ExecStartPre=/bin/rm -f /var/run/avahi-daemon/pid" to the [Service] section of /lib/systemd/system/avahi-daemon.service. With this addition, "beaglebone.local" now appears on the network 100% of reboots.
I say "fixed" (i.e., in quotes) because I have not been able to track down the root cause that is leaving around the stray avahi pid file(s) and thus don't have a true fix.
-- Frank
For 'beaglebone.local' to work, your host machine must recognize Zeroconf. The BeagleBone uses Avahi to tell other systems on the LAN that it is there and serving up applications and that it should be called a 'beaglebone'. If there are more than one, the second one is generally called 'beaglebone-2.local'.
I hate answering my own questions. The following hack will work until a better way emerges:
This shell script (where xxx.xxx.xxx is the first three numbers in your computer's IP) will find your beaglebone or beagleboard (that is plugged-into ethernet on a new network with DHCP) by looping through all the ip address on the subnet and attempting to login to each as root. If it finds one then try your password. If it doesn't work just hit enter until the loop starts again. If it doesn't find the board then something else is probably wrong.
for ip in $(seq 1 254); do ssh root#xxx.xxx.xxx.$ip -o ConnectTimeout=5; [ $? -eq 0 ] && echo "xxx.xxx.xxx.$ip UP" || : ; done
UPDATE 1
Today I plugged-in the beaglebone and saw Bonjour recognize that it joined the network. So I tried it and it worked. No idea why it decided to all of the sudden but it did. Strange, but true.
I had this issue quite often with Mac OS X 10.7. But unlike Frank Halasz "systemctl status avahi-daemon.service" shown no failure. And in fact the problem was on the Mac side. Restarting Bonjour with the following commands fixed the issue.
$ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
$ sudo launchctl load -F /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

Cannot bind to IPv6 address

I am facing a strange problem on my Ubuntu Karmic system.
When I call getaddrinfo() with AI_PASSIVE and AF_UNSPEC, for an empty host and the UDP 12000 port to get a bindable address, I only get back one IPv4 result (0.0.0.0:12000 for instance).
If I change my call and specify AF_INET6 instead of AF_UNSPEC, then getaddrinfo() returns "Name or service not known".
Shouldn't I get [::]:12000 as a result ?
The same thing happens if I set the host to ::1.
When I call getaddrinfo() without AI_PASSIVE (to get a "connectable" address) for the host "localhost" and the UDP 12000 port, I first get [::1]:12000 then 127.0.0.1:12000.
So apparently, my system is IPv6 ready (I can ping to both IPv4 and IPv6 addresses, as well as DNS resolution). But how is it that I can't get an IPv6 address to bind to with getaddrinfo() ?
Do you guys have any idea about what could be wrong ?
My OS is Ubuntu Karmic, fresh install without any networking tweaking.
Thank you.
P.S: If you have no idea but still want to help me, you can get this sample program or type:
wget http://people.apache.org/~jorton/gai.c
And give me the result of:
$ ./gai -ap null 12000
My result is:
$ ./gai -ap null 12000
getaddrinfo(NULL, "12000", {.family=AF_UNSPEC, .hints=0|AI_ADDRCONFIG|AI_PASSIVE}) = 0:
family= 2, proto= 6 inet4: addr=0.0.0.0, port=12000
There you can see that I only have one IPv4 result.
This happens on new systems that use eglibc: debian-glibc.
Apparently, there is a bug that requires you to set at least one valid IPv6 address to one of your network interfaces (the loopback doesn't count).
After I did this:
$ sudo ip -6 addr add 2001:660:4701:1001::1 dev eth0
I have:
$ ./gai -ap null 12000
getaddrinfo(NULL, "12000", {.family=AF_UNSPEC, .hints=0|AI_ADDRCONFIG|AI_PASSIVE}) = 0:
family= 2, proto= 6 inet4: addr=0.0.0.0, port=12000
family=10, proto= 6 inet6: addr=::, port=12000, flowinfo=0
I hope this can help someone.

Resources