Kubernetes. Unable connect to any pod from master - openstack

Question

I'm trying to setup Kubernetes in Openstack + CoreOS.
I have master 10.240.63.84 and 2 minions .63 and .83. I also created 3 redis pods:
redis-gopher-gziey 10.244.32.2 10.240.63.66/10.240.63.66
redis-managed-oh43e 10.244.32.3 10.240.63.66/10.240.63.66
redis-primary-fplln 10.244.54.2 10.240.63.83/10.240.63.83
master's routing table looks like:
10.240.63.0 * 255.255.255.0 U 0 0 0 eth0
10.240.63.1 * 255.255.255.255 UH 1024 0 0 eth0
10.244.0.0 * 255.255.0.0 U 0 0 0 flannel.1
10.244.50.0 * 255.255.255.0 U 0 0 0 docker0
and output of ifconfig -a is :
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.244.50.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::542f:6fff:fe4a:adf3 prefixlen 64 scopeid 0x20<link>
ether 56:84:7a:fe:97:99 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 90 (90.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.240.63.84 netmask 255.255.255.0 broadcast 10.240.63.255
inet6 fe80::f816:3eff:fe89:e9a0 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:89:e9:a0 txqueuelen 1000 (Ethernet)
RX packets 430706 bytes 559764129 (533.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 238519 bytes 116083693 (110.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.50.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::601f:62ff:feed:1556 prefixlen 64 scopeid 0x20<link>
ether 62:1f:62:ed:15:56 txqueuelen 0 (Ethernet)
RX packets 20 bytes 1504 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79 bytes 7686 (7.5 KiB)
TX errors 0 dropped 19 overruns 0 carrier 0 collisions 0
Flanneld config used for initialization is:
Master:
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
ExecStart=
ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
/usr/bin/docker run --net=host --privileged=true --rm \
--volume=/run/flannel:/run/flannel \
--env=NOTIFY_SOCKET=/run/flannel/sd.sock \
--env-file=/run/flannel/options.env \
--volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld --ip-masq=true --iface=eth0
Minion:
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
ExecStart=
ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
/usr/bin/docker run --net=host --privileged=true --rm \
--volume=/run/flannel:/run/flannel \
--env=NOTIFY_SOCKET=/run/flannel/sd.sock \
--env-file=/run/flannel/options.env \
--volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld -etcd-endpoints http://10.240.63.84:4001 --ip-masq=true --iface=eth0
So the issue is that i can't ping any of the pods from master, as well as connect to any port, error is:
ncat -v -t 10.244.32.2 6379
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: No route to host.

Answer 1

This sort of thing is hard to debug remotely. Things I would check:
1) on the sender: iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c > /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;
This will give you some insight into what the kernel is doing.
2) on the sender: tcpdump -i any host 10.244.32.2 & ncat -v -t 10.244.32.2 6379;`
This will give a bit more insight.
3) on the receiver: iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c > /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;
This will tell you if the packet came through the encapsulation.
You need to basically prove the plumbing through the whole connection.