An OpenStack Crime Story solved by tcpdump, sysdig, and iostat – Episode 3

Previously on OpenStack Crime Investigation … Two load balancers running as virtual machine in our OpenStack based cloud, sharing a keepalived based highly available IP address started to flap, switching the  IP address back and forth. After ruling out a misconfiguration of keepalived and issues in the virtual network, I finally got the hint that the problem might originate not in the virtual, but in the bare metal world of our cloud. Maybe high IO was causing the gap between the keep alive VRRP packets.

When I arrived at baremetal host node01, hosting virtual host loadbalancer01, I was anxious to see the IO statistics. The machine must be under heavy IO load when the virtual machine’s messages are waiting for up to five seconds.

I switched on my iostat flash light and saw this:

Nothing? Nothing at all? No IO on the disks? Maybe my bigger flash light iotop could help:

Unfortunately, what I saw was too ghastly to show here and therefore I decided to omit the screenshots of iotop 1 . It was pure horror. Six qemu processes eating the physical CPUs alive in IO.

So, no disk IO, but super high IO caused by qemu. It must be network IO then. But all performance counters show almost no network activity. What if this IO wasn’t real, but virtual? It could be the virtual network driver! It had to  be the virtual network driver.

I checked the OpenStack configuration. It was set to use the para-virtualized network driver vhost_net.

I checked the running qemu processes. They were also configured to use the para-virtualized network driver.

I was getting closer! I checked the kernel modules. Kernel module vhost_net was loaded and active.

I checked the qemu-kvm configuration and froze.

vhost_net was disabled by default for qemu-kvm. We were running all packets through userspace and qemu instead of passing them to the kernel directly as vhost_net does! That’s where the lag was coming from!

I acted immediately to rescue the victims. I made the huge, extremely complicated, full 1 byte change on all our compute nodes by modifying a VHOST_NET_ENABLED=0 to a VHOST_NET_ENABLED=1, restarted all virtual machines and finally, after days of constantly screaming in pain, the flapping between both load balancers stopped.

I did it! I saved them!

But I couldn’t stop here. I wanted to find out, who did that to the poor little load balancers. Who’s behind this conspiracy of crippled network latency?

I knew there was only one way to finally catch the guy. I set a trap. I installed a fresh, clean, virgin Ubuntu 14.04 in a virtual machine and then, well, then I waited — for apt-get install qemu-kvm to finish:

And then, I let the trap snap:

I could not believe it! It was Ubuntu’s own default setting. Ubuntu, the very foundation of our cloud decided that despite all modern hardware supporting vhost_net to turn it off by default. Ubuntu was convicted and I could finally rest.

This is the end of my detective story. I found and arrested the criminal Ubuntu default setting and were able to prevent him from further crippling our virtual network latencies.

Please feel free to leave comments and ask questions about details of my journey. I’am already negotiating to sell the movie rights. But maybe there will be another season of OpenStack Crime Investigation in the future. So stay tuned on codecentric Blog.

Footnotes

1. Eh, and because I lost them.↵