Recently I needed to build a test system at work which required a client to communicate with a server through a NAT router. I decided to assemble the system from three KVM-based virtual machines with
tap networking. The router would get two network interfaces, and two bridge interfaces on the host connect server and client to it on either side. So the plan was this:
I wanted to create all interfaces as virtio network devices inside the virtual machines and tap devices on the host, so I used
-net parameter pairs like these when starting qemu-kvm (up/down scripts for the tap device not shown):
-net nic,model=virtio,macaddr=52:54:00:12:34:56 -net tap
On client and server this worked as expected. The router, however, needed two pairs of network interfaces, and strangely packets were leaking between the bridges. This became especially obvious because of the NAT: When capturing packets, I could see the same packet twice: Once NATed and once unmodified. I spent quite a while debugging this, at first suspecting a problem with my NAT configuration, although NAT is extremely simple with the iptables
MASQUERADE target. In the end, I found the solution in a mailing list archive.
You have assumed (as I did, when I first tried this) that the first “-net nic” and “-net tap” are automatically associated with each other. They aren’t – you have to tell KVM explicitly.
— Jarrod Lowe, Re: Packet “leakage” between two bridges
This is indeed the same mistake I made. I had specified the interfaces in the order nic/tap/nic/tap and assumed they would get paired in order, but they acted like they were all connected to the same switch as shown on the right.
Luckily, this problem could be solved very easily, because this virtual “switch” is a VLAN switch. The network interface configuration for qemu-kvm takes an optional
vlan parameter, like this:
-net nic,model=virtio,macaddr=52:54:00:12:34:56,vlan=1 -net tap,vlan=1 -net nic,model=virtio,macaddr=52:54:00:12:34:57,vlan=2 -net tap,vlan=2
The result can be seen in the next figure. The red and green parts each represent one of the numbered VLANs.
With this change, the virtual networks work as expected. 🙂