Multiple Network Devices in KVM

Recently I needed to build a test system at work which required a client to communicate with a server through a NAT router. I decided to assemble the system from three KVM-based virtual machines with tap networking. The router would get two network interfaces, and two bridge interfaces on the host connect server and client to it on either side. So the plan was this:

Client linked to virbr1, Router linked to virbr1 and virbr0, Server linked to virbr0

I wanted to create all interfaces as virtio network devices inside the virtual machines and tap devices on the host, so I used -net parameter pairs like these when starting qemu-kvm (up/down scripts for the tap device not shown):

-net nic,model=virtio,macaddr=52:54:00:12:34:56 -net tap

On client and server this worked as expected. The router, however, needed two pairs of network interfaces, and strangely packets were leaking between the bridges. This became especially obvious because of the NAT: When capturing packets, I could see the same packet twice: Once NATed and once unmodified. I spent quite a while debugging this, at first suspecting a problem with my NAT configuration, although NAT is extremely simple with the iptables MASQUERADE target. In the end, I found the solution in a mailing list archive.

You have assumed (as I did, when I first tried this) that the first “-net nic” and “-net tap” are automatically associated with each other. They aren’t – you have to tell KVM explicitly.

— Jarrod Lowe, Re: Packet “leakage” between two bridges

Host and Router-VM, each with two network interfaces, all interconnected

This is indeed the same mistake I made. I had specified the interfaces in the order nic/tap/nic/tap and assumed they would get paired in order, but they acted like they were all connected to the same switch as shown on the right.

Luckily, this problem could be solved very easily, because this virtual “switch” is a VLAN switch. The network interface configuration for qemu-kvm takes an optional vlan parameter, like this:

-net nic,model=virtio,macaddr=52:54:00:12:34:56,vlan=1 -net tap,vlan=1 -net nic,model=virtio,macaddr=52:54:00:12:34:57,vlan=2 -net tap,vlan=2

The result can be seen in the next figure. The red and green parts each represent one of the numbered VLANs.

Host and Router-VM with two interfaces each. All are connected, but each interface on the host is connected to one of the VM interfaces with a VLAN link,

With this change, the virtual networks work as expected. 🙂

4 thoughts on “Multiple Network Devices in KVM

  1. In the name of everything that is good, this may be the most complicated thing I’ve seen this month.

    1. Despite finals? 😯
      Seriously though, I think few people work directly with KVM, and even fewer need multiple network interfaces on one instance.

  2. holy sh… !
    i’ve been teraing my hair for hours because of this problem….
    i was testing the same exact configuration as yours, 2 vm, with two interfaces each, and 2 bridges to link them. (to test fun stuff, like STP, or keepalived/heartbeat…)
    And like you i saw with tcpdump that packets were leaking between the bridge, and a packet that was coming from the 2nd interface went through the first bridge… assuming nic/tap were paired…
    Finally, the worst part was that the load avg of my 2 VM was going up rapidly because the 2 bridges were seing the same packets with the same mac, thus causing a network loop. this drove me crazy !
    Anyway, thanks A LOT for thir information 🙂
    now i set the vlan on my interfaces and i dont get this issue anymore \o/

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: