To overcome cloud networking scalability challenges as outlined in a previous blog post, cloud service providers look to solutions like Layer 2, VLANs and Layer 3. Discussed below are the issues service providers face with Layer 2.
Layer 2 issues
One of the first challenges in scaling the cloud provider’s network comes at layer 2.
To explain this, it is first necessary to recap how a switched Ethernet network works. Each network interface card (physical or virtual) participating in a layer 2 network has a unique MAC address. Each packet contains a header (the MAC header) containing the source and destination MAC address of the sending and receiving network interface. The switches on the layer 2 network learn the source addresses of any packet passing through them. When a switch comes to forward a packet to a given destination, it looks up that destination in the forwarding table that it has learnt, and if an entry is present, the packet is forwarded to that port. If an entry is not present (or if the packet is a broadcast packet), the packet is forwarded to all ports. Such forwarding to all ports is to be avoided as they use bandwidth on each port as well as switch capacity. It is thus vital the forwarding table can contain entries for all source MAC addresses in use.
Layer 2 forwarding
- NIC B sends a packet addressed to NIC C
- Switch 2 has no CAM table entry for the destination, so floods the packet through every interface
- Switch 2 learns the MAC address of NIC B and records it in its CAM table
- Switches 1 and 3 do similarly
Layer 2 forwarding
- NIC C sends a reply to to NIC B
- Switch 2 and 3 have already learnt CAM table entries for NIC B, so only forward the packet out of the appropriate interface
- Switches 2 and 3 learn the MAC address of NIC C and record it in their CAM tables
Layer 2 forwarding
- NIC B sends a second packet addressed to NIC C
- Switch 2 now has a CAM table entry for NIC C, so sends it out the appropiate interface
- Switch 3 does similarly
- Flooding is avoided
To achieve satisfactory performance, switches need to start forwarding packets almost as soon as the packet header is received. At 10Gb/s, the Ethernet header is received in approximately 10 nanoseconds. This does not give sufficient time for the addresses to be looked up in conventional memory. Thus switches use content addressable memory (CAM), a type of memory which rather than being addressed linearly is addressed by the content (in this case the destination MAC address); hence the layer 2 forwarding table is often referred to as the CAM table. This memory is very expensive compared to conventional memory, and hence networking equipment has a limited amount of it. A typical switch might support between a few thousand and few tens of thousand CAM table entries, though some backbone switches might have more. The size of the CAM table produces an upper bound on the number of MAC addresses on the network, and hence the number of participating network interface cards. Note that as the CAM uses a hashing algorithm, the maximum number of NICs may be significantly lower than the size of the CAM table.
Consider a cloud provider’s network contains N virtual machines, where each virtual machine has a single (virtual) network interface card. In a naive implementation, where each vNIC forms part of the same layer 2 network, the number of CAM table entries required cannot exceed N . Or to put it another way, the number of virtual machines is limited to the number of entries of the smallest CAM table of any switching device used. Moreover, the number of switches will be proportional to N (as each switch will support a fixed maximum number of virtual machines), and the CAM table required in each switch will also be proportional to N . Thus the cost of CAM table is proportionate to the square of the number of virtual machines. As may be readily understood, this presents a serious scaling challenge.
The above argument is admittedly a simplification of the problem. Some larger and more expensive modern switches use different techniques to achieve the same results and can cope with larger numbers of MAC addresses. However, there are other pressing reasons to limit the number of MAC addresses in a layer two network (and hence the number of MAC addresses within the CAM table). IP broadcast and (probably) IP multicast packets translate to layer 2 broadcasts, which together with layer 2 broadcasts such as ARP are flooded throughout the network. The number of such flooded packets increases according to the square of the participants. Further, the larger the layer 2 network, and the less trusted the participants, the greater the likelihood of layer 2 problems occurring. These include MAC spoofing, layer 2 broadcast storms, and attacks on the switching layer (for instance VMs speaking spanning-tree protocol and confusing the switches). For this reason, it makes sense to limit the number of MAC addresses within each CAM table whether the CAM table size itself is an upper bound or not.
To learn more about VLANs and Layer 3 in detail, download our guide to network scalability.