Public Virtual IP, Router Nodes and q-in-q support

Posted on October 10, 2011 by rgamble

In Extility 1.5, we’ve completely rewritten our networking stack to bring you two cool new features.

Public Virtual IP

Public Virtual IP (PVIP) brings lightweight, efficient, Layer 3 connectivity to the Extility Platform. Prior to version 1.5, customers with routable IP addresses used what we call ‘Public VLAN mode’. Here, each virtual NIC with an IP address is allocated to a subnet, and each subnet lives on a VLAN. That VLAN is then routed by a router appliance. This is great in terms of security, as the VLAN provides isolation from other customers (and, indeed, other VLANs belonging to the same customer). However, for the provider operating at scale, it has a number of problems:

• Whilst small subnets can be used (e.g. /29 IPv4 subnets), usage of IP addresses is not particularly efficient;
• Each customer uses one or more VLAN, and some switches carry limits on the number of usable VLANs; and
• Each virtual NIC has its own MAC address. These take up space in the CAM (Content Addressable Memory) tables within the switch. This little known resource in the switch is actually the tightest constraint on scaling.

With PVIP mode, each customer is allocated a single IPv4 address (and a /64 IPv6 address if you are using IPv6). We have a virtual router appliance within each compute node, so the layer 2 network is not exposed to your switch fabric. That means:

• IP addressing is optimally efficient;
• No additional VLANs are used for each customer;
• There is one CAM table entry per node, rather than one (or more) per VM; and
• We retain all the advantages of Public VLAN mode in terms of security and isolation. The server is behind its own router.

The only disadvantage of PVIP mode is that two servers cannot share the same Layer 2 network. That causes issues for various redundancy protocols (such as VRRP). For that reason, we allow PVIP mode and Public VLAN mode to co-exist on the same platform.

Router Nodes and Router Groups

We didn’t want to let Public VLAN mode become the poor relation, so we’ve made a number of enhancements to how these works too. Rather than having a fixed configuration of routers managing your Public VLAN traffic, we now allow you to add as many router nodes as you want. You can use our web-based admin interface to do this at the touch of a button, just like adding a compute node. Each router node belongs to a router group, and each router group consists of a redundant set of router nodes which provide routing for a set of Public VLANs. This makes ensuring load is spread between devices child’s play.

While we were at it, we completely rewrote the way we deal with firewalls. Firewall rules are now far faster to apply.

802.11 q-in-q / 802.1ad support

Last but not least, we now support 802.1 q-in-q / 802.1ad throughout the platform. In English, that’s VLANs within VLANs. That theoretically allows for support of 16,777,216 VLANs (as opposed to 4096), so we think it is unlikely you are going to run out any time soon.