NSX-T provides true multi-tenancy capabilities to a SDDC/Cloud infrastructure and there are various ways of achieving it based on the use cases. In the simplest deployment architecture, multi-tenancy is achieved via connecting various Tier1 gateways to a Tier-0 gateway and each T1 gateway can belong to a dedicated tenant. Another way of implementing this is to have multiple T0 gateways, where each tenant will have their dedicated T0.

Things have changed with NSX-T 3.0. One of the newest feature that was introduced in NSX-T 3.0 was VRF (virtual routing and forwarding) gateway aka VRF Lite.

VRF Lite allows us to virtualize the routing table on a T0 and provide tenant separation from a routing perspective. With VRF Lite we are able to configure per tenant data plane isolation all the way up to the physical network without creating Tier0 gateway per tenant.

VRF Architecture

At a high level VRF architecture can be described as below:

We have a parent Tier0 gateway to which multiple VRF connects. Each VRF can be imagined as one tenant. The parent T0 houses the virtual routing and forwarding tables for the various tenants connecting to same T0 via VRF’s. In this architecture tenants can run overlapping IP addresses without having to configure NAT.

Below diagram is sample representation of a VRF Lite implementation.

In this post I will show how we can implement and test this new feature.

Current NSX-T State: Current state of NSX-T in my lab looks like as shown in below diagram. 

I have one T0 gateway which is BGP peering with upstream routers. The two uplinks which T0 is using to connect to routers are in VLAN 600 & 700 respectively. 

Future NSX-T State: Once I am done with VRF Lite configuration, my architecture will look like as shown below.

Below are some key configuration points for this implementation:

• 2 VRF’s will be carved out of Parent T0 gateway, one for Tenant A and another for Tenant B.
• Both tenants will connect to Parent T0 via dedicated pair of VLAN backed uplinks.
• Tenant A is using VLAN 10 & 11 and Tenant B is using VLAN 20 & 21.

Note: I have intentionally dropped T0 uplinks to upstream routers in below picture to keep the architecture diagram simple. 

VRF Implementation Steps

Below are high-level steps for implementing VRF:  

Step 1: Create a Parent Tier-0 Gateway
Step 2: Create VLAN backed uplink segments for the VRF’s.
Step 3: Create VRF enabled Tier-0 Gateways.
Step 4: Configure uplink interfaces on the VRF’s for northbound connectivity.
Step 5: Configure Physical Routers with the VRF gateway addresses and BGP info.
Step 6: Configure BGP on the VRF’s.
Step 7: Create Tier-1 Gateways and attach these to the VRF’s.
Step 8: Create segments and attach it to Tier-1 gateways.

Let’s dive into lab and implement above steps.

Step 1: Parent Tier-0 gateway Deployment

I already have T0 deployed in my lab so I am skipping this step.

Step 2: Create Uplink Segments for VRF

As mentioned earlier, each VRF will connect to upstream router via 2 LAN backed uplinks. These uplinks are created as trunk uplinks and is configured to allow a subset of VLAN’s. 

Note: Teaming policy is optional but if you are using them in your infrastructure, specify them at segment level as it make sure that traffic from each segment is steered towards specific Edge uplinks. This is done to establish a deterministic routing path.

Note: Make sure you create segments in Edge VLAN TZ and don’t connect segments to any gateways.

In total I have created 4 uplinks for the 2 VRF’s.

Step 3: Create VRF’s

To create a new VRF, navigate to Networking > Tier-0 Gateways > Add Gateway > VRF

Specify name of VRF and associate it with parent Tier-0 gateway.

Here are my 2 VRF’s created for Tenant A & B.

Step 4: Create VRF External Interfaces

For northbound connectivity, we need to create external interfaces for the VRF. In total we need to create 4 interfaces per VRF. Each edge node will host 2 interfaces.

Note: Since we have 2 VRF’s here, we will have 8 external interfaces. 

To create interfaces for VRF, edit the VRF and click on Set under Interfaces

Make sure to define Access VLAN ID for the external interface. This VLAN ID is used for BGP peering with physical router and thus same VLAN should exist on physical router.

Below screenshot shows all 4 external interfaces for TenantB VRF

Step 5: Configure Physical Router

I am using vyos in my lab as northbound router and below is the vyos configuration specific to the VRF’s which I created earlier

Show More

Step 6: Configure BGP on the VRF’s

BGP on VRF’s are configured in the same was as we configure on T0 gateway. VRF’s inherit Parent T0 BGP local AS number and some other BGP settings, but BGP neighbor configuration is unique for each VRF.

Important: Due to a bug in In NSX-T 3.0.1 and 3.0.1.1, you may get following error message when you try to enable BGP on a VRF Gateway via UI:

“Only ECMP, enabled, BGP Aggregate to be configured for VRF BGP Config”

The workaround of this problem is API. 

Before enabling BGP on VRF, lets first configure BGP neighbors on VRF.

Edit VRF settings and expand BGP section and click on Set. 

Make sure to enable BFD and Allowas-in BGP neighbor configuration.

Note: Leave source address blank when specifying BGP neighbors for VRF.

To enable BGP on VRF, you can follow the steps mentioned in my earlier blog post

Post enabling BGP, verify status of BGP neighbors report as success. 

My upstream router is also happy and exchanging BGP packets with VRF neighbors

Create Tier-1 gateways and connect it with VRF gateways

Nothing fancy here. T1 creation is straight forward. To create T1 gateway, navigate to Networking > Tier-1 Gateways > Add Tier-1 Gateway

Provide a name for the T1 gateway and attach it to VRF that belongs to Tenant A.

Under Route Advertisement, select the routes that will be advertised to connected VRF gateways and then click on save.

Repeat the process for Tenant-B T1 gateway.

Here are the two Tier-1 gateways that I have created for Tenant A & B.

Enable Route Re-Distribution on VRF

Next is to enable Route Re-Distribution on VRF gateway so that physical network receives routes of the specific tenants. 

Edit settings of VRF and expand Route Re-Distribution and click on set. 

Provide a name for the Re-Distribution rule and click on Set. 

Select Tier-1 subnets that will be advertised to physical network and click on Apply.

Repeat the process for Tenant-B T1 gateway. 

Create Segments for Workloads

Create logical segments for Tenant A & B so that tenants can attach their workloads on the created segments. 

I have created one segment each for both the tenants. 

And that’s it for this post. In next post I will show traffic flow for VRF gateways.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing 🙂