NSX 6.2 Communication Channel Health

A great feature released as part of NSX 6.2 is the Communication Channel Health which can be used to troubleshoot communications between NSX Managers and Controllers to the individual hosts or clusters.  The following diagram taken from a VMware presentation shows the communication channels involved:

The VSFWD is the Firewall Agent and NETCPA is the Network Control Plane agent that both run on all hosts / clusters that have been prepared for NSX.  To check the Communication Channel Health log into the vSphere Web Client and navigate to the Networking & Security plug in -> Installation and then the Host Preparation.  Click on a cluster or individual host and selection Actions and then Communications Channel Health:

The following shows the health of the PA-MGMT cluster:

The following shows the health of an individual host:

When your hosts looses the ability to communicate with an NSX Controller then you will see the Control Plane Agent to Controller being down (To simulate this I just powered off the only controller in my lab):

If there are issues with the NETCPA agent then you will see the following (To simulate this I just stopped the NETCAP service on a single host):

Because the host uses the NETCPA agent to communicate with the controllers you will see an unknown error as well.  Finally if there are issues with the firewall agent you will see the following (To simulate this I just stopped the vShield-Stateful-Firewall service on a single host):

You will also notice that NSX Manager to Control Plane Agent is down and Control Plane Agent to Controller is unknown and this is because the connection status from each host to controller is locally monitored and the health status is then reported by each host to the NSX Manager over the messaging bus which is dependent on the VSFWD service.

Creating a Universal Logical Switch in NSX 6.2 and vMotioning a VM from one vCenter to another

Now that we have our Universal Transport Zone that spans clusters across multiple vCenter servers it's time to create a Universal Logical Switch, attach two virtual machines to it and then vMotion one from the Palo Alto vCenter to the New York vCenter.  One thing to remember when creating any universal objects is that they need to be created on the NSX Manager that has the primary role.  If you try to do it whilst connected to an NSX Manager with the secondary role you will not see any Universal Transport Zones.  Log into the vSphere Web Client and navigate to Logical Switches within the Network & Security tab and create a new Logical Switch whilst ensuring you are connected to the NSX Manager with the primary role:

Once the switch has been created you will notice that it's VNI number falls inline with the range we specified within the Universal Segment ID Pool:

If you switch across to the NSX Manager with the secondary role you will also see the same Universal Logical Switch created with the same VNI number:

If we check the VDS portgroups you should see the newly created portgroup in both vCenters:

I have the following two VM's which I am going to attach to the Universal-Web-Tier-01 Logical Switch and power on ensuring that are both in the same vCenter:

WEB01 - 192.168.1.11

WEB02 - 192.168.1.12

As you can see we have layer 2 connectivity between the two virtual machines:

Now time to vMotion WEB02 from the Palo Alto vCenter (PA-VCSA-01) to the New York vCenter (NY-VCSA-01).  Right click on the VM and select Migrate:

Select the option to Change Compute Resources only and click Next (The same storage is available to both vCenters / hosts in my lab):

Click on the clusters tab and select the cluster you want to migrate the VM to and click Next.  In my example I am moving WEB02 from the PA-Compute cluster in PA-VCSA-01 to the NY-Compute cluster in NY-VCSA-01:

Select the folder to place the VM into and click Next:

Select the network you want to attach the VM to in the other vCenter.  In my example I'm connecting it to the same Universal Logical Network.  It would be nice for this to have been automatically selected for me:

Set the appropriate priority and click Next:

Verify your setting, click Next and watch the magic happen:

WEB02 has now been vMotioned from the Palo Alto vCenter across to the New York vCenters with no drop in connectivity:

The possibilities for this are endless especially around Disaster Recovery with Site Recovery Manager.

Creating a Universal Transport Zone in NSX 6.2

One of the new features in NSX 6.2 is the ability to span logical networks across multiple vCenters. In order to do this we now have the ability to create a Universal Transport Zone than can contain clusters from different vCenters. In my lab I'm going to create three new Transport Zones:

PA-TransportZone - A dedicated transport zone for the Palo Alto DataCenter that will contain the PA-Compute and PA-MGMT clusters.

NY-TransportZone - A dedicated transport zone from the New York DataCenter that will contain the NY-Compute and NY-MGMT clusters.

Universal-TransportZone - A dedicated transport zone that will contain all four clusters that span over two vCenters.

In order to achieve this we first need to create local and universal segment ID pools.  Segment ID pools are used to allocate VNI's (VXLAN Network Identifiers) to logical networks. I'm going to use the following VNI pools:

Palo Alto Segment ID Pool - 5000-5999
New York Segment ID Pool - 6000-6999
Universal Segment ID Pool - 10000-10999

This will allow me to create 1000 logical networks in Palo Alto and New York but then also another 1000 than can span both vCenters.

To start log into the vSphere Web Client and navigate to Installation -> Logical Network Preparation and then click on the Segment ID tab.  Ensure that you are connected to the NSX Manager that has the Primary Role:

Edit the setting and add the segment ID pools for the primary NSX Manager:

Click OK and then change to the secondary NSX Manager.  You should already notice that the Universal Segment ID's have already been populated.  Add the segment ID's for the secondary ensuring that they don't overlap with the Universal segment ID's:

Click on the Transport Zone tab and create a new Transport Zone.  The first Transport Zone we are going to create is the PA-TransportZone and add the two Palo Alto DataCenter clusters:

Now create a new Transport Zone but ensure you tick the option to Mark this object for Universal Synchronization and add the two Palo Alto DataCenter Clusters again:

We now have two Transport Zones:

Switch to the secondary NSX Manager and you will notice that we already have the Universal Transport Zone available.  Add the two New York DataCenter clusters to it:

Finally create the second local transport zone and add the two New York DataCenter clusters to it:

We should end up with a Local and Universal Transport Zone in both the Palo Alto and New York sites:

Installing NSX 6.2 within a multi vCenter Environment

With the release of NSX 6.2 we now have the ability to span Transport Zones, Logical Networks, Distributed Logical Routers (DLR's) and even firewall rules across multiple vCenters.  In this post I'm going to show you how to install NSX 6.2 in a multi vCenter environment.  In my lab at home I have two physical ESXi hosts each running four nested ESXi hosts to simulate a Palo Alto and New York Datacenter.  All hosts are running vSphere 6.0 and there are two vCenters in linked mode with the Platform Services Controller running on the Palo Alto vCenter. The screen shot below shows the configuration via the vSphere Web Client:

I'm not going to show you how to deploy NSX Manager as this has been very well documented by numerous bloggers. I'd recommend Giuliano Bertello's (Blog | Twitter) NSX for newbies series.  So I've deployed two NSX managers called PA-NSX-01 and NY-NSX-01 and associated them with the corresponding vCenters:

PA-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and also it's vCenter server:

NY-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and NY-VCSA-01 as it's vCenter server:

Remember when using vCenter 6 the lookup service port is now 443 instead of 7444.

After logging into the Web Client you will now see that you can manage multiple NSX Managers:

I've deployed a single controller in the Palo Alto management cluster as this is just a test lab:

I'm also going to prepare all 4 clusters and configure the host VTEP's:

Palo Alto Management

Palo Alto Compute

New York Management

New York Compute

You will notice that you now have the ability to select the NSX Manager that you want to perform tasks on via the NSX Manager menu:

I've now prepared all four clusters:

I'm now going to configure PA-NSX-01 to be the primary NSX Manager and NY-NSX-01 to be a secondary.  Navigate back to Installation and then the Management tab and highlight the NSX Manager that you want to assign the primary role to, click Action and then Assign Primary Role

Click Yes to confirm the operation:

Keep the primary NSX manager highlighted and click Actions and then click Add Secondary NSX Manager:

Ensure the IP address is correct for the secondary NSX Manager and enter the Admin credentials for the secondary NSX Manager:

Once finished we should have Primary and Secondary NSX Manager instances and you should see the secondary instance using the NSX Controller from the primary site:

Within NSX 6.2 there are four NSX Manager Roles:

Standalone - Default option for new installation or upgrade. All objects in a Standalone NSX Manager are local.

Primary - One NSX Manager per Cross VC NSX deployment is assigned the Primary role. Deploys and manages the Universal Controller Cluster and any Universal objects.

Secondary - The Secondary role is assigned when adding a Standalone NSX Manager to a Primary. Universal objects from the primary are synchronised to the Secondary where they are read-only.

Transit - Temporary role used when an NSX Manager has a Primary or Secondary Role removed. If universal objects still exist the NSX Manager is assigned a Transit role until these objects are deleted then it can move to a Standalone role.