Configuring and testing VXLAN within a nested environment

In this post I'm going to show you how I managed to get VXLAN working (Which is required for NSX) within my test environment that I'm currently using to learn more about NSX.  My homelab currently consists of:

3 x HP ML310e G8
1 x RouterBOARD RB751
1 x Linksys SRW2016 16 Gigabit Switch
1 x Synology DS412+

Within the Production cluster I run all my home production services but I also run a Nested ESXi lab that I previously used for vCNS and now NSX as per the screenshot below:

My Test cluster contains three hosts and the vCenter Server Appliance.  I installed NSX manager outside of this cluster as I didn't want it to take up the nested lab resources.  My Test cluster looks like this:

I've deployed a single NSX controller in this cluster as I'm not too bothered about redundancy and would rather save the resources for VM's.  So, now onto the blog post.  In order for VXLAN to work you need a dedicated VLAN and the MTU needs to be set to 1600 end to end.  First step was to enable jumbo frames on my Linksys SRW2016 switch.  This simply involved ticking a box and rebooting the switch.  This was a pain as I had to power down all VM's and hosts but I also took the opportunity to upgrade my Synology DS412+ to the latest firmware.  To enable jumbo frames on the Linksys switch you simply browse to the admin interface, select Admin and then tick the option next to Jumbo Frames and then reboot the switch:

Next step was to create a new VLAN dedicated for VXLAN traffic and ensure the MTU was set to 1600.  I use a RouterBOARD RB751 as my router / Layer3 switch which I highly recommend.  This was simply a case of creating a new VLAN, I used 15, setting the MTU to 1600 and assigning it to the bridge-local interface:

It was then a case of creating the same VLAN within the Linksys switch and passing this VLAN down the trunks to the three ESXi hosts.  Once the VLAN was presented to the three ESXi hosts I then needed to set the MTU on both the Production and Test cluster Distributed Virtual Switchs as by default they are set to 1500.  To do this simply go to Networking, right click on the DVS and select Edit Settings.  From there, select Advanced and then change the MTU to 1600

It's then a case of configuring VXLAN for the cluster using VLAN 15, ensuring the MTU is set to 1600 and configuring an IP pool for the VMK nics:

In order to test network connectivity from one host to another you can use the following command after SSH'ing into the host:

ping ++netstack=vxlan -d -s 1600 -I vmk5 <IP ADDRESS>

In my case when pining another host with the MTU set to 1600 it failed:

After reading this article by Kamau Wanguhu (Blog | Twitter) I found the the actual VXLAN packet size is 1572.  I then pinged another host with the new MTU size and it worked fine:

So, hopefully that's VXLAN all configured in my NSX Test cluster.  Big thanks to Dan Watson (Twitter) and Geordy Korte (Blog | Twitter) for their help.  I'm sure I'll be harassing you more and more  in the coming months.