Routing to a connected VPC when attached to VMware Transit Connect

Recently there was an internal discussion around a customer request to access an Amazon FSx for Windows File Servers that was currently running in a connected VPC from another SDDC. The topology that the customer was looking at was as follows:

The customer had two SDDC's with one of the SDDC's (SDDC 01) being connected to a VPC that was running Amazon FSx for Windows File Servers. They also wanted the second SDDC (SDDC 02) to be able to access the Amazon FSx For Windows File Servers as well as workloads running in SDDC 01. The main question was what route the traffic would take from SDDC 01 to the Amazon FSx for Windows File Servers. 

During the SDDC deployment, a static route will be added into SDDC 01 to send any traffic destined to Connected VPC 01 main CIDR across the ENI. In the example above the route, 172.16.0.0/16 would be added as a static route into the T0, and for every segment that is created, we would also update the Connected VPC 01 main route table to send the return traffic back across the ENI to avoid asymmetric routing. This takes care of traffic from SDDC 01 to Connected VPC 01.

When we add a VMware Transit Connect into the mix and add both SDDC 01 and SDDC 02 to the SDDC group this handles our requirement of being able to route between SDDC 01 and SDDC 02. If we added Connected VPC 01 as an external VPC to the SDDC Group then a static route would be added to the T0 on SDDC 02 to send all traffic destined to 172.16.0.0/16 across the vTGW (You would manually have to add a static route in the main route table of Connected VPC 01 for the return traffic to route across the TGW attachment. Since we already have a static route on the T0 in SDDC 01 the route is not added as the /16 prefix already exists.

Traffic flow looks like this: 

Things to watch out for:

• If Connected VPC 02 has the same CIDR range as Connected VPC 01 then traffic would prefer to go across the ENI rather than the vTGW which means it would NOT be able to access the Amazon FSx for Windows Files Servers in Connected VPC 01. A way around this is to add a more specific route i.e. /32 address to the Amazon FSx for Windows File Server but this would also inject this route into SDDC 01 so ALL traffic from SDDC 01 and 02 would route across the vTGW and you would also need more specific routes to send traffic back across the vTGW for both SDDC 01 and SDDC 02.
• If you add a static route in the Connected VPC 01 main route table for a more specific route to go via the TGW attachment then you would end up with Asymmetric routing which will cause the traffic to be blocked. By default, VMware Cloud on AWS will always add a route into the Connected VPC main route table to send the return traffic to an NSX segment back across the ENI. If the segment is a /24 and you add a static route to a /32 to the vTGW then traffic would route across the vTGW and you would end up with Asymmetric routing.