Using Virtualization to migrate between two DMZ’s

We currently have a project ongoing to migrate away from an ASA5510 to two shiny new PaloAlto firewall’s which are located at out ISP.  This part of the project was to migrate all DMZ servers which currently reside on a single ESXi host onto a new DMZ cluster with shared storage at our ISP.  To make things a little complicated, not only were we migrating all our public internet facing services from one firewall to another, but also changing from one public IP address range to another all with minimal downtime.

So, how did virtualization help with this?  Since all public facing servers were already virtualized (Aaprt from our Threat Management Gateway which will be by the end of the project) the actual migration of the VM’s from our current site up to our ISP was very easy using vMotion.  We simple stretched the VLAN up to our ISP and then migrated the servers over night via our 100mb link onto the new infrastructure.  Now this isn’t ideal as traffic was entering the ASA via another link, going across to our second site, going back to our ISP via another link and then hitting the server and then having to travel all the way back down to leave the ASA.  Since this was only a stop gap it was acceptable.  At the same time as stretching the VLAN across to our ISP we created another VLAN for the new DMZ and passed it through to both hosts.  So, both hosts up at our ISP could see the old VLAN hanging off the ASA and the new VLAN hanging of the PaloAlto firewalls.  We then verified and migrated the rule set from the ASA to the new PaloAlto devices and NAT’d the new public IP addresses to private IP addresses in the new VLAN for all the servers we were migrating.

This is where having these servers virtualized saved us potentially 24 to 48 hours downtime (Not including roll back).  We needed to find a way to test the NAT and security rules on the PaloAlto firewalls whilst minimising downtime on the VM’s as some of these servers run public facing websites.  We also needed to change the public IP address for these servers whilst maintaining connectivity.  Luckily all servers were simple web servers with static content.  We arranged downtime to clone each server and then power on the clone which was still connected to the old DMZ and who’s traffic flowed through the ASA.  Once the service was resumed via the clone we could change the network for the original server to be the new VLAN, who’s traffic flowed via the PaloAlto, re-ip it to fall in line with the new DMZ’s IP range and then test both internally and externally by changing the hosts file on our client pc’s to point the hostname to ether the public or private IP address.  This allowed us to test both the NAT and security rules whilst not interrupting any public facing services.  Once we were happy that the rule set was correct and signed off we simply contacted our ISP and requested them to change the public IP addresses for the hosts to point to the new addresses and within 24 to 48 hours users would start connecting to the original server rather than the clone.  After DNS had updated globally we could destroy the clone.  If all these servers were physical we would either have to look into migrating the services to virtual or another physical host which would involve significant work to build new servers and deploy applications or manually move them to our ISP, test firewall rules and then request the DNS changes which could have taken potentially a few days to fully complete and document.