Real World Fault Tolerance Use Cases

I've always been a big fan of VMware's Fault Tolerance feature but could never really find a valid use case to actually implement it.  Fault Tolerance (FT) has some very strict requirements / limitations:

• VM is limited to one vCPU
• VM disks need to be thick-eager zeroed
• VM must be running a supported OS
• Snaphosts are not supported on the VM (Think about how you can backup the VM)
• VM Hot add memory or CPU cannot be utilised
• DRS cannot be utilised
• Physical processors needs to support FT

It is also recommended that you have a dedicated redundant FT network with minimum 1GB pNics for the FT traffic.  In our current infrastructure we are limited to the number of physical nics we can present to the ESXi hosts due to the blade chassis we are currently using.  We were unable to allocate 2 nics to dedicate to FT so we never really utilized FT.

We recently had a requirement to move our DMZ to our ISP and thus reused some decommissioned Dell R610 hosts.  We bumped up the memory and pNic count and now we could comfortably allocate 2 pNics per host for FT.  We use a Citrix Access Gateway (CAG) VPX appliance for remote access to both published apps and desktops and although this is protected with VMware HA, FT offered more protection in the event of a complete host outage.  The only issue that I had was that the CAG was not on the supported list of OS's but it seems to work fine.

We configured the hosts with the required FT network and enabled FT on the VM when it was powered off and 15 minutes later the VM was fully protected with FT

vSphere 5.1 Web Client US date locale

After recently upgrading my lab from 5.0 to 5.1 I was keen to start playing around with the new Web Interface Client since the thick GUI client might not be around for too much longer.  I'm not using the vCenter appliance so vCenter is installed on a Windows 2008 R2 server.  After stumbling through I noticed that when viewing tasks and events date format was showing US format of MM/DD/YYYY rather than the UK of DD/MM/YYYY:

Thinking this was simply that my windows locale was set to US I double checked but as expected it was showing correctly as United Kingdom.  This was for both the account that I was logged on as and the account the vCenter service was started with.  When checking the same events in the thick GUI client the format is showing correctly:

After a twitter conversation with Graham French the issue could be with a tomcat config file that could potentially be set to a US locale somewhere.  I've trawled both Google and the VMware KB site for anything referencing this and have so far came up with nothing.  I've checked as many config files as I can but just can't seem to find anything relating to this.  If anyone can shed any light on this I'd be much appreciated.

Update 18/01/2013

I spent countless hours looking through log files to try and find the default locale and then I suddenly remembered a KB article I read a while back about changing the language in the web client.  It then suddenly dawned on me that I should try the following:

https://<vCenterServer>:9443/vsphere-client/?locale=en_GB

hey presto:

North East VMUG – 07/02/2013

The next North East England VMUG will be held on Thursday 7th February from 12:30 onwards at:

Simpson Room, The CastleGate
Melbourne Street
Newcastle Upon Tyne
NE1 2JQ

Direction here

The agenda is as follows:

12:30 - Networking and Lunch
13:15 - VMUG Introduction
13:30 - Deep Dive and Demonstrations of VMware End User Computing Technologies Including Horizon, Octopus, HTML Remoting and Mirage - Peter Von Oven, VMware’s Lead UK EUC Specialist
14:45 - Refreshment and Networking Break
15:00 - Sponsor Presentation – Fusion-iO
15:45 - Refreshment and Networking Break
16:00 - Community Presentation – Upgrading from vSphere 4.1 to 5.1, Real World Experiences from the Field Graham French
16:45 - Raffle, Socrative, Other News
17:00 - Onwards – vBeers Sponsored by 10-Zig (Bridge Hotel)

Click here to register for the event

Big thanks to our sponsors Fusion-IO and 10Zig

         

Using Virtualization to migrate between two DMZ’s

We currently have a project ongoing to migrate away from an ASA5510 to two shiny new PaloAlto firewall’s which are located at out ISP.  This part of the project was to migrate all DMZ servers which currently reside on a single ESXi host onto a new DMZ cluster with shared storage at our ISP.  To make things a little complicated, not only were we migrating all our public internet facing services from one firewall to another, but also changing from one public IP address range to another all with minimal downtime.

So, how did virtualization help with this?  Since all public facing servers were already virtualized (Aaprt from our Threat Management Gateway which will be by the end of the project) the actual migration of the VM’s from our current site up to our ISP was very easy using vMotion.  We simple stretched the VLAN up to our ISP and then migrated the servers over night via our 100mb link onto the new infrastructure.  Now this isn’t ideal as traffic was entering the ASA via another link, going across to our second site, going back to our ISP via another link and then hitting the server and then having to travel all the way back down to leave the ASA.  Since this was only a stop gap it was acceptable.  At the same time as stretching the VLAN across to our ISP we created another VLAN for the new DMZ and passed it through to both hosts.  So, both hosts up at our ISP could see the old VLAN hanging off the ASA and the new VLAN hanging of the PaloAlto firewalls.  We then verified and migrated the rule set from the ASA to the new PaloAlto devices and NAT’d the new public IP addresses to private IP addresses in the new VLAN for all the servers we were migrating.

This is where having these servers virtualized saved us potentially 24 to 48 hours downtime (Not including roll back).  We needed to find a way to test the NAT and security rules on the PaloAlto firewalls whilst minimising downtime on the VM’s as some of these servers run public facing websites.  We also needed to change the public IP address for these servers whilst maintaining connectivity.  Luckily all servers were simple web servers with static content.  We arranged downtime to clone each server and then power on the clone which was still connected to the old DMZ and who’s traffic flowed through the ASA.  Once the service was resumed via the clone we could change the network for the original server to be the new VLAN, who’s traffic flowed via the PaloAlto, re-ip it to fall in line with the new DMZ’s IP range and then test both internally and externally by changing the hosts file on our client pc’s to point the hostname to ether the public or private IP address.  This allowed us to test both the NAT and security rules whilst not interrupting any public facing services.  Once we were happy that the rule set was correct and signed off we simply contacted our ISP and requested them to change the public IP addresses for the hosts to point to the new addresses and within 24 to 48 hours users would start connecting to the original server rather than the clone.  After DNS had updated globally we could destroy the clone.  If all these servers were physical we would either have to look into migrating the services to virtual or another physical host which would involve significant work to build new servers and deploy applications or manually move them to our ISP, test firewall rules and then request the DNS changes which could have taken potentially a few days to fully complete and document.