Forming a vSphere upgrade strategy: Upgrading vCenter Server and ESX, ESXi

Solution provider’s takeaway: Knowing the best way to handle a vSphere upgrade on all levels, vCenter Server, ESX and ESXi hosts and virtual machines, will make life easier for you and your customers. Learn about the upgrade tools available and get step-by-step instruction on using them, including the pre-check tool, licensing updates, Oracle database updates, the host update utility and more.

Phase 1: Upgrading vCenter Server
Before you upgrade vCenter Server, you need to make sure you have a good backup of the SQL database and that the SQL database you are using is supported by vCenter Server 4.0. Also make sure you have downloaded the vCenter Server installer and generated your license keys. Upgrading your vCenter Server does not require any host or VM downtime.

Backing up key files
In addition to a database backup, you should also back up the following key files in case you need to roll back to the previous version.

• Back up the SSL certificate folder under %ALLUSERSPROFILE%Application Data\VMware\VMware VirtualCenter or %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\.
• Back up the vpxd.cfg file which is located in %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter or %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\.
• Back up the license server .lic key file which is usually located in C:\Program Files\VMware\VMware License Server\Licenses.
• Make note of your ODBC DSN, username, and password.

Agent pre-upgrade check tool
As part of the upgrade to vCenter Server, all of the hosts that are managed by it must have their vCenter Server agent (vpxa) that is running on them upgraded as well. It is fairly common to have this process fail on some hosts for various reasons, and as a result, they become disconnected from vCenter Server after the upgrade. When this happens you can manually get the agent to install, but the Agent Pre-upgrade Check tool is available that is designed to prevent the failure from happening. The tool checks each host for some of the usual things that cause the agent installation to fail. This includes making sure the host has sufficient disk space on the Service Console partitions, the host is reachable, the filesystem is intact, and any required patches are applied.

You can launch the tool from the vCenter Server installer; it is listed under Utility. Once you run the tool, it goes out to VMware’s website and downloads any new updates that are available for it. When the wizard launches follow these steps to complete it.

1. At the Database Connect screen, choose your ODBC DSN to connect to, and enter a username and password for it as well as the credential type. If you are using the built-in database, you can select Use VirtualCenter credentials.

2. At the Select Mode screen, choose to scan all hosts or select custom mode to select the hosts; if you select custom mode, the next screen will let you choose the hosts.

3. At the Run Test screen, click the “Run precheck” button to begin checking the hosts. This will not cause any disruption to your hosts or to the VMs running on them. Once the scan completes, click the Next button to see the results.

4. At the Pre-check screen, you will see your hosts listed as the results of the test. Click the links to see each host’s individual report, or click the Run Report button for the full results. If any of the hosts fail, read the results and resolve any issues with them. When you’re finished, you’re ready to upgrade vCenter Server.

Running the vCenter Server installer
When you are ready to begin the upgrade, run the vCenter Server 4.0 installer by running the autorun.exe file on the install media. From the vSphere vCenter Server installer menu, choose the vCenter Server product installer and follow these steps.

1. At the Welcome screen, the installer will detect a previous vCenter Server version. Click Next to begin the upgrade.

2. At the License Agreement screen, accept the agreement and click Next.

3. At the Customer Information screen, enter your information and optionally enter a vCenter Server 4.0 license key. If you do not enter one, you can always enter one later, as it will use a 60-day evaluation key instead. Click Next to continue.

4. At the Database Options screen, the ODBC DSN that is already used by vCenter Server will be selected and you must provide a username/password if you are using SQL authentication; if you are using Windows authentication, you can leave this field blank. If you are using the built-in database, you can leave those fields blank also. Click Next to continue.

5. You may receive a message that some of the existing extensions (e.g., Update Manager, Converter) will not be compatible after the upgrade. You can upgrade them manually afterward using the vCenter Server product installer. Click OK to continue.

6. At the Database Upgrade screen, you have an option to upgrade the existing vCenter Server database. If you choose not to upgrade, you cannot continue, as the database upgrade is required. You also must check the box confirming that you have backed up your database and SSL certificates before you can continue. This screen is meant as a final warning to let you know your database will be upgraded, and if you do not have a backup, you cannot roll back to vCenter Server 2.x. Click Next to continue.

7. At the vCenter Server Service screen, either accept the default to use the built-in SYSTEM account or specify a user domain account instead. Click Next to continue.

8. At the Configure Ports screen, you have the option to change any of the default port numbers that are used by vCenter Server. It’s recommended that you leave these at the defaults, unless you have changed the ports on your other servers because of a conflict. Be aware that if you are running another web server on the vCenter Server, such as Microsoft Internet Information Services (IIS), ports 80 and 443 will conflict with vCenter Server’s built-in Tomcat web server. Click Next to continue.

9. At the Ready to Install screen, click the Install button to begin the upgrade. Your database will be upgraded as part of the vCenter Server installation. If you are upgrading from vCenter Server 2.5, the Database Upgrade Wizard runs in the background and requires no interaction. If you are upgrading from vCenter Server 2.0.x, the Database Upgrade Wizard appears and you must complete the wizard. If the Database Upgrade Wizard does display, you will have the option to keep the existing performance data, discard it, or keep only the last 12 months of it. If you keep the data, the upgrade will take quite a while if there is a lot of old performance data in the database.

10. The installer will run and will install and configure various components as well as upgrade the database. Once it completes, click Finish and the wizard will close. You will need to restart the system next.

Post-installation steps
Once the vCenter Server restarts, you will need to connect to it using the vSphere Client. If you try to connect with the VI3 Client, you will get an error message stating that the required client support files need to be retrieved from the server and installed. You can run the vSphere Client installer by choosing the Run the Installer option; you can also download it and run it manually. In addition, you can access the vSphere Client installer from the vCenter Server install image or by accessing the vCenter Server with a web browser. When you run the vSphere Client installer you will also have the option to install the Host Update utility which you can use to upgrade hosts from VI3 to vSphere.

Once you install the vSphere Client and connect to the vCenter Server, the first thing you will want to do is set up the legacy licensing for your VI3 hosts. The hosts will be okay for 14 days if a licensing server is not available, but it is best to do this right away. If you did an in-place upgrade, you will not have to do anything, as this is taken care of automatically, even if your license server is on another server. You should verify, though, that your VMware License Server service is running.

If you uninstalled vCenter Server or did a clean installation, you will need to configure the legacy licensing support. To do this select vCenter Server Settings under the Administration section and then select Licensing in the left pane. In the right pane, you can enter the IP address/host name of a new or existing VI3 licensing server and check the option to Reconfigure ESX 3 hosts. You can verify that the VI3 licenses are being read by selecting Licensing under the Administration section. All your VI3 licenses should be displayed in the Product view.

After the installation, if you are using an Oracle database, you should copy the Oracle JDBC Driver (ojdbc14.jar) to the <VMware vCenter Server>\tomcat\lib folder. For SQL Server databases, if you enabled bulk logging for the upgrade, disable it after the upgrade is complete.

Finally, you will want to upgrade your extensions to the new vSphere versions. To do this, run the vCenter Server 4.0 installer and select each extension. You can install the Update Manager, Converter, and Guided Consolidation extensions. When you run the installation, the installer will detect the old version of each extension and automatically upgrade it. After you upgrade the extensions, you must also upgrade the plug-in component in each vSphere Client instance by going to the Plug-in Manager and installing the new plug-ins.

Phase 2: Upgrading ESX and ESXi
You have several options for upgrading your ESX and ESXi hosts to vSphere. The first option is to use the Update Manager plug-in that comes with vCenter Server. The second is to use the Host Update utility that installs with the vSphere Client that is normally used to patch/upgrade ESXi hosts. The Host Update utility is typically used to patch stand-alone hosts when vCenter Server is not being used, but in vSphere, it can now upgrade VI3 ESX hosts to vSphere as well. The final option is to simply burn the vSphere install ISO to a DVD, boot the host from it, and run through the installer.

Update Manager is much more robust than the Host Update utility, and you can orchestrate upgrades so that both hosts and VMs are updated. You can schedule upgrades via Update Manager, and automation and workflow are built into it to make the process much easier. The Host Update utility provides an easy way to upgrade hosts from a remote location, without a CD, and with minimal down time. The utility upgrades the VMkernel and the service console on ESX hosts, and upgrades the image of ESXi hosts. It does not upgrade VMFS datastores, VM hardware, or VMware Tools, as Update Manager does.

Using the host update utility
The Host Update utility is an optional add-on when installing the vSphere Client. If you do not have it, you can simply reinstall the vSphere Client to install it. Once you are ready to use it to upgrade your host, follow these steps.

1. Launch the Host Update utility and you will first be prompted that the utility needs to download patches from the VMware patch repository. Click No for this, as we are using the utility to upgrade hosts and not patch them.

2. From the top menu, select Host and then select Add Host and enter the IP address/host name of the host you wish to update. The host information will be displayed; note that patching is supported only for ESXi hosts and not for ESX hosts, but you can use the utility to upgrade ESX hosts.

3. You will need to put the host in maintenance mode before you can upgrade it, so shut down the VMs on the host or move them to other hosts before proceeding.

4. To begin the upgrade, click the Upgrade Host button and the Upgrade Wizard will launch.

5. At the Image Location screen, select the ISO or ZIP file of the ESX/ESXi 4.0 image that you wish to use to upgrade the host to, and click Next to continue.

6. The ISO will be validated. At the EULA screen, accept the licensing agreement and click Next to continue.

7. At the Host Credentials screen, enter the host user credentials (root) and click Next. If you have not placed the host in maintenance mode yet, you will be warned that you need to before proceeding. At the Host Compatibility Check screen, the host will be checked and any problems will be noted; if there are no problems, it will automatically go to the next screen.

8. At the Console OS Settings screen, select a datastore and size for the Service Console virtual disk file. You must choose either a local or a storage area network (SAN) datastore, as neither software iSCSI nor NFS datastores are supported for this. If you are using a SAN datastore, it cannot be shared with other hosts and it must be zoned to only that host. The minimum size for the virtual disk is 7.82GB; however, if you want to create additional partitions inside the Service Console per the recommendations in Chapter 11, you should increase this to at least 18GB. Once you select a datastore and size, click Next to continue.

9. At the Post-Upgrade Options screen, choose whether a rollback will be attempted in case the upgrade fails. In addition, you can choose to run a post-upgrade script and also select to roll back the upgrade if the script fails. Click Next to continue.

10. At the Ready to Complete screen, click Finish to begin the upgrade. The upgrade will proceed and the status will be displayed. First the ISO image will be copied to the host, and then the host will be shut down and restarted and the upgrade will begin. The new Service Console VM will be created and partitioned, and the packages will be installed inside it; then it will reboot again once it is complete.

When the host restarts you will get a message that the upgrade has succeeded. If the upgrade failed for whatever reason, you should restart the Host Update utility before trying again. You can also view the log located on the host in the /var/log/vmware/vua.log file. If the host shows Disconnected in vCenter Server afterward, right-click on it and select Connect. Finally, exit maintenance mode, and the upgrade will be complete and your host will be ready to use.

Using update manager

Update Manager uses baselines that are created and then used to determine if a host is compliant with the specification of the baseline. If it is not, the host is remediated to bring it into compliance with the baseline. To upgrade hosts to vSphere with Update Manager, you need to first create a new upgrade baseline for vSphere 4.0. Once you create the baseline, you attach it to the host and then the upgrade begins. To create a new baseline, follow these steps.

1. Launch Update Manager from the vSphere Client, select the Baseline and Groups tab, and then select the Upgrade Baselines tab below it.

2. Click the Create link to create an Upgrade Baseline.

3. At the Baseline Name & Type screen, enter a name for the baseline and choose the Host Upgrade baseline type; click Next to continue.

4. At the Upgrade Version screen, select an upgrade ISO file to use for ESX hosts and an upgrade ZIP file to use for ESXi hosts. These files are available on the vSphere download page on VMware’s website. Once you click Next, the files will be uploaded to the Update Manager server and imported.

5. At the COS VMDK Location screen, choose to automatically use a datastore which will use a local datastore, or manually choose a datastore instead. Click Next to continue.

6. At the Post-upgrade Options screen, choose whether a rollback will be attempted in case the upgrade fails. In addition, you can choose to run a post-upgrade script and select to roll back the upgrade if the script fails. Click Next to continue.

7. At the Ready to Complete screen, click Finish to create the baseline. Now that you have a baseline created, you can attach it to your VI3 hosts to bring them into compliance with the baseline. When you attach the baseline to your hosts, your hosts will be evaluated, and if they are not in compliance with the baseline, they will be upgraded to bring them into compliance. Follow these steps to attach the baseline to a host.

1. In the Hosts and Clusters tab, select the VI3 host that you want to upgrade to vSphere, and in the right pane select the Update Manager tab.

2. Click the link to Attach the vSphere upgrade baseline to the host.

3. Once the baseline is attached, click the Remediate button to begin the process of upgrading the host to make it compliant.

4. The Remediation Wizard will launch; at the Remediation Selection screen, select the Upgrade baseline and click Next to continue.

5. At the EULA screen, accept the licensing agreement and click Next to continue.

6. At the ESX 4.0 Upgrade screen, you have the option to change the COS VMDK Location, Rollback on Failure, and Post-upgrade Script settings. These settings will display even if you are upgrading an ESXi host, in which case you can ignore them. Click Next to continue.

7. At the Host Remediation Options screen, choose to have the remediation occur immediately, or scheduled for a later time. In addition, you can specify failure options for when the host is placed in maintenance mode. If VMs cannot be powered off or VMotioned to other hosts, you can specify to retry, fail the task, power them off, or suspend them. Click Next to continue.

8. At the Ready to Complete screen, click Finish to begin the remediation. The host will automatically enter maintenance mode and restart, and the upgrade will begin.

Once the remediation completes, the host will show as compliant and will automatically exit maintenance mode. If the upgrade fails, you can check the logs to try to find out why. On ESXi hosts, you can use the vSphere Client to export the logs, which are located in /esx3–installation/esx4–upgrade/ and /var/log/vmware/. For ESX hosts, the logs are located in /esx4–upgrade/ and /var/log/vmware/ on the host server.

Post-upgrade considerations
Once you upgrade your hosts, there are some steps you may need to perform afterward to clean them up.

• If vCenter Server manages the host, you must reconnect the host to vCenter Server by right-clicking the host in the vCenter Server inventory and selecting Connect.
• When the upgrade is complete, ESX/ESXi is in evaluation mode. Evaluation mode lasts for 60 days. You must assign an upgraded license to your product within 60 days of the upgrade.
• After the ESX/ESXi upgrade, third-party agents are disabled but remain on the disk. To reenable them, you must reinstall them. You can use the vihostupdate vSphere CLI command to install third-party extensions.
• For ESX hosts only, the web access service is disabled by default in vSphere as a security measure. If you want to enable it, follow these steps.

1. Log in to the ESX Service Console.

2. Type service vmware–webAccess status to check the status of the service.

3. Type service vmware–webAccess start to start the service. The service will run only until the host is restarted.

4. To have the service start each time the host boots, type chkconfig––level 345 vmware–webAccess on.

• Once you have determined the upgrade is stable, you can remove the ESX3 instance from the bootloader menu that displays when the host starts up. This removes the ability to roll back to ESX3 and deletes the ESX3 files from the /boot directory, deletes the ESX3 references from the /etc/fstab directory, and deletes the /usr/sbin/rollback-to-esx3 script. 

To remove ESX3 from the bootloader menu, follow these steps.

1. Log in to the ESX Service Console.

2. Type cleanup–esx3; optionally, you can use the –f parameter to suppress the confirmation message.

3. Reboot the host. The ESX3 option will be gone from the bootloader menu.

• Once you have upgraded all your VI3 hosts to vSphere, you can uninstall the VMware License Server application from the vCenter Server.

Phase 3: Upgrading virtual machines

After you upgrade your hosts to vSphere, you need to upgrade your VMs as well. This includes upgrading the virtual hardware from version 4 that is used in VI3 to version 7 that is used in vSphere, and also upgrading VMware Tools to the latest version. However, if there is a possibility that the VMs will be migrated to a VI3 host due to a VMotion, HA, or DRS event, you should wait as the version 7 VM hardware is not supported on VI3 hosts. There are two ways you can upgrade both the virtual hardware and VMware Tools. You can use the vSphere Client and upgrade VMs one by one, or you can use Update Manager and update many VMs at once. You must first power off the VMs to upgrade their virtual hardware. VMware Tools upgrades can be done while the VM is powered on, but Windows VMs must be restarted afterward. The process for upgrading Windows VMs is as follows.

1. Power on the VM.

2. Upgrade VMware Tools.

3. Reboot the VM at the end of the VMware Tools upgrade.

4. Power off the VM.

5. Upgrade the virtual hardware.

6. Power on the VM. Windows detects new devices and prompts you to reboot the VM.

7. Reboot the VM to make the devices work properly.

Upgrading VMware tools

To upgrade VMware Tools on a VM using the vSphere Client, follow these steps.

1. Make sure the VMs are powered on, open the console for the VM, and log in to the guest OS.

2. Select the VM in the vSphere Client, right-click on it, and select Guest and then Install/Upgrade VMware Tools.

3. Choose either an Interactive or Automatic upgrade. The Interactive option allows you to select which components to install. The Automatic option will install VMware Tools without prompts and automatically restart the VM afterward.

To upgrade VMware Tools on multiple VMs at once, follow these steps.

1. Select a host in the left pane, and then select the Virtual Machine tab.

2. Hold the Ctrl key while clicking on multiple powered-on VMs.

3. Once you have selected them all, right-click and select Guest→Install/Upgrade VMware Tools.

To configure VMs to automatically upgrade VMware Tools when they boot, follow these steps.

1. Power off the VM.

2. Edit the settings of the VM.

3. On the Options tab, select VMware Tools in the left pane, and in the right pane under Advanced, put a checkmark by the “Check and upgrade Tools before each power-on” field.

The next time the VM is powered on it will install or upgrade VMware Tools automatically and restart if necessary.

Upgrading Virtual Machine Hardware
Upgrading virtual hardware from version 4 to version 7 is reversible if you take a VM backup or snapshot before performing the upgrade. If you do not take a snapshot before upgrading, you must use vCenter Converter to create a new VM with version 4 hardware. VMs must be powered off when their virtual hardware is upgraded. Also make sure you upgrade VMware Tools before upgrading virtual hardware. To upgrade the virtual hardware of a VM, follow these steps.

1. Power off the VM.

2. Select the VM in the vSphere Client, right-click on it, and select Upgrade Virtual Hardware.

3. Choose Yes at the confirmation message and the upgrade will proceed.

4. Power on the VM.

5. Log in to Windows. Restart once the new devices are detected.

To upgrade virtual hardware on multiple VMs at once, follow these steps.

1. Select a host in the left pane, and then select the Virtual Machine tab.

2. Hold the Ctrl key while clicking on multiple powered-off VMs.

3. Once you have selected them all, right-click and select Upgrade Virtual Hardware.

Using update manager to upgrade VMware tools and virtual hardware
You can also use Update Manager to upgrade VMware Tools on VMs. Baselines have already been created for VMs to upgrade their VMware Tools and virtual hardware version to match the host, as shown in Figure 12.2.

Figure 12.2 Upgrading virtual hardware and VMware Tools using baselines

To upgrade VMs using Update Manager, just attach those baselines to the VMs and remediate them, and they will automatically be upgraded.

Once you upgrade your VM hardware, you can take advantage of the features in vSphere that require the new hardware. One new feature in particular that you may want to take advantage of is the new VMXNET3 virtual NIC (vNIC) type that offers more features and better performance than the VMXNET2 vNIC.

Upgrading your virtual environment to vSphere is exciting, as you can finally start using the many new features that are available in vSphere. It is best to get some experience using vSphere before upgrading so that you are better prepared to use it. Read through the documentation, set up a test lab if you can, and look through the Upgrade Center web page (www.vmware.com/products/vsphere/upgrade-center/) on VMware’s website.

There are many great guides, tools, and videos that can help prepare you for your upgrade to vSphere. The more prepared and experienced you are beforehand the smoother your upgrade will be when you go to do it.

Eric Siebert is a 25-year IT veteran whose primary focus is VMware virtualization and Windows server administration. He is one of the 300 vExperts named by VMware Inc. for 2009. He is the author of the book VI3 Implementation and Administration and a frequent TechTarget contributor. In addition, he maintains vSphere-land.com, a VMware information site.

Printed with permission from Pearson Publishing. Copyright 2010. Maximum vSphere: Tips, How-Tos, and Best Practices for Working with VMware vSphere 4 by Eric Siebert. For more information about this title and other similar books, please visit http://www.pearsonhighered.com.