Monitoring vSphere CPU and memory usage

Monitoring CPU Usage

When monitoring a virtual machine, it's always a good starting point to keep an eye on CPU consumption. Many virtual machines started out in life as underperforming physical servers. One of VMware's most successful sales pitches is being able to take all those lackluster physical boxes that are not busy and convert them to virtual machines. Once converted, virtual infrastructure managers tend to think of these virtual machines as simple, lackluster, and low-utilization servers with nothing to worry over or monitor. The truth, though, is quite the opposite.

When the server was physical, it had an entire box to itself. Now it must share its resources with many other workloads. In aggregate, they represent quite a load, and if some or many of them become somewhat busy, they contend with each other for the finite capabilities of the ESX/ESXi host on which they run. Of course, they don't know they are contending for resources, but the VMkernel hypervisor tries to placate them. Virtual CPUs need to be scheduled, and ESX/ESXi does a remarkable job given that there are more virtual machines than physical processors most of the time. Still, the hypervisor can do only so much with the resources it has, and invariably there comes a time when the applications running in that virtual machine need more CPU time than the host can give.

When this happens, it's usually the application owner who notices first and raises the alarm with the system administrators. Now the vSphere administrators have the task of determining why this virtual machine is underperforming. Fortunately, vCenter Server provides a number of tools that make monitoring and analysis easier. These are the tools you've already seen: alarms, performance graphs, and command-line utilities.

Let's begin with a hypothetical scenario. A help desk ticket has been submitted indicating that an application owner isn't getting the expected level of performance on a particular server, which in this case is a virtual machine. As the vSphere administrator, you need to first delve deeper into the problem and ask as many questions as necessary to discover what the application owner needs to be satisfied with performance. Some performance issues are subjective, meaning some users might complain about the slowness of their applications, but they have no objective benchmark for such a claim. Other times, this is reflected in a specific benchmark, such as the number of transactions by a database server or throughput for a web server. In this case, our issue revolves around benchmarking CPU usage, so our application is CPU intensive when it does its job.

Assessments, Expectations, and Adjustments
If an assessment was done prior to virtualizing a server, there might be hard numbers to look at to give some details as to what was expected with regard to minimum performance or a service-level agreement (SLA). If not, the vSphere administrator needs to work with the application's owner to make more CPU resources available to the virtual machine when needed.

vCenter Server's graphs, which you have explored in great detail, are the best way to analyze usage, both short- and long-term. In this case, let's assume the help desk ticket describes a slowness issue in the last hour. As you've already seen, you can easily create a custom performance graph to show CPU usage over the last hour for a particular virtual machine or ESX/ESXi host.

Perform the following steps to create a CPU graph that shows data for a virtual machine from the last hour:

1. Connect to a vCenter Server instance with the vSphere Client.
2. Navigate to the Hosts And Clusters or VMs And Templates inventory view.
3. In the inventory tree, select a virtual machine.
4. Select the Performance tab from the content pane on the right, and then change the view to Advanced.
5. Click the Chart Options link.
6. In the Customize Performance Chart dialog box, select CPU from the resource type list. Select the Custom interval.
7. Near the bottom of the Chart Options section, change the interval to Last 1 Hours.
8. Set the chart type to Line graph.
9. Select the virtual machine itself from the list of objects.
10. From the list of counters, select CPU Usage In MHz (Average) and CPU Ready. This shows you how much processor is actually being used and how long it's taking to schedule the VM on a physical processor.
11. Click OK to apply the chart settings.

CPU Ready

CPU Ready shows how long a virtual machine is waiting to be scheduled on a physical processor. A virtual machine waiting many thousands of milliseconds to be scheduled on a processor might indicate that the ESX/ESXi host is overloaded, a Resource Pool has too tight a limit, or the virtual machine has too few CPU shares (or, if no one is complaining, nothing at all). Be sure to work with the server or application owner to determine an acceptable amount of CPU Ready for any CPU-intensive virtual machine.

This graph shows CPU utilization for the selected virtual machine, but it won't necessarily help you get to the bottom of why this particular virtual machine isn't performing as well as expected. In this scenario, I would fully expect the CPU Usage in MHz (Average) counter to be high; this simply tells you that the virtual machine is using all the CPU cycles it can get. Unless the CPU Ready counters are also high, indicating that the virtual machine is waiting on the host to schedule it onto a physical processor, you still haven't uncovered the cause of the slowness that triggered the help desk ticket. Instead, you'll need to move to monitoring host CPU usage.

Monitoring a host's overall CPU usage is fairly straightforward. Keep in mind that other factors usually come into play when looking at spare CPU capacity. Add-ons such as VMotion, VMware DRS, and VMware HA directly impact whether there is enough spare capacity on a server or a cluster of servers. Compared to earlier versions of ESX, the Service Console will usually not be as competitive for processor 0 because there are fewer processes to consume CPU time. Agents installed on the Service Console will have some impact, again on processor 0.

Service Console Stuck on 0

The Service Console, as noted, uses processor 0, but it will use processor 0 only. The Service Console does not get migrated to other processors even in the face of heavy contention.

Perform the following steps to create a real-time graph for a host's CPU usage:

1. Launch the vSphere Client if it is not already running, and connect to a vCenter Server instance.
2. Navigate to the Hosts And Clusters or VMs And Templates inventory view.
3. In the inventory tree, select a host. This shows you the Summary tab.
4. Click the Performance tab, and switch to Advanced view.
5. Click the Chart Options link.
6. In the Customize Performance Chart dialog box, select the CPU resource type and the Real-Time display interval.
7. Set Chart Type to Stacked Graph (Per VM).
8. Select all objects. You should see a separate object for each VM hosted on the selected ESX/ESXi host.
9. Select the CPU Usage (Average) performance counter.
10. Click OK to apply the chart settings and return to the Performance tab.

This chart shows the usage of all the virtual machines on the selected ESX/ESXi host in a stacked fashion. From this view, you should be able to determine whether there is a specific virtual machine or group of virtual machines that are consuming abnormal amounts of CPU capacity.

VMkernel Balancing Act

Always remember that on an oversubscribed ESX/ESXi host the VMkernel will load balance the virtual machines based on current loads, reservations, and shares represented on individual virtual machines and/or resource pools.

In this artificial scenario, I identified the application within the virtual machine as CPU-bound, so these two performance charts should clearly identify why the virtual machine isn't performing well. In all likelihood, the ESX/ESXi host on which the virtual machine is running doesn't have enough CPU capacity to satisfy the requests of all the virtual machines. Your solution, in this case, would be to use the resource allocation tools described in Chapter 10 to ensure that this specific application receives the resources it needs to perform at acceptable levels.

Monitoring Memory Usage

Monitoring memory usage, whether on a host or a virtual machine, can be challenging. The monitoring itself is not difficult; it's the availability of the physical resource that can be a challenge. Of the four resources, memory can be oversubscribed without much effort. Depending on the physical form factor chosen to host VMware ESX/ESXi, running out of physical RAM is easy to do. Although the blade form factor creates a very dense consolidation effort, the blades are sometimes constrained by the amount of physical memory and network adapters that can be installed. But even with other regular form factors, having enough memory installed comes down to how much the physical server can accommodate and your budget.

If you suspect that memory usage is a performance issue, the first step is to isolate whether this is a memory shortage affecting the host (you've oversubscribed physical memory and need to add more memory) or whether this is a memory limit affecting only that virtual machine (meaning you need to allocate more memory to this virtual machine or change resource allocation policies). Normally, if the ESX/ESXi host is suffering from high memory utilization, the predefined vCenter Server alarm will trigger and alert the vSphere administrator. However, the alarm doesn't allow you to delve deeper into the specifics of how the host is using memory. For that, you'll need a performance graph.

Perform the following steps to create a real-time graph for a host's memory usage:

1. Connect to a vCenter Server instance with the vSphere Client.
2. Navigate to the Hosts And Clusters inventory view.
3. In the inventory tree, click an ESX/ESXi host. This shows you the Summary tab.
4. Click the Performance tab, and switch to Advanced view.
5. Click the Chart Options link.
6. In the Customize Performance Chart dialog box, select the Memory resource type and the Real-Time display interval.
7. Select Line Graph as the chart type. The host will be selected as the only available object.
8. In the Counters area, select the Memory Usage (Average), Memory Overhead (Average), Memory Active (Average), Memory Consumed (Average), Memory Used by VMkernel, and Memory Swap Used (Average). This should give you a fairly clear picture of how memory is being used by the ESX/ESXi host.

Counters, Counters, and More Counters

As with virtual machines, a plethora of counters can be utilized with a host for monitoring memory usage. Which ones you select will depend on what you're looking for. Straight memory usage monitoring is common, but don't forget that there are other counters that could be helpful, such as Ballooning, Unreserved, VMkernel Swap, and Shared, just to name a few. The ability to assemble the appropriate counters for finding the right information comes with experience and depends on what is being monitored.

9. Click OK to apply the chart options and return to the Performance tab. These counters, in particular the Memory Swap Used (Average) counter, will give you an idea of whether the ESX/ESXi host is under memory pressure. If the ESX/ESXi host is not suffering from memory pressure and you still suspect a memory problem, then the issue likely lies with the virtual machine.

Perform the following steps to create a real-time graph for a virtual machine's memory usage:

1. Use the vSphere client to connect to a vCenter Server instance.
2. Navigate to either the Hosts And Clusters or the VMs And Templates inventory view.
3. In the inventory tree, click a virtual machine. This shows you the Summary tab.
4. Click the Performance tab, and switch to the Advanced view.
5. Click the Chart Options link.
6. In the Customize Performance Chart dialog box, select the Memory resource type and the Real-Time display interval.
7. Select Line Graph as the chart type.
8. In the list of counters, select to show the Memory Usage (Average), Memory Overhead (Average), Memory Consumed (Average), and Memory Granted (Average) counters. This shows memory usage, including usage relative to the amount of memory configured for the virtual machine.
9. Click OK to apply the chart options and return to the Performance tab.

From this performance graph, you will be able to tell how much of the memory configured for the virtual machine is actually being used. This might reveal to you that the applications running inside that virtual machine need more memory than the virtual machine has been assigned and that adding more memory to the virtual machine -- assuming that there is sufficient memory at the host level -- might improve performance.

Memory, like CPU, is just one of several different factors that can impact virtual machine performance. Network usage is another area that can impact performance, especially perceived performance.

Monitoring VMware vSphere performance
  Monitoring vSphere performance using vCenter Server alarms
  Monitoring vSphere performance with vCenter Server performance graphs 
  Monitoring vSphere performance with command-line tools
  Monitoring vSphere CPU and memory usage
 

Printed with permission from Wiley Publishing Inc. Copyright 2009. Mastering VMware vSphere 4 by Scott Lowe. For more information about this title and other similar books, please visit Wiley Publishing.