Monitoring vSphere performance with vCenter Server performance graphs
vCenter Server's performance graphs provide solutions providers with detailed information on vSphere performance, including CPU, memory, disk, network and system data.
Solutions provider takeaway: Use this chapter excerpt to learn how to read vCenter Server's performance graphs and interpret data on your customer's CPU, memory, disk, network and system information.
Working with Performance Graphs
Alarms are a great tool for alerting administrators of specific conditions or events, but alarms don't provide the detailed information that administrators sometimes need to have. This is where vCenter Server's performance graphs come in. vCenter Server has many new and updated features for creating and analyzing graphs. Without these graphs, analyzing the performance of a virtual machine would be nearly impossible. Installing agents inside a virtual machine will not provide accurate details about the server's behavior or resource consumption. The reason for this is elementary: a virtual machine is configured only with virtual devices. Only the VMkernel knows the exact amount of resource consumption for any of those devices because it acts as the arbitrator between the virtual hardware and the physical hardware. In most virtual environments, the virtual machines' virtual devices can outnumber the actual physical hardware devices, necessitating the complex sharing and scheduling abilities in the VMkernel.
By clicking the Performance tab for a data center, cluster, host, or virtual machine, you can learn a wealth of information. Before you use these graphs to help analyze resource consumption, I need to help you get to know the performance graphs and legends. I'll start with covering the two different layouts available in performance graphs: the overview layout and the advanced layout. First up is the overview layout.
Overview Layout
The Overview layout is the default view when you access the Performance tab. Figure 12.8 shows you the Overview layout of the Performance tab for an ESX host. Note the horizontal and vertical scrollbars; there's a lot more information here than the vSphere Client can fit in a single screen.
Figure 12.8
The Overview layout provides information on a range of performance counters.
At the top of the Overview layout are options to change the view or to change the date range. The contents of the View drop-down list change depending upon the object you select in the vSphere Client. Table 12.1 lists the different options available, depending upon what type of object you select in the vSphere Client.
Next to the View drop-down list is an option to change the date range for the data currently displayed in the various performance graphs. This allows you to set the time range to a day, a week, a month, or a custom value.
In the upper-right corner of the Overview layout, you'll see a button for refreshing the display and a button for getting help.
Below the gray title bar (where you'll find the View and Time Range drop-down lists, the Refresh button, and the Help button) are the actual performance graphs. The layout and the graphs that are included vary based on the object selected and the option chosen in the View drop-down list. I don't have the room here to list all of them, but a couple of them are shown in Figure 12.9 and Figure 12.10. I encourage you to explore a bit and find the layouts that work best for you.
Table 12.1: View Options on the Performance Tab
|
|
|
| Datacenter | Clusters Storage |
| Cluster | Home Resource Pools & Virtual Machines Hosts |
| Resource pool | Home Resource Pools & Virtual Machines |
| Host | Home Virtual Machines |
| Virtual machine | Home Fault Tolerance Storage |
Figure 12.9
The Performance tab for an ESX/ESXi host in Overview layout includes eight charts, many of which are shown off-screen.
The Overview layout works well if you need a broad overview of the performance data for a datacenter, cluster, resource pool, host, or virtual machine. But what if you need more specific data in a more customizable format? The Advanced layout is the answer, as you'll see in the next section.
Advanced Layout
Figure 12.11 shows the Advanced layout of the Performance tab for a cluster of ESX/ESXi hosts. Here, in the Advanced layout, is where the real power of vCenter Server's performance graphs is made available to you.
Figure 12.10
The Storage view of the Performance tab for a virtual machine in Overview layout displays a breakdown of storage utilization.
Figure 12.11
The Advanced layout of the Performance tab provides much more extensive controls for viewing performance data.
Starting from the top left, you'll see the name of the object being monitored. Just below that is the type of the chart and the time range. The Chart Options link provides access to customize settings for the chart. To the right, you'll find a drop-down list to quickly switch graph settings, followed by buttons to print the chart, refresh the chart, save the chart, or view the chart as a pop-up chart. The Print button allows you to print the chart; the Save button allows you to export the chart as a JPEG graphic. I'll discuss this functionality in the section ''Saving Performance Graphs.'' The Refresh button refreshes the data. The pop-up button opens the chart in a new window. This allows you to navigate elsewhere in the vSphere Client while still keeping a performance graph open in a separate window. Pop-up charts also make it easy to compare one ESX/ESXi host or virtual machine with another host or virtual machine. On each side of the graph are units of measure. In Figure 12.13, the counters selected are measured in percentages and megahertz. Depending on the counters chosen, there may be only one unit of measurement, but no more than two. Next, on the horizontal axis, is the time interval. Below that, the Performance Chart Legend provides color-coded keys to help the user find a specific object or item of interest. This area also breaks down the graph into the object being measured; the measurement being used; the units of measure; and the Latest, Maximum, Minimum, and Average measurements recorded for that object.
Hovering the mouse pointer over the graph at a particular recorded interval of interest displays the data points at that specific moment in time.
Another nice feature of the graphs is the ability to emphasize a specific object so that it is easier to pick out this object from the other objects. By clicking the specific key at the bottom, the key and its color representing a specific objectwill be emphasized, while the other keys and their respective colors become lighter and less visible. For simple charts such as the one shown previously in Figure 12.11, this might not be very helpful. For busier charts with many performance counters, this feature is very useful.
Now that you have a feel for the Advanced layout, take a closer look at the Chart Options link. This link exposes vCenter Server's functionality in creating highly customized performance graphs. Figure 12.12 shows the Customize Performance Chart dialog box. This dialog box is the central place where you will come to customize vCenter Server's performance graphs. From here, you select the counters to view, the time ranges, and the kind of graph (line graph or stacked graph) to display.
Figure 12.12
The Customize Performance Chart dialog box offers tremendous flexibility to create exactly the performance graph you need.
Because there is so much information available in the Customize Performance Chart dialog box, I've grouped the various options and types of information into the sections that follow.
Choosing a Resource Type
On the left side of the Customize Performance Chart dialog box, you can choose which resource (Cluster Services, CPU, Disk, Management agent, Memory, Network, or System) to monitor or analyze. The actual selections available in this area change depending upon the type of object that you have selected in vCenter Server. That is, the options available when viewing the Performance tab for an ESX/ESXi host are different from the options available when viewing the Performance tab of a virtual machine, a cluster, or a datacenter.
Within each of these resources, different objects and counters are available. Be aware that other factors affect what objects and counters are available to view; for example, in some cases the real-time interval shows more objects and counters than other intervals. The next few sections list the various counters that are available for the different resource types in the Customize Performance Chart dialog box.
If a particular counter is new to you, click it to highlight the counter. At the bottom of the dialog box, in a section called Counter Description, you'll see a description of the counter. This can help you determine which counters are most applicable in any given situation.
Setting a Custom Interval
Within each of the resource types, you have a choice of intervals to view. Some objects offer a Real-Time option; this option shows what is happening with that resource right now. The others are self-explanatory. The Custom option allows you to specify exactly what you'd like to see on the performance graph. For example, you could specify that you'd like to see performance data for the last eight hours. Having all of these interval options allows you to choose exactly the right interval necessary to view the precise data you're seeking.
Viewing CPU Performance Information
If you select the CPU resource type in the Chart Options section of the Customize Performance Chart dialog box, you can choose what specific objects and counters you'd like to see in the performance graph. Note that the CPU resource type is not available when viewing the Performance tab of a datacenter object. It is available for clusters, ESX/ESXi hosts, resource pools, and individual virtual machines.
Table 12.2 lists the objects and counters available for CPU performance information. Because CPU performance counters are not available at the datacenter object, the DC column is shaded. Not all these counters are available with all display intervals.
Quite a bit of CPU performance information is available. In the section ''Monitoring CPU Usage,'' I'll discuss how to use these CPU performance objects and counters to monitor CPU usage.
Viewing Memory Performance Information
If you select the memory resource type in the Chart Options section of the Customize Performance Chart dialog box, different objects and counters are available for display in the performance graph. The memory resource type is not available when viewing the Performance tab of a datacenter object. It is available for clusters, ESX/ESXi hosts, resource pools, and individual virtual machines.
In Table 12.3 you'll find the objects and counters available for memory performance information, depending upon the inventory object and display interval selected. As in Table 12.2, the DC column is shaded because memory counters are not available at the datacenter object. Not all these objects are available with all display intervals.
Later, in the section ''Monitoring Memory Usage,'' you'll get the opportunity to use these different objects and counters to monitor how ESX/ESXi and virtual machines are using memory.
Table 12.2: Available CPU Performance Counters
|
|
|
|
|
|
|
| CPU used | X | X | |||
| CPU usage (Average) | X | X | X | X | |
| CPU usage in MHz (Average) | X | X | X | X | |
| CPU reserved capacity | X | ||||
| CPU idle | X | X | |||
| CPU ready | X | ||||
| CPU system | X | ||||
| CPU wait | X | ||||
| Cluster total | X | ||||
| CPU entitlement | X | X |
Viewing Disk Performance Information
Disk performance is another key area that vSphere administrators need to monitor. Table 12.4 shows you the performance counters that are available for disk performance. Note that these counters aren't supported for datacenters, clusters, and resource pools, but they are supported for ESX/ESXi hosts and virtual machines. I've shaded the DC, CL, and RP columns in Table 12.4 because these counters are not available for datacenter, cluster, or resource pool objects. Not all counters are visible in all display intervals.
You'll use these counters in the section ''Monitoring Disk Usage'' later in this chapter.
Viewing Network Performance Information
To monitor network performance, the vCenter Server performance graphs cover a wide collection of performance counters. Network performance counters are available only for ESX/ESXi hosts and virtual machines; they are not available for datacenter objects, clusters, or resource pools.
Table 12.5 shows the different network performance counters that are available. The DC, CL, and RP columns are shaded because network performance counters are not available for datacenter, cluster, and resource pool objects.
You'll use these network performance counters in the ''Monitoring Network Usage'' section later in this chapter.
Viewing System Performance Information
ESX/ESXi hosts and virtualmachines also offer some performance counters in the System resource type. Datacenters, clusters, and resource pools do not support any system performance counters.
Table 12.3: Available Memory Performance Counters
|
|
|
|
|
|
|
| Memory usage (Average) | X | X | X | ||
| Memory overhead (Average) | X | X | X | X | |
| Memory consumed (Average) | X | X | X | X | |
| Memory total | X | ||||
| Memory shared common (Average) | X | ||||
| Memory granted (Average) | X | X | |||
| Memory balloon (Average) | X | X | |||
| Memory shared (Average) | X | X | |||
| Memory swap in (Average) | X | X | |||
| Memory active (Average) | X | X | |||
| Memory zero (Average) | X | X | |||
| Memory heap (Average) | X | ||||
| Swap out rate | X | X | |||
| Memory state | X | ||||
| Memory unreserved (Average) | X | ||||
| Memory reserved capacity | X | ||||
| Memory used by VMkernel | X | ||||
| Swap in rate | X | X | |||
| Memory swap out (Average) | X | X | |||
| Available heap memory | X | ||||
| Memory swap used (Average) | X | ||||
| Memory entitlement | X | ||||
| Memory balloon target (Average) | X | ||||
| Memory swap target (Average) | X | ||||
| Memory swapped (Average) | X |
More information on the system performance counters is available in Table 12.6. Because system performance counters are not available for datacenter, cluster, and resource pool objects, thesecolumns are shaded in Table 12.6.
Table 12.4: Available Disk Performance Counters
|
|
|
|
|
|
|
| Kernel disk command latency | X | ||||
| Disk read rate | X | X | |||
| Physical device command latency | X | ||||
| Queue write latency | X | ||||
| Disk commands issued | X | X | |||
| Physical device read latency | X | ||||
| Disk write requests | X | X | |||
| Kernel disk read latency | X | ||||
| Disk write latency | X | ||||
| Stop disk command | X | X | |||
| Disk write rate | X | X | |||
| Queue command latency | X | ||||
| Disk bus resets | X | X | |||
| Disk command latency | X | ||||
| Disk read latency | X | ||||
| Disk read requests | X | X | |||
| Queue read latency | X | ||||
| Kernel disk write latency | X | ||||
| Physical device write latency | X | ||||
| Disk usage (Average) | X | X | |||
| Memory swap used (Average) | X | ||||
| Highest disk latency | X |
Table 12.5: Available Network Performance Counters
|
|
|
|
|
|
|
| Network data receive rate | X | X | |||
| Network packets received | X | X | |||
| droppedRx | X | ||||
| Network usage (Average) | X | X | |||
| Network packets transmitted | X | X | |||
| droppedTx | X | ||||
| Network data transmit rate | X | X |
The majority of these counters are valid only for ESX/ESXi hosts, and they all center around how resources are allocated or how the ESX/ESXi host itself is consuming CPU resources or memory. As such, I won't be discussing them in any greater detail later in this chapter. I've included them here for the sake of completeness.
Viewing Other Performance Counters
These are the other available performance counter types:
ESX/ESXi hosts also offer a resource type (found in the Customize Performance Chart dialog box in the Chart Options section) marked as Management Agent. This resource type has only two performance counters associated with it: Memory used (Average) and Memory swap used (Average). These counters monitor how much memory the vCenter Server agent is using on the ESX/ESXi host.
ESX/ESXi hosts participating in a cluster also have a resource type of Cluster Services, with two performance counters: CPU fairness and Memory fairness. Both of these counters show the distribution of resources within a cluster.
The datacenter object contains a resource type marked as Virtual Machine Operations. This resource type contains performance counters that simply monitor the number of times a particular VM operation has occurred. This includes VM power-on events, VM power-off events, VM resets, VMotion operations, and Storage VMotion operations.
I've included this brief description of these counters for the sake of completeness, but I won't be discussing them any further.
Managing Chart Settings
There's one more area of the Customize Performance Chart dialog box that I'll discuss, and that's the Manage Chart Settings and Save Chart Settings buttons in the lower-right corner.
Table 12.6: Available System Performance Counters
|
|
|
|
|
|
|
| Resource CPU usage (Average) | X | ||||
| Resource memory allocation maximum (in KB) | X | ||||
| Resource CPU running (1 min. average) | X | ||||
| Resource memory overhead | X | ||||
| Resource memory mapped | X | ||||
| Resource memory shared | X | ||||
| Resource memory swapped | X | ||||
| Resource memory zero | X | ||||
| Resource memory share saved | X | ||||
| Resource memory touched | X | ||||
| Resource allocation minimum (in KB) | X | ||||
| Resource CPU maximum limited (1 min.) | X | ||||
| Resource CPU allocation (in MHz) | X | ||||
| Resource CPU active (5 min. average) | X | ||||
| Resource CPU allocation maximum (in MHz) | X | ||||
| Resource CPU running (5 min. average) | X | ||||
| Resource CPU active (1 min. average) | X | ||||
| Resource CPU maximum limited (5 min.) | X | ||||
| Resource CPU allocation shares | X | ||||
| Resource memory allocation shares | X | ||||
| Uptime | X | X | |||
| cosDiskUsage | X | ||||
| Heartbeat | X |
After you've gone through and selected the resource type, display interval, objects, and performance counters that you'd like to see in the performance graph, you can save that collection of chart settings using the Save Chart Settings button. vCenter Server prompts you to enter a name for the saved chart settings. After a chart setting is saved, you can easily access it again from the drop-down list at the top of the performance graph advanced layout. Figure 12.13 shows the Switch To drop-down list, where two custom chart settings -- VM Activity and Cluster Resources -- are shown. By selecting either of these from the Switch To drop-down list, you can quickly switch to those settings. This allows you to define the performance charts that you need to see and then quickly switch between them.
Figure 12.13
You can access saved chart settings from the Switch To drop-down list.
The Manage Chart Settings button allows you to delete chart settings you've saved but no longer need.
In addition to offering you the option of saving the chart settings, vCenter Server also allows you to save the graph.
Saving Performance Graphs
When I first introduced you to the Advanced layout view of the Performance tab, I briefly mentioned the Save button. This button, found in the upper-right corner of the Advanced layout, allows you to save the results of the performance graph to an external file for long-term archiving, analysis, or reporting.
When you click the Save button, a standard Windows Save dialog box appears. You have the option of choosing where to save the resulting file as well as the option of saving the chart either as a graphic file or as a Microsoft Excel spreadsheet. If you are going to perform any additional analysis, the option to save the chart data as an Excel spreadsheet is quite useful. The graphics options are useful when you need to put the performance data into a report of some sort.
There's a lot of information exposed via vCenter Server's performance graphs. I'll revisit the performance graphs again in the sections on monitoring specific types of resources later in this chapter, but first I'll introduce you to a few command-line tools you might also find useful in gathering performance information.
Monitoring VMware vSphere performance
Monitoring vSphere performance using vCenter Server alarms
Monitoring vSphere performance with vCenter Server performance graphs
Monitoring vSphere performance with command-line tools
Monitoring vSphere CPU and memory usage
Printed with permission from Wiley Publishing Inc. Copyright 2009. Mastering VMware vSphere 4 by Scott Lowe. For more information about this title and other similar books, please visit Wiley Publishing.
