Selecting benchmarking tools for specific needs
Narrowing down the best benchmarking tool requires testing and a careful evaluation of needs, including system compatibility, level of technical depth and scalability.
This tip is the first installment in our series on benchmarking best practices in the data center.
Selecting a benchmarking tool can be a daunting task because there are dozens of different tools available. Not only does Windows have its own built-in tool for benchmarking (Performance Monitor), but a quick Internet search reveals a wide array of third-party tools, ranging from freeware to expensive commercial products. So how do you know which tool to choose?
Believe it or not, product functionality is often irrelevant when choosing a benchmarking tool. After all, pretty much any of the available tools will tell you how hard your CPU is working or how fast your hard drive is (although it is always a good idea to make sure your tool of choice includes such basic functionality). The key to choosing a benchmarking tool is to look beyond the basics and decide whether or not a tool that you are considering will actually do the job that you need it to do.
Benchmarks and system compatibility
One of the first considerations that you need to take into account is system compatibility. It goes without saying that your tool of choice should work on the operating system that you're using, but there is more to it than that. For instance, you may discover that some older benchmarking tools do not work correctly on 64-bit servers.
And every once in a while, you may find that the tool's system requirements are a bit misleading. A few years ago, for example, I downloaded a freeware benchmarking tool that shall remain nameless. The minimum system requirements stated that you had to have Windows XP and a CD-RW drive. I needed to benchmark a computer that was running Windows XP, so I decided to give the tool a try. As it turned out, the tool wasn't actually designed for Windows XP. Instead, the download consisted of a bootable CD image. It was designed this way because the tool was intended to measure the system's raw capabilities from outside of Windows. As such, there was no way to use the tool to find out how well Windows was performing.
Benchmark's level of technical depth
Another important consideration is the level of technical depth that you require. Over the years, I have seen one or two tools that were so unbelievably complex that they ended up being completely useless to me. On the other hand, the Windows Experience Index is a built-in, consumer-oriented benchmarking tool that assesses various aspects of system performance on a scale from 1.0 to 7.9 (in Windows 7). Although the Windows Experience Index can give you a general idea of how well a system performs, it does not provide nearly enough technical information for it to be taken seriously by most IT pros.
You will find that most of the benchmarking tools on the market fall somewhere between the tools that I have just described. Even so, it is important to look closely at the level of technical depth that the tool actually provides, because some tools are geared more toward consumers while others are geared more toward engineers or IT professionals. Decide what kind of information you require and then verify that the tool actually provides those details in your specific environment.
Benchmarks and scalability
Do you need to benchmark one server or a thousand servers? Scalability will be another important factor to consider. The number of servers that you need to benchmark will weigh heavily on the type of benchmarking tool that you should purchase.
For instance, if you need to benchmark a large number of servers, then you probably don't want to buy a tool that has to be manually installed on each server and that stores the benchmark data locally. A better tool for such environments would be a product that uses agents to send performance data to a central server, where the data is stored in a database. Such a tool would also likely include a central management and reporting console used to manage the data that is being collected. One possible example of such a tool is OpManager from ManageEngine.
On the other hand, if you only have two or three servers to benchmark, you probably don't want to invest in an enterprise-class product like the one I just described. Such a tool is probably overkill for smaller environments, and the fact that it requires a dedicated server for collecting performance data may render this type of solution cost-prohibitive.
Benchmarks, virtualization and applications
Today, almost every organization uses virtual servers to at least some degree. What a lot of people don't realize, however, is that it can be extremely difficult to find out how well a virtual server is actually performing. Time-slicing and hardware agnostics cause many of the conventional benchmarking tools to produce results that are way out in left field. Therefore, if you plan to benchmark virtual servers, it is critical for you to make sure that your benchmarking software is virtualization-aware. One free option for benchmarking virtual machines is VMmark from VMware.
For more Server Month resources
Server features and functionality: An introduction
Spotting undervalued server features
Turning to 2U, 4U servers in virtualized data centers
Blade servers: An introduction and overview
Tactics for extending server life
Inside Intel and AMD processor technologies for Unix systems
Another factor to take into account is the type of benchmarking that you want to do. While there are dozens of general benchmarking tools available, some benchmarking tools are application-specific. For example, there are benchmarking tools that are specifically designed to monitor Exchange servers or SQL servers.
Keep in mind that you do not have to buy an application-specific benchmarking tool. There is no reason why you can't use a general benchmarking tool to monitor your Exchange servers, for example. However, application-specific tools often provide more insight than you can get with a general benchmarking tool. For example, a general benchmarking tool might be able to tell you how hard the volume containing an Exchange database is working, but it probably won't be able to monitor the number of database transactions per second.
Application-specific benchmarking tools are good for capacity planning and for alerting. For example, an Exchange-specific benchmarking tool might be able to alert you to a backup in a message queue (which would indicate mail flow issues), whereas a general tool would not have this capability. A tool like Microsoft's System Center Operations Manager is designed to monitor network servers using Performance Monitor counters. Once again, evaluate a variety of tools and select a product with the features and capabilities that best match your needs.
Benchmark licensing
One last consideration that you should take into account is the product's licensing model. Benchmarking your servers could become cost-prohibitive if the product that you choose requires a separate license for each server. If you are going to use a commercial product, then ideally you should look for one that is licensed in a way that allows you to benchmark as many systems as you like. A perfect example of such a product is PerformanceTest from PassMark Software. If you have difficulty finding such a product, then at least be aware of the licensing costs before you make the purchase.
Look before you leap
Being that there are so many different benchmarking products on the market, it can be difficult to find one that is a good fit for your needs. Thankfully, many vendors offer free trial versions of their wares. Downloading and installing the trial version of a benchmarking product is well worth the effort because doing so will allow you to definitively determine whether or not the product in question will meet your needs before you commit to purchasing it.
Check out the rest of our Server Month resources.
What did you think of this tip? Write to SearchDataCenter.com's Nicole Harding about your data center concerns at [email protected].
