Is Google's PerfKit a reliable cloud benchmarking tool?

In a growing pool of infrastructure as a service options, choosing the right vendor can be a challenge. With that in mind, Google released PerfKit Benchmarker, an open source tool that evaluates virtual machine performance in the three major IaaS providers: Amazon Web Services, Microsoft Azure and Google Compute Engine.

However, potential conflict of interest may cause some users to question whether PerfKit Benchmarker provides accurate and objective results.

The best way to assess any benchmark is to dig into the details. And, since all the PerfKit code is available on GitHub, this is easy to do.

Written in Python, the PerfKit benchmarking tool provides classes and methods to implement common IaaS operations. For example, there is code to implement VM objects, create scratch disks and execute commands on VMs. Additionally, specialized code manages IaaS-specific object features, such as Azure disks and AWS firewalls.

PerfKit implements benchmark tests for basic operations, such as block I/O, as well as many commonly used applications, including Cassandra, MongoDB and Redis. The benchmarker uses the Yahoo Cloud Serving Benchmark (YCSB) framework to run some of the loads, such as the MongoDB benchmark. Other tests are implemented using reasonable choices for operations. For example, the block I/O test involves tasks such as writing through the file system instead of using direct I/O, sequentially writing data, randomly reading 10% of bytes written and sequentially reading all bytes written.

Documentation states that PerfKit Benchmarker uses default configurations in the benchmark tests, which mitigates the risk of tests favoring one IaaS provider over another -- in terms of their configuration settings, at least. And because many cloud users accept default configurations when spinning up VMs, it seems reasonable for benchmark comparisons to use cloud providers' defaults. In addition, this practice isn't likely to produce biased benchmark results in ways that don't reflect typical use cases.

PerfKit Benchmarker builds on open source tools, such as the boto Python package and YCSB. The benchmarker code is straightforward and accessible to developers with Python and cloud admin experience. While there's always the risk of biased benchmark results, transparency is the best way to mitigate that risk.

About the author:
Dan Sullivan holds a master of science degree and is an author, systems architect and consultant with more than 20 years of IT experience. He has had engagements in advanced analytics, systems architecture, database design, enterprise security and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail and education. Dan has written extensively about topics that range from data warehousing, cloud computing and advanced analytics to security management, collaboration and text mining.