Hadoop cluster configuration best practices streamline workflows

Every workload is unique, so a Hadoop cluster configuration that works well for one workload might not be optimal for another. The key to running jobs efficiently is to match your configuration to other active workloads.

One way to accomplish this is by creating a performance baseline. You can compare your configuration changes against that baseline to determine whether they had the desired effect.

To create a baseline, start with a default configuration and run a job; it should be representative of the types of jobs that you plan to run once you put the cluster into production. After you run a job, check its history to see how long it took, then calculate an average.

When you launch a job, the system creates two files: a job configuration XML file and a job status file. The job status file is useful for tracking the effect of your configuration changes because it contains status information and runtime metrics. You can view job status data with the Hadoop job command. Append the -List All parameter if you aren't sure of the job's identity.

Assuming you use the baseline comparison method to quantify your configuration changes, the most important thing to keep in mind is to make one change at a time. If you change several different parameters at the same time, then receive an unexpected result in your job status file, it can be difficult to determine which change -- or changes -- caused the unexpected behavior. Making one change at a time can be tedious, but it provides the most control over the cluster's performance.

There are several individual files that you might want to examine as you fine-tune your Hadoop cluster configuration. These are site-specific configuration files and default configuration files.