Hyperparameter tuning techniques to optimize ML models

Why does hyperparameter tuning matter?

Hyperparameters are important because, in effect, they tell a model how to approach the tasks of parsing and finding relationships in data.

At best, choosing suboptimal hyperparameters leads to a more time-consuming training process -- which also raises computational costs by increasing CPU and memory usage. At worst, poor hyperparameter choices significantly reduce model performance or accuracy. This could happen if a model fails to generate the right parameters due to poor hyperparameter configurations that prevent the model from learning the data relationships necessary to make accurate predictions.

Hyperparameter tuning addresses this challenge by iterating across different configurations to determine which is ideal for a given model or use case. Although data scientists and machine learning engineers can tune hyperparameters manually, they can also partially or fully automate the process using tools that evaluate hyperparameter values and compare them to a desired outcome.

Hyperparameter examples

Some examples of common hyperparameters include the following:

• Number of neurons. This defines the total individual units that the model will parse within each layer of a neural network. More neurons usually mean better model performance, but using more neurons than necessary for the desired level of performance can needlessly complicate or lengthen the training process.
• Number of layers. The number of layers determines how deeply the model processes data, with each layer the model passes through adding a level of abstraction. Like the above, more layers generally translate to better performance, but having too many layers can slow training to an unacceptable pace.
• Learning rate. This hyperparameter controls the rate at which a model updates its parameters during training. A higher learning rate speeds up training, but risks causing the model to miss important nuances when generating parameters, potentially reducing accuracy.
• Train-test split. This value determines what proportion of the available data is allocated for training versus testing and validation. Generally, the more data used for training, the better the model will perform, but it's important to reserve enough data to effectively validate the model's performance. A standard split is 80% for training and 20% for testing and validation.
• Dropout rate. This setting defines the proportion of nodes ignored during training to prevent overfitting, which occurs when a model becomes too tailored to its original training data and struggles to generalize to new information.

These are just a handful of hyperparameter examples. In general, any configuration variable that shapes how a model interacts with its training data is a hyperparameter. Note, too, that not every type of hyperparameter is relevant to every model; hyperparameter choices depend on factors such as algorithm type and model architecture.

Hyperparameter tuning and optimization best practices

The first step in hyperparameter tuning is to decide whether to use a manual or automated approach.

Manual tuning means experimenting with different hyperparameter configurations by hand. This approach provides the greatest control over hyperparameters, maximizing the ability to tailor settings to specific use cases. The downside is that it requires more time and effort. Thus, manual hyperparameter tuning might not be worth the trouble for models that are relatively generic, but it can be helpful when dealing with specialized or nuanced use cases.

Automated hyperparameter tuning speeds up the process with tools that automatically test various hyperparameter configurations and optimize model performance by observing the effect on model behavior. In this way, automation makes it feasible to iterate through many different configurations quickly.

Some widely used techniques for automated hyperparameter tuning include the following:

• Grid search. This method systematically assesses every possible hyperparameter combination within a specified range. While grid search is effective at finding the ideal hyperparameter configuration because it considers all possible values, it can take a long time and is computationally expensive, especially for large models.
• Random search. Unlike grid search, which evaluates every value within a preset range, random search evaluates only a randomly selected subset of configurations. This is considerably faster than grid search, but might overlook the best configuration, given that it doesn't consider each possible value. This often results in a "good enough," rather than optimal, configuration.
• Bayesian optimization. This approach uses probabilistic modeling to predict promising hyperparameter values, then evaluates those configurations. The goal of Bayesian optimization is to identify an ideal configuration without considering every possible value, combining the speed of random search with the accuracy of grid search. The caveat, of course, is that ideal values might be overlooked if the probabilistic model fails to predict them.

The best tuning technique for a given project depends on several factors -- chief among them, time and computational resources. Projects with more time and resources available can afford to invest in the exhaustive but highly accurate process of tuning via grid search. Random search, while less precise, is likely a better fit in situations where choosing the absolute best hyperparameter values isn't critical for model performance. Bayesian optimization offers a compromise, balancing speed and accuracy, and accepting some risk of missing ideal configurations.

Chris Tozzi is a freelance writer, research adviser and professor of IT and society who has previously worked as a journalist and Linux systems administrator.