Hyperparameter Tuning vs Fine-Tuning
People think both kinds of tuning change the model, and only one does. Hyperparameter tuning picks the settings for training, like the learning rate, by training many times. Fine-tuning takes a model that is already trained and trains it more on new data. The first picks the recipe. The second keeps cooking.
Hyperparameter Tuning
Train a model many times, each with new settings. Keep the best.
Hyperparameter tuning is the search for the best settings to train a model with. Settings like the learning rate or the depth of a tree are chosen before training starts. You train the model many times, each with different settings. Then you score each one on the validation set and keep the best. A grid search tries every mix on a list. A random search tries mixes picked by chance, and often finds a good one sooner.
Fine-Tuning
More training on a model that is done, so it gets good at one job.
Fine-tuning is more training on a model that is already trained, using a small set of examples for one job. The model keeps what it knew and gets better at that job or style. A few hundred to a few thousand good examples are often enough. It is much cheaper than training from scratch. It changes the model's weights, so the new skill is built in.
In one line
- Hyperparameter Tuning
- Train a model many times, each with new settings. Keep the best.
- Fine-Tuning
- More training on a model that is done, so it gets good at one job.