Training-Serving Skew vs Data Drift
People think training-serving skew is just data drift, and it is not. Drift is the world changing after the model was trained, so the live data slowly moves away. Skew is your own system building the same input two ways, one for training and one for live use. Drift needs new data and a retrain. Skew needs a bug fix.
Training-Serving Skew
Live data and training data are made in two ways. So a good model does badly.
Training-serving skew is when the data a model sees in real use differs from the data it was trained on, because of how the system was built. Often two pieces of code compute the same input in different ways. For example, training may use prices in dollars while the live app sends cents. The model still answers, but the answers are worse. It is a bug in the system, not a change in the world.
Data Drift
The data that comes in starts to look less like what the model learned on.
Data drift is when a model's live inputs drift away from its training data. The model was tuned for one mix of inputs, such as the ages, places or phones of its users. Now that mix has moved. The model may still be right about the cases it knows, but it sees more cases it learned little about. You find drift by comparing live inputs with training inputs over time. It is often the first sign that a model needs new training.
In one line
- Training-Serving Skew
- Live data and training data are made in two ways. So a good model does badly.
- Data Drift
- The data that comes in starts to look less like what the model learned on.