How wrong is the model? Measuring error
Every forecast is a little wrong. The question is: how wrong on average? And is that acceptable for the decision at hand? We compare the model's forecasts with the real figures in the test set:
| Month | Actual | Forecast | Error |
|---|---|---|---|
| June | 520 | 500 | +20 |
| July | 610 | 580 | +30 |
| August | 590 | 600 | −10 |
| September | 480 | 530 | −50 |
We can't simply add the errors, because positives cancel negatives. Two common measures:
| MAE | Mean absolute error: drop the sign and average. (20 + 30 + 10 + 50) ÷ 4 = 27.5. Easy to explain: “on average we're off by 27.5 cartons”. |
| RMSE | Root mean squared error: square the errors, average them, take the root. It comes to 31.2. Larger than MAE because squaring punishes big misses (like −50) more. |
PYTHON
actual = [520, 610, 590, 480] predicted = [500, 580, 600, 530] errors = [] for a, p in zip(actual, predicted): errors.append(a - p) print("Errors:", errors) mae = sum(abs(e) for e in errors) / len(errors) rmse = (sum(e ** 2 for e in errors) / len(errors)) ** 0.5 print("MAE: ", round(mae, 1)) print("RMSE:", round(rmse, 1))
Watch outA number alone isn't enough. An error of 27 cartons is excellent if sales are 5,000 and a disaster if they're 50. Compare the error with the scale of the numbers, and with today's method: if the planner is off by 60 on average and the model by 27, the model wins.