Module 5 · Machine Learning for Time Series Forecasting

The Learning Curve

Do we need more data, or a different model? Pick a model, then grow the training set and watch where training and validation error settle. The gap between them is variance. How far they sit above the noise floor is bias.

Dr. Pedram Jahangiry Website YouTube GitHub
Model complexity · polynomial degree
Shrinks the coefficients. A bigger λ trades variance for bias.
Sets the noise floor σ² = 4.00. No model can beat it.
Fresh points held out to score the model.
Try

Error vs training set size median of 40 random training sets

Training error Validation error Benchmark: predict the average Noise floor σ²

What the model learns at n = 30

Training points Validation points Average fit Fits on 24 other samples True function

Blue points train the model; orange points are held out to score it. Spread of the faint lines is variance. Distance between the average fit and the dashed truth is bias.

Training error–MSE on the n points it saw
Validation error–MSE on held-out points
Gap–validation minus training
Noise floor–σ², irreducible
–

High bias

You see
Both curves meet quickly, far above the floor. Small gap.
More data?
No. The curves are already flat.
Fix
More complexity, more features, less λ.

High variance

You see
Training error near zero, validation error high. Big gap.
More data?
Yes. Validation is still falling.
Fix
More data, lower degree, or raise λ.

Good fit

You see
Both curves squeeze the noise floor from either side.
More data?
A little, with fast diminishing returns.
Fix
Nothing big. You are near the best possible.

Variance shrinks with n. Bias does not.

Press play on degree 10. At n = 10 the faint fits fly everywhere and the gap is huge. By n = 150 they have calmed down. Now try degree 1: the fits agree with each other from the start, but they all miss the curve in the same way, at every n.

The noise floor is a hard limit.

On average, validation error can approach σ² but not go below it, because no model can predict the random noise. A small validation set gives a noisy score that can dip under the floor by luck. Raise the validation size and the orange curve steadies. Drop σ to 0.5 and the floor falls, so even the small cubic term suddenly matters and degree 2 starts to look biased.