Start from a dumb forecast: last month plus average growth. Then add small trees one at a time, each one trained on the errors the model still makes. On this data the first job the trees learn is the seasonal shape, summer up and November down. Train on 1950 to 1958, forecast 1959 and 1960 one month ahead.
–
Training error keeps falling because every tree is built to shrink it. Test error falls while the trees learn real structure, then flattens or creeps back up once they start fitting noise. In practice you hold out a validation window and stop at its best round. That is early stopping.
Every tree has to search for its best split. Both panels below use the real residuals of the model above at round 60, so move the rounds slider and watch them change. For a cleaner toy version, see Histogram vs GOSS Splitting.
Exact search sorts the rows and tries every gap between neighbors. The histogram drops each row into a bin once, adds up residuals per bin, and only tries the bin edges. With a decent number of bins it usually keeps nearly all of the gain.
Rows with big residuals are where the model is most wrong, so GOSS always keeps them. From the rest it takes a small random sample and scales it up by (1 − a)/b, so totals stay roughly right. The error mass tiles show what share of all the absolute residual the sampled rows carry, GOSS against a plain random sample of the same size. GOSS pays off most when a few rows hold most of the error, which is common in large, messy data. Airline residuals are spread fairly evenly, so here the gain is modest.
| Histogram | GOSS | |
|---|---|---|
| What it cuts | Candidate thresholds | Rows per tree |
| How | Bucket feature values, test only the bin edges | Keep the large residual rows, sample and upweight the rest |
| In LightGBM | Always on, max_bin defaults to 255 | Optional, the default strategy is bagging |
| Risk | Too few bins blur a fine split. Try 4 bins on calendar month above. | Upweighted small rows add noise and can overfit on small data |
| Pays off on | Any data size | Large data, where cutting 70% of rows per tree saves real time |
import lightgbm as lgb model = lgb.LGBMRegressor(max_bin=255, data_sample_strategy="goss", top_rate=0.2, other_rate=0.1)
Older LightGBM versions turned GOSS on with boosting_type="goss" instead.
Round 0 predicts the average target for every row. Here that is last month's passengers grown by the average monthly change, which lags the seasonal swings badly.
Each new tree is trained on the current residuals, the negative gradient of squared error. Its answer is shrunk by η and added on. Small trees and a small η make many careful steps instead of a few big ones.
Training error only goes down. Test error does not. Choose the number of rounds on a validation window that comes after the training data, never by looking at training error.
Boosting adds shallow trees in sequence to cut bias, so stumps work well here. Bagging averages deep trees in parallel to cut variance, where stumps barely help. Neither can extrapolate a trend, so both need a stationary target.