A boosted model builds hundreds of trees, and every tree has to search for its best splits. That search is where the time goes. LightGBM cuts it two ways: histograms test fewer thresholds, and GOSS looks at fewer rows. Turn the knobs and see what each one saves and what it costs.
In gradient boosting each row carries a gradient: how wrong the current model is on that row, and in which direction. For squared error it is just the residual with the sign flipped. A good split puts rows that need pushing up on one side and rows that need pushing down on the other. Its quality, the gain, depends only on the gradient sums in each child: GL²/nL + GR²/nR − G²/n. Both tricks below are clever ways to get those sums cheaply.
Exact. Sort the rows by the feature, then try every gap between two neighbors. That is up to n − 1 thresholds per feature, per node, per tree.
Histogram. Drop each row into a bin once, add up gradients per bin, and try only the bin edges. The bars are those sums. The best edge is usually right next to the best exact split, so almost no gain is lost. Push the bins down to 2 or 3 to see where it breaks.
The idea. Rows with big gradients are where the model is most wrong, so they matter most for the next split. GOSS always keeps the top a share of them. From the rest it samples a b share and multiplies their gradients by (1 − a)/b, so the totals stay about right.
The error tiles. Each method estimates the total gradient with the same number of rows, 500 times. The tile shows the typical miss as a share of the total absolute gradient. Random sampling sometimes drops a big row and swings wildly. GOSS never does. Slide the concentration to low and the advantage disappears, because when every row is about equally wrong there is nothing special to keep.
| Histogram | GOSS | |
|---|---|---|
| What it cuts | Candidate thresholds | Rows per tree |
| How | Bucket feature values, test only the bin edges | Keep the large gradient rows, sample and upweight the rest |
| In LightGBM | Always on, max_bin defaults to 255 | Optional, the default strategy is bagging |
| Risk | Too few bins blur a fine split | Upweighted small rows add noise and can overfit on small data |
| Pays off when | Almost always, it is cheap and nearly lossless | Data is large and a minority of rows carries most of the error |
import lightgbm as lgb model = lgb.LGBMRegressor(max_bin=255, data_sample_strategy="goss", top_rate=0.2, other_rate=0.1)
Older LightGBM versions turned GOSS on with boosting_type="goss" instead. GOSS cannot be combined with bagging, so leave bagging_fraction at 1.
Boosting is sequential. Tree 200 cannot start until tree 199 is done, so you cannot just throw more machines at it. Making each split search cheaper is how LightGBM and XGBoost's hist mode train in seconds instead of hours.
It also grows trees leaf-wise, splitting whichever leaf gains most instead of a whole level at a time, and bundles sparse features that are rarely nonzero together. Leaf-wise growth is fast and accurate but needs num_leaves or max_depth to keep it from overfitting.
The Boosting lab runs both tricks inside a real model on the airline passengers data, using the model's actual residuals at every round.
Bagging also samples rows, but uniformly and with replacement, to make trees different from each other. GOSS samples rows by importance, to make each tree cheaper.