Native vs LightGBM benchmark

Medium-scale holdout comparison of the built-in histogram trainer (backend = 'native') against in-process LightGBM (backend = 'lightgbm') on a deterministic synthetic tabular problem. Numbers come from the committed results file produced by scripts/native_benchmark/run.py.

How to read this

Same hyperparameter MAP for both backends, ~200k rows, ~20% hash holdout. This is evidence that the native path learns useful structure at a moderate scale — not a bake-off that LightGBM loses in general. LightGBM still wins on huge data, GPU, and vendor-only features.

Generated: 2026-10-01T08:09:33.147409+00:00
Rows: 200,000 (seed 7)
Host: Linux-6.12.94+-x86_64-with-glibc2.39

Shared train options

Option	Value
`n_estimators`	`80`
`max_depth`	`5`
`learning_rate`	`0.1`
`min_samples_leaf`	`20`
`max_bins`	`64`
`reg_lambda`	`1.0`
`subsample`	`0.8`
`colsample_bytree`	`0.8`
`seed`	`7`

Both runs also set categorical_features to cat_a,cat_b,cat_c,cat_d.

Holdout results

Task	Backend	Metric	Holdout	Secondary	Train seconds
regression	`native`	rmse	0.238832		10.650
regression	`lightgbm`	rmse	0.244557		16.337
binary	`native`	logloss	0.048162	accuracy=0.9937	8.343
binary	`lightgbm`	logloss	0.057630	accuracy=0.9921	20.416

Takeaways from this run

Regression RMSE: native 0.2388 vs LightGBM 0.2446 (train 10.7 s vs 16.3 s).
Binary logloss: native 0.0482 vs LightGBM 0.0576 (accuracy 0.9937 vs 0.9921; train 8.3 s vs 20.4 s).

Problem design

Synthetic rows from range(n):

Feature group	Columns	Role
Categorical	`cat_a`…`cat_d`	Low-cardinality integer codes (`% 17`, `% 9`, …)
Continuous	`x1`…`x8`	Sines/cosines, scaled remainders, logs
Regression target	`y_reg`	Additive signal + `sin × cat` interaction + small noise
Binary target	`y_bin`	Threshold on a related score
Split	`split`	`hash(i, seed) % 5 = 0` → `test`, else `train`

Train wall time is measured with epoch_ms(current_timestamp) immediately before and after the duckboost_train aggregate (dataset build excluded). Holdout metrics use duckboost_evaluate_agg on the test split only.

Reproduce

# LightGBM-linked release build
EXT_FLAGS='-DDUCKBOOST_WITH_LIGHTGBM=ON' make release

python3 scripts/native_benchmark/run.py --n-rows 200000 --seed 7
# refreshes scripts/native_benchmark/results.json — re-render this page to update the table

See scripts/native_benchmark/README.md for flags and caveats.

What this does not show

Public leaderboard datasets (Higgs, airline, …)
Multi-million-row throughput or multithreaded native training
Vendor-only knobs (monotonic constraints, GPU, GOSS/EFB)
Perfect hyperparameter parity — DuckDB maps a shared MAP into each backend

For the SQL walkthrough on real CSVs, see the native trainer vignette.