Seeing the Forest for the Trees: The Gaussian Process Limit of BART
arXiv:2607.28844
2026
Architecture
1 ideas extracted · analyzed Aug 31, 2026
What the math gives to ML
The paper identifies an explicit finite-dimensional representation of tree ensembles through leaf-indicator features and argues that, as the number of trees grows, these random features converge to a Gaussian-process model with a favorable kernel and RKHS. The transferable asset is not merely using trees, but treating randomly generated partitions as a fixed feature map whose width can be increased independently of the downstream predictor. A practical neural-network adaptation is a random-tree embedding layer for tabular inputs, concatenated with learned continuous features and followed by an MLP or ridge/linear head. This can provide low-variance interaction features and a strong small-data baseline without MCMC or end-to-end optimization of discrete split structures.
Ideas from this paper
Unverified
2026
Generate many random symmetric decision trees and encode each input by the one-hot indicator of its reached leaf. Use the resulting fixed random feature vector as an additional input to an MLP, or train only a ridge/linear prediction head on it. The tree ensemble's Gaussian-process-limit interpretation predicts that increasing the number of independent trees should approximate a stable kernel while avoiding MCMC and difficult optimization over discrete split structures.
Useful6/10
Difficulty4/10
Novelty7/10