XGBoost

XGboost is an end-to-end boosting system. It is sparsity-aware. (Chen and Guestrin, n.d.)

Regularized Learning Objective

For a given data set of $n$ examples and $m$ features, a tree ensemble model uses $K$ additive functions to predict the output:

${\hat{y}}_{i} = ϕ (x_{i}) = \sum_{k = 1}^{K} f_{k} (x_{i}), f_{k} \in F$

where $F$ is the space of regression trees, Each regression tree contains a continuous score on each of the leaf.

source: https://books.google.com.sg/books/about/Classification%5Fand%5FRegression%5FTrees.html?id=JwQx-WOmSyQC&redir%5Fesc=y

We wish to minimise the regularized objective:

$L (ϕ) = \sum_{i} l ({\hat{y}}_{i}, y_{i}) + \sum_{k} Ω (f_{k})$

where

$Ω (f) = γ T + \frac{1}{2} λ ‖ w ‖^{2}$

$T$ being the number of leaves in the tree, and $w_{i}$ being the ith weight of the leaf.

Gradient Tree Boosting

Since the regularized objective contains functions, the objective cannot be optimized in the Euclidean space.

Let ${\hat{y}}_{i}^{t}$ be the prediction of the ith instance on the $t$-th iteration. We need to add $f_{t}$ to minimize the following objective:

$L^{(t)} = \sum_{i = 1}^{n} l (y_{i}, {\hat{y}}_{i}^{(t - 1)} + f_{t} (x_{i})) + Ω (f_{t})$

We can perform a Taylor expansion, and obtain:

$L^{(t)} = \sum_{i = 1}^{n} l (y_{i}, {\hat{y}}_{i}^{(t - 1)}) + g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{t}^{2} (x_{i}) + Ω (f_{t})$

where $g_{i} = \partial_{{\hat{y}}^{(t - 1)}} l (y_{i}, {\hat{y}}^{(t - 1)})$ , and $h_{i} = \partial_{{\hat{y}}^{(t - 1)}}^{2} l (y_{i}, {\hat{y}}^{(t - 1)})$ are the first and second order gradient statistics (gradients and Hessians). Removing the constant terms, at step $t$ , the objective is simplified to:

${\tilde{L}}^{(t)} = \sum_{i = 1}^{n} g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{t}^{2} (x_{i}) + Ω (f_{t})$

If we define $I_{j} = {i | q (x_{i}) = j}$ as the instance set of leaf $j$ , then we can rewrite:

${\tilde{L}}^{(t)} = \sum_{j = 1}^{T} [(\sum_{i \in I_{j}} g_{i}) w_{j} + \frac{1}{2} (\sum_{i \in I_{j}} h_{i} + λ) w_{j}^{2}] + γ T$

and we find that for a fixed structure $q (x)$ , we can compute the optimal weight $w_{j}^{*}$ of leaf $j$ :

$w_{j}^{*} = - \frac{\sum_{i \in I_{j}} g_{i}}{\sum_{i \in I_{j}} h_{i} + λ}$

This can then be used to score a tree structure $q$ . However, since enumerating all possible trees is computationally expensive, we use an algorithm to iteratively add branches to the tree, by computing the loss reduction from splitting:

$L_{split} = \frac{1}{2} [\frac{{(\sum_{i \in I_{L}} g_{i})}^{2}}{\sum_{i \in I_{L}} h_{i} + λ} + \frac{{(\sum_{i \in I_{R}} g_{i})}^{2}}{\sum_{i \in I_{R}} h_{i} + λ} - \frac{{(\sum_{i \in I} g_{i})}^{2}}{\sum_{i \in I} h_{i} + λ}]$

Implementing Distributed XGBoost

The implementation of distributed XGBoost uses RABIT, and the Allreduce framework. XGBoost requires gradients and hessians from each distributed worker. This fit the allreduce framework, which broadcasts each worker’s reduce result across all processes. (Chen, Cano, and Zhou, n.d.)

Bibliography

Chen, Tianqi, and Carlos Guestrin. n.d. “Xgboost: A Scalable Tree Boosting System.” http://arxiv.org/abs/1603.02754v3.

Chen, Tianqi, Ignacio Cano, and Tianyi Zhou. n.d. “RABIT: A Reliable Allreduce and Broadcast Interface” 3 (2).