We’ve always been told that decision trees are the best base learners for Gradient Boosting Machine Learning. I’ve always wanted to see for myself. AdaBoostClassifier is working well, but is relatively slow (by my own standards). A few days ago, I noticed that my Cython implementation of LSBoost in Python package mlsauce was already quite generic (never noticed before), and I decided to adapt it to any machine learning model with fit and predict methods. It’s worth mentioning that only regression algorithms are accepted as base learners, and classification is regression-based. The results are promising indeed; I’ll let you see for yourself below, for regression and classification. All the algorithms, including xgboost and RandomForest, are used with their default hyperparameters. Which means, there’s still a room for improvement. The notebook is here.

Install mlsauce (version 0.20.3) from GitHub:

!pip install git+https://github.com/Techtonique/mlsauce.git --verbose --upgrade --no-cache-dir

1 - Classification

import os
import pandas as pd
import mlsauce as ms
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from time import time

load_models = [load_breast_cancer, load_wine, load_iris]

for model in load_models:

    data = model()
    X = data.data
    y= data.target
    X = pd.DataFrame(X, columns=data.feature_names)

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 13)

    clf = ms.LazyBoostingClassifier(verbose=0, ignore_warnings=True,
                                    custom_metric=None, preprocess=False)

    start = time()
    models, predictions = clf.fit(X_train, X_test, y_train, y_test)
    print(f"\nElapsed: {time() - start} seconds\n")

    display(models)


2it [00:01,  1.52it/s]
100%|██████████| 30/30 [00:21<00:00,  1.38it/s]


Elapsed: 23.019137859344482 seconds
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
GenericBooster(LinearRegression) 0.99 0.99 0.99 0.99 0.35
GenericBooster(Ridge) 0.99 0.99 0.99 0.99 0.27
GenericBooster(RidgeCV) 0.99 0.99 0.99 0.99 1.07
GenericBooster(TransformedTargetRegressor) 0.99 0.99 0.99 0.99 0.40
GenericBooster(KernelRidge) 0.97 0.96 0.96 0.97 2.05
XGBClassifier 0.96 0.96 0.96 0.96 0.91
GenericBooster(ExtraTreeRegressor) 0.94 0.94 0.94 0.94 0.25
RandomForestClassifier 0.92 0.93 0.93 0.92 0.40
GenericBooster(RANSACRegressor) 0.90 0.86 0.86 0.90 15.22
GenericBooster(DecisionTreeRegressor) 0.87 0.88 0.88 0.87 0.98
GenericBooster(KNeighborsRegressor) 0.87 0.89 0.89 0.87 0.49
GenericBooster(ElasticNet) 0.85 0.76 0.76 0.84 0.10
GenericBooster(Lasso) 0.82 0.71 0.71 0.79 0.09
GenericBooster(LassoLars) 0.82 0.71 0.71 0.79 0.10
GenericBooster(DummyRegressor) 0.68 0.50 0.50 0.56 0.01 
2it [00:00,  8.29it/s]
100%|██████████| 30/30 [00:15<00:00,  1.92it/s]


Elapsed: 15.911818265914917 seconds
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
RandomForestClassifier 1.00 1.00 None 1.00 0.18
GenericBooster(ExtraTreeRegressor) 1.00 1.00 None 1.00 0.16
GenericBooster(KernelRidge) 1.00 1.00 None 1.00 0.38
GenericBooster(LinearRegression) 1.00 1.00 None 1.00 0.23
GenericBooster(Ridge) 1.00 1.00 None 1.00 0.17
GenericBooster(RidgeCV) 1.00 1.00 None 1.00 0.24
GenericBooster(TransformedTargetRegressor) 1.00 1.00 None 1.00 0.26
XGBClassifier 0.97 0.96 None 0.97 0.06
GenericBooster(Lars) 0.94 0.94 None 0.95 0.99
GenericBooster(DecisionTreeRegressor) 0.92 0.92 None 0.92 0.23
GenericBooster(KNeighborsRegressor) 0.92 0.93 None 0.92 0.21
GenericBooster(RANSACRegressor) 0.81 0.81 None 0.80 12.63
GenericBooster(ElasticNet) 0.61 0.53 None 0.53 0.04
GenericBooster(DummyRegressor) 0.42 0.33 None 0.25 0.01
GenericBooster(Lasso) 0.42 0.33 None 0.25 0.02
GenericBooster(LassoLars) 0.42 0.33 None 0.25 0.01 
2it [00:00,  5.14it/s]
100%|██████████| 30/30 [00:15<00:00,  1.92it/s]


Elapsed: 16.0275661945343 seconds
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
GenericBooster(Ridge) 1.00 1.00 None 1.00 0.23
GenericBooster(RidgeCV) 1.00 1.00 None 1.00 0.25
RandomForestClassifier 0.97 0.97 None 0.97 0.26
XGBClassifier 0.97 0.97 None 0.97 0.12
GenericBooster(DecisionTreeRegressor) 0.97 0.97 None 0.97 0.27
GenericBooster(ExtraTreeRegressor) 0.97 0.97 None 0.97 0.22
GenericBooster(LinearRegression) 0.97 0.97 None 0.97 0.15
GenericBooster(TransformedTargetRegressor) 0.97 0.97 None 0.97 0.37
GenericBooster(KNeighborsRegressor) 0.93 0.95 None 0.93 1.52
GenericBooster(KernelRidge) 0.87 0.83 None 0.85 0.63
GenericBooster(RANSACRegressor) 0.63 0.59 None 0.61 10.86
GenericBooster(Lars) 0.50 0.46 None 0.48 0.99
GenericBooster(DummyRegressor) 0.27 0.33 None 0.11 0.01
GenericBooster(ElasticNet) 0.27 0.33 None 0.11 0.01
GenericBooster(Lasso) 0.27 0.33 None 0.11 0.01
GenericBooster(LassoLars) 0.27 0.33 None 0.11 0.01 
!pip install shap

import shap

best_model = clf.get_best_model()

# load JS visualization code to notebook
shap.initjs()

# explain all the predictions in the test set
explainer = shap.KernelExplainer(best_model.predict_proba, X_train)
shap_values = explainer.shap_values(X_test)
# this is multiclass so we only visualize the contributions to first class (hence index 0)
shap.force_plot(explainer.expected_value[0], shap_values[..., 0], X_test)

xxx

2 - Regression

import os
import mlsauce as ms
from sklearn.datasets import load_diabetes
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split


data = load_diabetes()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

regr = ms.LazyBoostingRegressor(verbose=0, ignore_warnings=True,
                                custom_metric=None, preprocess=True)
models, predictioms = regr.fit(X_train, X_test, y_train, y_test)
model_dictionary = regr.provide_models(X_train, X_test, y_train, y_test)
display(models)

3it [00:00,  4.75it/s]
100%|██████████| 30/30 [00:58<00:00,  1.95s/it]
Adjusted R-Squared R-Squared RMSE Time Taken
Model
GenericBooster(HuberRegressor) 0.55 0.60 50.13 3.73
GenericBooster(SGDRegressor) 0.55 0.60 50.40 0.36
GenericBooster(RidgeCV) 0.54 0.59 50.53 0.40
GenericBooster(LinearSVR) 0.54 0.59 50.54 0.18
GenericBooster(PassiveAggressiveRegressor) 0.54 0.59 50.63 0.30
GenericBooster(Ridge) 0.54 0.59 50.70 0.31
GenericBooster(TransformedTargetRegressor) 0.54 0.59 50.75 0.46
GenericBooster(LinearRegression) 0.54 0.59 50.75 0.39
GenericBooster(KernelRidge) 0.53 0.59 50.99 3.09
GenericBooster(TweedieRegressor) 0.53 0.59 51.10 0.66
GenericBooster(LassoLars) 0.53 0.58 51.17 0.44
GenericBooster(Lasso) 0.53 0.58 51.17 0.20
GenericBooster(ElasticNet) 0.53 0.58 51.24 0.31
GenericBooster(SVR) 0.52 0.57 51.97 3.54
GenericBooster(BayesianRidge) 0.50 0.56 52.93 0.97
GenericBooster(LassoLarsIC) 0.49 0.55 53.20 0.39
GradientBoostingRegressor 0.49 0.55 53.23 0.14
GenericBooster(ElasticNetCV) 0.49 0.55 53.43 3.73
GenericBooster(LassoLarsCV) 0.49 0.55 53.44 1.23
GenericBooster(LassoCV) 0.49 0.55 53.45 4.01
GenericBooster(LarsCV) 0.49 0.54 53.54 0.90
GenericBooster(NuSVR) 0.46 0.53 54.67 2.39
RandomForestRegressor 0.46 0.52 55.16 0.36
GenericBooster(RANSACRegressor) 0.44 0.50 56.14 23.45
GenericBooster(ExtraTreeRegressor) 0.41 0.47 57.52 0.78
XGBRegressor 0.31 0.39 61.96 0.13
GenericBooster(DecisionTreeRegressor) 0.28 0.36 63.57 1.06
GenericBooster(Lars) 0.19 0.28 67.43 0.73
GenericBooster(DummyRegressor) -0.13 -0.00 79.39 0.01
GenericBooster(QuantileRegressor) -0.15 -0.02 80.00 3.37
GenericBooster(KNeighborsRegressor) -7.86 -6.85 222.42 1.14 
data = fetch_california_housing()
n_points = 1000
idx_inputs = range(n_points)
X = data.data[idx_inputs,:]
y= data.target[idx_inputs]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

regr = ms.LazyBoostingRegressor(verbose=0, ignore_warnings=True,
                                custom_metric=None, preprocess=True)
models, predictioms = regr.fit(X_train, X_test, y_train, y_test)
model_dictionary = regr.provide_models(X_train, X_test, y_train, y_test)
display(models)

3it [00:03,  1.01s/it]
100%|██████████| 30/30 [02:32<00:00,  5.10s/it]
Adjusted R-Squared R-Squared RMSE Time Taken
Model
GenericBooster(ExtraTreeRegressor) 0.82 0.83 0.34 0.93
RandomForestRegressor 0.82 0.83 0.34 1.27
GradientBoostingRegressor 0.79 0.79 0.37 0.28
GenericBooster(NuSVR) 0.79 0.79 0.37 18.97
GenericBooster(SVR) 0.78 0.79 0.38 15.78
XGBRegressor 0.78 0.79 0.38 1.48
GenericBooster(HuberRegressor) 0.77 0.78 0.39 5.49
GenericBooster(LinearSVR) 0.77 0.77 0.39 7.15
GenericBooster(TransformedTargetRegressor) 0.75 0.76 0.40 3.12
GenericBooster(LinearRegression) 0.75 0.76 0.40 1.94
GenericBooster(Ridge) 0.75 0.76 0.40 0.48
GenericBooster(RANSACRegressor) 0.75 0.76 0.41 32.76
GenericBooster(RidgeCV) 0.75 0.76 0.41 2.54
GenericBooster(PassiveAggressiveRegressor) 0.74 0.75 0.41 0.55
GenericBooster(SGDRegressor) 0.73 0.74 0.42 0.66
GenericBooster(DecisionTreeRegressor) 0.73 0.74 0.42 2.48
GenericBooster(KernelRidge) 0.71 0.72 0.43 13.27
GenericBooster(LassoLarsIC) 0.71 0.72 0.44 1.33
GenericBooster(BayesianRidge) 0.71 0.72 0.44 2.82
GenericBooster(LassoLarsCV) 0.70 0.71 0.44 2.51
GenericBooster(LassoCV) 0.70 0.71 0.44 9.69
GenericBooster(ElasticNetCV) 0.70 0.71 0.44 10.05
GenericBooster(TweedieRegressor) 0.69 0.71 0.45 1.88
GenericBooster(LarsCV) 0.69 0.70 0.45 1.65
GenericBooster(Lars) 0.42 0.44 0.62 1.01
GenericBooster(ElasticNet) 0.25 0.28 0.70 0.27
GenericBooster(QuantileRegressor) -0.04 -0.00 0.83 10.72
GenericBooster(Lasso) -0.08 -0.04 0.84 0.02
GenericBooster(DummyRegressor) -0.08 -0.04 0.84 0.02
GenericBooster(LassoLars) -0.08 -0.04 0.84 0.03
GenericBooster(KNeighborsRegressor) -0.46 -0.40 0.98 4.75

Comments powered by Talkyard.