T. Moudiki's Webpage - nnetsauce

Gradient-Boosting anything (alert: high performance): Part4, Time series forecasting

2024-11-10T00:00:00+00:00

A few weeks ago, I introduced a model-agnostic gradient boosting (XGBoost, LightGBM, CatBoost-like) procedure for supervised regression and classification, that can use any base learner (available in R and Python package mlsauce). You can find the previous posts here:

LightGBM is widely used in the context of time series forecasting (see e.g for M5 forecasting competition and VN1 competition), and is based on decision trees. However, it’s possible to use many other base learners, such as ridge regression, kernel ridge regression, etc. In this post, I will show how to use mlsauce version 0.24.0 for time series forecasting, with any base learner.

1 - Hold-out set cross-validation

Here, Generic Gradient Boosting is compared to popular models such as VAR and VECM (without hyperparameter tuning), on the macrodata dataset from the statsmodels package. The dataset is split into a training set (90% of the data) and a testing set (10% of the data), and model performance is evaluated using Root Mean Squared Error (RMSE) and Winkler score (uncertainty quantification). Uncertainty quantification uses conformal prediction and numerical simulation as described in my paper, and more details can be found in these slides.

import mlsauce as ms 
import numpy as np
import pandas as pd
import statsmodels.api as sm

try: 
    from statsmodels.tsa.base.datetools import dates_from_str
except ImportError:
    ModuleNotFoundError

# some example data
mdata = sm.datasets.macrodata.load_pandas().data
# prepare the dates index
dates = mdata[['year', 'quarter']].astype(int).astype(str)
quarterly = dates["year"] + "Q" + dates["quarter"]
quarterly = dates_from_str(quarterly)
mdata = mdata[['realgovt', 'tbilrate', 'cpi']]
mdata.index = pd.DatetimeIndex(quarterly)
data = np.log(mdata).diff().dropna()

n = data.shape[0]
max_idx_train = np.floor(n*0.9)
training_index = np.arange(0, max_idx_train)
testing_index = np.arange(max_idx_train, n)
df_train = data.iloc[training_index,:]
df_test = data.iloc[testing_index,:]


regr_mts = ms.LazyBoostingMTS(verbose=0, ignore_warnings=True, 
                      lags = 20, n_hidden_features=7, n_clusters=2,
                      type_pi="scp2-block-bootstrap", 
                      #kernel="gaussian",
                      replications=250, 
                      show_progress=False, preprocess=False, 
                      sort_by="WINKLERSCORE",)
models = regr_mts.fit(df_train, df_test)

print(models[["RMSE", "WINKLERSCORE", "Time Taken"]].iloc[0:25,:])

  0%|          | 0/30 [00:00<?, ?it/s]100%|██████████| 30/30 [00:13<00:00,  2.20it/s]

                                                 RMSE  WINKLERSCORE  \
Model                                                                 
MTS(GenericBooster(RidgeCV))                     0.32          1.71   
MTS(GenericBooster(PassiveAggressiveRegressor))  0.34          1.78   
MTS(GenericBooster(SGDRegressor))                0.32          1.87   
MTS(GenericBooster(Ridge))                       0.35          1.93   
MTS(GenericBooster(HuberRegressor))              0.37          1.95   
MTS(GenericBooster(ElasticNet))                  0.33          1.96   
MTS(GenericBooster(Lasso))                       0.33          1.96   
MTS(GenericBooster(DummyRegressor))              0.33          1.96   
MTS(GenericBooster(LassoLars))                   0.33          1.96   
MTS(GenericBooster(DecisionTreeRegressor))       0.33          1.97   
MTS(GenericBooster(QuantileRegressor))           0.33          1.98   
MTS(GenericBooster(LassoLarsIC))                 0.33          1.99   
MTS(GenericBooster(TweedieRegressor))            0.33          2.01   
MTS(GenericBooster(BayesianRidge))               0.33          2.01   
MTS(GenericBooster(LassoCV))                     0.33          2.02   
MTS(GenericBooster(LassoLarsCV))                 0.33          2.02   
MTS(GenericBooster(LarsCV))                      0.33          2.02   
MTS(GenericBooster(ElasticNetCV))                0.33          2.02   
MTS(GenericBooster(KNeighborsRegressor))         0.33          2.03   
MTS(GenericBooster(SVR))                         0.34          2.07   
MTS(GenericBooster(ExtraTreeRegressor))          0.33          2.19   
VAR                                              0.33          2.21   
VECM                                             0.34          2.39   
MTS(GenericBooster(LinearSVR))                   0.45          3.03   
MTS(GenericBooster(TransformedTargetRegressor))  0.49          3.04   

                                                 Time Taken  
Model                                                        
MTS(GenericBooster(RidgeCV))                           0.13  
MTS(GenericBooster(PassiveAggressiveRegressor))        0.29  
MTS(GenericBooster(SGDRegressor))                      0.09  
MTS(GenericBooster(Ridge))                             0.09  
MTS(GenericBooster(HuberRegressor))                    0.18  
MTS(GenericBooster(ElasticNet))                        0.09  
MTS(GenericBooster(Lasso))                             0.09  
MTS(GenericBooster(DummyRegressor))                    0.08  
MTS(GenericBooster(LassoLars))                         0.09  
MTS(GenericBooster(DecisionTreeRegressor))             0.11  
MTS(GenericBooster(QuantileRegressor))                 0.15  
MTS(GenericBooster(LassoLarsIC))                       0.30  
MTS(GenericBooster(TweedieRegressor))                  0.09  
MTS(GenericBooster(BayesianRidge))                     0.14  
MTS(GenericBooster(LassoCV))                           5.01  
MTS(GenericBooster(LassoLarsCV))                       0.54  
MTS(GenericBooster(LarsCV))                            0.46  
MTS(GenericBooster(ElasticNetCV))                      4.74  
MTS(GenericBooster(KNeighborsRegressor))               0.10  
MTS(GenericBooster(SVR))                               0.09  
MTS(GenericBooster(ExtraTreeRegressor))                0.10  
VAR                                                    0.01  
VECM                                                   0.01  
MTS(GenericBooster(LinearSVR))                         0.16  
MTS(GenericBooster(TransformedTargetRegressor))        0.13  

2 - Individual examples

Here, I will show how to use mlsauce for time series forecasting, with Ridge regression and Kernel Ridge regression as base learners. 250 conformal predictive simulations are obtained, and the prediction is made for the next 20 periods.

import nnetsauce as ns

regr_ridge = ms.GenericBoostingRegressor(ms.RidgeRegressor(reg_lambda=1e3))
regr_krr = ms.GenericBoostingRegressor(ms.KRLSRegressor())

regr_mts = ns.MTS(regr_ridge, lags=20, replications=250,
                  type_pi="scp2-block-bootstrap")
regr_mts.fit(df_train)
regr_mts.predict(h=20)
regr_mts.plot('tbilrate')

100%|██████████| 46/46 [00:00<00:00, 1612.53it/s]
100%|██████████| 46/46 [00:00<00:00, 1678.79it/s]
100%|██████████| 46/46 [00:00<00:00, 1663.07it/s]
100%|██████████| 46/46 [00:00<00:00, 1298.61it/s]
100%|██████████| 46/46 [00:00<00:00, 1410.21it/s]
100%|██████████| 46/46 [00:00<00:00, 1676.63it/s]
100%|██████████| 3/3 [00:00<00:00, 13.82it/s]

regr_mts = ns.MTS(regr_krr, lags=20, replications=250,
                  type_pi="scp2-block-bootstrap")
regr_mts.fit(df_train)
regr_mts.predict(h=20)
regr_mts.plot('tbilrate')

100%|██████████| 34/34 [00:01<00:00, 19.22it/s]
100%|██████████| 34/34 [00:01<00:00, 18.17it/s]
100%|██████████| 34/34 [00:01<00:00, 19.39it/s]
100%|██████████| 34/34 [00:01<00:00, 19.90it/s]
100%|██████████| 34/34 [00:01<00:00, 19.76it/s]
100%|██████████| 34/34 [00:01<00:00, 17.15it/s]
100%|██████████| 3/3 [00:14<00:00,  4.76s/it]

Gradient-Boosting anything (alert: high performance): Part3, Histogram-based boosting

2024-10-28T00:00:00+00:00

Update 2024-10-29: Fixed an error, and the histogram-based is actually failing miserably. Still trying to wrap my head around it (why is it not only failing, but so badly). The original implementation of the GenericBooster is still doing great as shown below.

A few weeks ago, I introduced a model-agnostic gradient boosting (XGBoost, LightGBM, CatBoost-like) procedure for supervised regression and classification, that can use any base learner (available in R and Python package mlsauce):

The rationale is different from other histogram-based gradient boosting algorithms, as histograms are only used here for feature engineering of continuous features. So far, I don’t see huge differences with the original implementation of the GenericBooster, but it’s still a work in progress. I envisage to try it out on a data set that contains a ‘higher’ mix of continuous and categorical features (as categorical features are not histogram-engineered).

Here are a few results that can give you an idea of the performance of the algorithm. Keep in mind that the models are not tuned, and that the GenericBooster can be tuned (in addition to the boosting model’s hyperparamters) with the base learner’s hyperparameters. That makes, potentially, a lot of degrees of freedom and room for improvement/exploration.

!pip install git+https://github.com/Techtonique/mlsauce.git --verbose --upgrade --no-cache-dir

import os
import mlsauce as ms
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from time import time

load_models = [load_breast_cancer, load_iris, load_wine, load_digits]

for model in load_models:

    data = model()
    X = data.data
    y= data.target

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 13)

    clf = ms.LazyBoostingClassifier(verbose=0, ignore_warnings=True, #n_jobs=2,
                                    custom_metric=None, preprocess=False)

    start = time()
    models, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=True)
    models2, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=False)
    print(f"\nElapsed: {time() - start} seconds\n")

    display(models)
    display(models2)

2it [00:00,  2.27it/s]
100%|██████████| 38/38 [00:41<00:00,  1.09s/it]
2it [00:00,  5.14it/s]
100%|██████████| 38/38 [00:43<00:00,  1.14s/it]


Elapsed: 85.95083284378052 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
GenericBooster(MultiTask(TweedieRegressor))	0.99	0.99	0.99	0.99	1.73
GenericBooster(LinearRegression)	0.99	0.99	0.99	0.99	0.37
GenericBooster(TransformedTargetRegressor)	0.99	0.99	0.99	0.99	0.40
GenericBooster(RidgeCV)	0.99	0.99	0.99	0.99	1.28
GenericBooster(Ridge)	0.99	0.99	0.99	0.99	0.27
XGBClassifier	0.96	0.96	0.96	0.96	0.50
RandomForestClassifier	0.96	0.96	0.96	0.96	0.37
GenericBooster(ExtraTreeRegressor)	0.94	0.94	0.94	0.94	0.40
GenericBooster(MultiTask(BayesianRidge))	0.94	0.93	0.93	0.94	4.97
GenericBooster(KNeighborsRegressor)	0.87	0.89	0.89	0.87	0.70
GenericBooster(DecisionTreeRegressor)	0.87	0.88	0.88	0.87	2.24
GenericBooster(MultiTaskElasticNet)	0.87	0.79	0.79	0.86	0.11
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.86	0.79	0.79	0.85	1.28
GenericBooster(MultiTaskLasso)	0.85	0.76	0.76	0.84	0.06
GenericBooster(ElasticNet)	0.85	0.76	0.76	0.84	0.16
GenericBooster(MultiTask(QuantileRegressor))	0.82	0.72	0.72	0.80	10.42
GenericBooster(Lasso)	0.82	0.71	0.71	0.79	0.09
GenericBooster(LassoLars)	0.82	0.71	0.71	0.79	0.08
GenericBooster(MultiTask(LinearSVR))	0.81	0.69	0.69	0.78	14.75
GenericBooster(DummyRegressor)	0.68	0.50	0.50	0.56	0.01
GenericBooster(MultiTask(SGDRegressor))	0.50	0.46	0.46	0.51	1.67

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
GenericBooster(MultiTask(TweedieRegressor))	0.99	0.99	0.99	0.99	1.67
GenericBooster(LinearRegression)	0.99	0.99	0.99	0.99	0.30
GenericBooster(TransformedTargetRegressor)	0.99	0.99	0.99	0.99	0.74
GenericBooster(RidgeCV)	0.99	0.99	0.99	0.99	2.77
GenericBooster(Ridge)	0.99	0.99	0.99	0.99	0.28
XGBClassifier	0.96	0.96	0.96	0.96	0.13
GenericBooster(MultiTask(BayesianRidge))	0.94	0.93	0.93	0.94	7.81
GenericBooster(ExtraTreeRegressor)	0.94	0.94	0.94	0.94	0.23
RandomForestClassifier	0.92	0.93	0.93	0.92	0.25
GenericBooster(KNeighborsRegressor)	0.87	0.89	0.89	0.87	0.42
GenericBooster(DecisionTreeRegressor)	0.87	0.88	0.88	0.87	0.97
GenericBooster(MultiTaskElasticNet)	0.87	0.79	0.79	0.86	0.11
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.86	0.79	0.79	0.85	1.20
GenericBooster(MultiTaskLasso)	0.85	0.76	0.76	0.84	0.06
GenericBooster(ElasticNet)	0.85	0.76	0.76	0.84	0.09
GenericBooster(MultiTask(QuantileRegressor))	0.82	0.72	0.72	0.80	10.57
GenericBooster(LassoLars)	0.82	0.71	0.71	0.79	0.09
GenericBooster(Lasso)	0.82	0.71	0.71	0.79	0.09
GenericBooster(MultiTask(LinearSVR))	0.81	0.69	0.69	0.78	14.20
GenericBooster(DummyRegressor)	0.68	0.50	0.50	0.56	0.01
GenericBooster(MultiTask(SGDRegressor))	0.50	0.46	0.46	0.51	1.33

2it [00:00,  6.46it/s]
100%|██████████| 38/38 [00:12<00:00,  3.11it/s]
2it [00:00, 10.38it/s]
100%|██████████| 38/38 [00:11<00:00,  3.18it/s]


Elapsed: 24.71835470199585 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
GenericBooster(RidgeCV)	1.00	1.00	None	1.00	0.18
GenericBooster(Ridge)	1.00	1.00	None	1.00	0.14
GenericBooster(LinearRegression)	0.97	0.97	None	0.97	0.13
GenericBooster(DecisionTreeRegressor)	0.97	0.97	None	0.97	0.18
GenericBooster(TransformedTargetRegressor)	0.97	0.97	None	0.97	0.23
GenericBooster(ExtraTreeRegressor)	0.97	0.97	None	0.97	0.14
XGBClassifier	0.97	0.97	None	0.97	0.05
RandomForestClassifier	0.93	0.95	None	0.93	0.26
GenericBooster(KNeighborsRegressor)	0.93	0.95	None	0.93	0.27
GenericBooster(MultiTask(SGDRegressor))	0.90	0.92	None	0.90	0.75
GenericBooster(MultiTask(TweedieRegressor))	0.90	0.92	None	0.90	1.61
GenericBooster(MultiTask(LinearSVR))	0.80	0.85	None	0.80	2.15
GenericBooster(MultiTaskElasticNet)	0.80	0.85	None	0.80	0.07
GenericBooster(MultiTask(BayesianRidge))	0.63	0.72	None	0.57	2.42
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.57	0.67	None	0.45	1.05
GenericBooster(Lars)	0.50	0.46	None	0.48	0.59
GenericBooster(MultiTask(QuantileRegressor))	0.43	0.33	None	0.26	2.19
GenericBooster(LassoLars)	0.27	0.33	None	0.11	0.01
GenericBooster(MultiTaskLasso)	0.27	0.33	None	0.11	0.01
GenericBooster(Lasso)	0.27	0.33	None	0.11	0.01
GenericBooster(ElasticNet)	0.27	0.33	None	0.11	0.01
GenericBooster(DummyRegressor)	0.27	0.33	None	0.11	0.01

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
GenericBooster(RidgeCV)	1.00	1.00	None	1.00	0.16
GenericBooster(Ridge)	1.00	1.00	None	1.00	0.16
RandomForestClassifier	0.97	0.97	None	0.97	0.15
GenericBooster(LinearRegression)	0.97	0.97	None	0.97	0.13
GenericBooster(DecisionTreeRegressor)	0.97	0.97	None	0.97	0.16
GenericBooster(TransformedTargetRegressor)	0.97	0.97	None	0.97	0.24
GenericBooster(ExtraTreeRegressor)	0.97	0.97	None	0.97	0.14
XGBClassifier	0.97	0.97	None	0.97	0.04
GenericBooster(KNeighborsRegressor)	0.93	0.95	None	0.93	0.28
GenericBooster(MultiTask(SGDRegressor))	0.90	0.92	None	0.90	0.78
GenericBooster(MultiTask(TweedieRegressor))	0.90	0.92	None	0.90	1.35
GenericBooster(MultiTask(LinearSVR))	0.80	0.85	None	0.80	2.15
GenericBooster(MultiTaskElasticNet)	0.80	0.85	None	0.80	0.07
GenericBooster(MultiTask(BayesianRidge))	0.63	0.72	None	0.57	1.81
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.57	0.67	None	0.45	1.21
GenericBooster(Lars)	0.50	0.46	None	0.48	0.58
GenericBooster(MultiTask(QuantileRegressor))	0.43	0.33	None	0.26	2.63
GenericBooster(LassoLars)	0.27	0.33	None	0.11	0.01
GenericBooster(MultiTaskLasso)	0.27	0.33	None	0.11	0.01
GenericBooster(Lasso)	0.27	0.33	None	0.11	0.01
GenericBooster(ElasticNet)	0.27	0.33	None	0.11	0.02
GenericBooster(DummyRegressor)	0.27	0.33	None	0.11	0.01

2it [00:00,  5.45it/s]
100%|██████████| 38/38 [00:14<00:00,  2.63it/s]
2it [00:00,  9.26it/s]
100%|██████████| 38/38 [00:14<00:00,  2.58it/s]


Elapsed: 29.76035761833191 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
RandomForestClassifier	1.00	1.00	None	1.00	0.30
GenericBooster(ExtraTreeRegressor)	1.00	1.00	None	1.00	0.17
GenericBooster(TransformedTargetRegressor)	1.00	1.00	None	1.00	0.26
GenericBooster(RidgeCV)	1.00	1.00	None	1.00	0.23
GenericBooster(Ridge)	1.00	1.00	None	1.00	0.15
GenericBooster(LinearRegression)	1.00	1.00	None	1.00	0.15
XGBClassifier	0.97	0.96	None	0.97	0.06
GenericBooster(MultiTask(SGDRegressor))	0.97	0.98	None	0.97	1.10
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.97	0.98	None	0.97	1.18
GenericBooster(MultiTask(LinearSVR))	0.97	0.98	None	0.97	3.71
GenericBooster(MultiTask(BayesianRidge))	0.97	0.98	None	0.97	1.86
GenericBooster(MultiTask(TweedieRegressor))	0.97	0.98	None	0.97	1.39
GenericBooster(Lars)	0.94	0.94	None	0.95	0.93
GenericBooster(KNeighborsRegressor)	0.92	0.93	None	0.92	0.19
GenericBooster(DecisionTreeRegressor)	0.92	0.92	None	0.92	0.22
GenericBooster(MultiTaskElasticNet)	0.69	0.61	None	0.61	0.03
GenericBooster(ElasticNet)	0.61	0.53	None	0.53	0.05
GenericBooster(MultiTaskLasso)	0.42	0.33	None	0.25	0.01
GenericBooster(LassoLars)	0.42	0.33	None	0.25	0.01
GenericBooster(Lasso)	0.42	0.33	None	0.25	0.01
GenericBooster(DummyRegressor)	0.42	0.33	None	0.25	0.01
GenericBooster(MultiTask(QuantileRegressor))	0.25	0.33	None	0.10	2.73

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
RandomForestClassifier	1.00	1.00	None	1.00	0.15
GenericBooster(ExtraTreeRegressor)	1.00	1.00	None	1.00	0.16
GenericBooster(TransformedTargetRegressor)	1.00	1.00	None	1.00	0.24
GenericBooster(RidgeCV)	1.00	1.00	None	1.00	0.22
GenericBooster(Ridge)	1.00	1.00	None	1.00	0.16
GenericBooster(LinearRegression)	1.00	1.00	None	1.00	0.15
XGBClassifier	0.97	0.96	None	0.97	0.06
GenericBooster(MultiTask(SGDRegressor))	0.97	0.98	None	0.97	0.84
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.97	0.98	None	0.97	1.18
GenericBooster(MultiTask(LinearSVR))	0.97	0.98	None	0.97	3.41
GenericBooster(MultiTask(BayesianRidge))	0.97	0.98	None	0.97	2.15
GenericBooster(MultiTask(TweedieRegressor))	0.97	0.98	None	0.97	1.91
GenericBooster(Lars)	0.94	0.94	None	0.95	0.93
GenericBooster(KNeighborsRegressor)	0.92	0.93	None	0.92	0.20
GenericBooster(DecisionTreeRegressor)	0.92	0.92	None	0.92	0.23
GenericBooster(MultiTaskElasticNet)	0.69	0.61	None	0.61	0.03
GenericBooster(ElasticNet)	0.61	0.53	None	0.53	0.04
GenericBooster(MultiTaskLasso)	0.42	0.33	None	0.25	0.01
GenericBooster(LassoLars)	0.42	0.33	None	0.25	0.01
GenericBooster(Lasso)	0.42	0.33	None	0.25	0.01
GenericBooster(DummyRegressor)	0.42	0.33	None	0.25	0.01
GenericBooster(MultiTask(QuantileRegressor))	0.25	0.33	None	0.10	2.78

2it [00:01,  1.90it/s]
100%|██████████| 38/38 [09:30<00:00, 15.02s/it]
2it [00:01,  1.03it/s]
100%|██████████| 38/38 [09:27<00:00, 14.94s/it]


Elapsed: 1141.7054164409637 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
RandomForestClassifier	0.97	0.97	None	0.97	0.56
XGBClassifier	0.97	0.97	None	0.97	0.50
GenericBooster(ExtraTreeRegressor)	0.96	0.96	None	0.96	1.75
GenericBooster(KNeighborsRegressor)	0.95	0.95	None	0.95	4.34
GenericBooster(LinearRegression)	0.94	0.94	None	0.94	4.47
GenericBooster(MultiTask(BayesianRidge))	0.94	0.94	None	0.94	51.97
GenericBooster(TransformedTargetRegressor)	0.94	0.94	None	0.94	2.54
GenericBooster(RidgeCV)	0.94	0.94	None	0.94	4.55
GenericBooster(Ridge)	0.94	0.94	None	0.94	0.63
GenericBooster(MultiTask(TweedieRegressor))	0.93	0.93	None	0.93	13.86
GenericBooster(DecisionTreeRegressor)	0.88	0.88	None	0.88	6.14
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.79	0.79	None	0.80	13.46
GenericBooster(MultiTask(LinearSVR))	0.37	0.39	None	0.26	297.07
GenericBooster(Lars)	0.20	0.20	None	0.21	19.23
GenericBooster(MultiTask(QuantileRegressor))	0.12	0.10	None	0.03	140.91
GenericBooster(MultiTask(SGDRegressor))	0.10	0.10	None	0.06	9.46
GenericBooster(LassoLars)	0.07	0.10	None	0.01	0.05
GenericBooster(Lasso)	0.07	0.10	None	0.01	0.07
GenericBooster(MultiTaskLasso)	0.07	0.10	None	0.01	0.04
GenericBooster(ElasticNet)	0.07	0.10	None	0.01	0.03
GenericBooster(DummyRegressor)	0.07	0.10	None	0.01	0.02
GenericBooster(MultiTaskElasticNet)	0.07	0.10	None	0.01	0.05

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
RandomForestClassifier	0.97	0.97	None	0.97	0.67
XGBClassifier	0.97	0.97	None	0.97	1.27
GenericBooster(ExtraTreeRegressor)	0.96	0.96	None	0.96	1.69
GenericBooster(KNeighborsRegressor)	0.95	0.95	None	0.95	4.76
GenericBooster(LinearRegression)	0.94	0.94	None	0.94	2.01
GenericBooster(MultiTask(BayesianRidge))	0.94	0.94	None	0.94	46.87
GenericBooster(TransformedTargetRegressor)	0.94	0.94	None	0.94	5.40
GenericBooster(RidgeCV)	0.94	0.94	None	0.94	3.93
GenericBooster(Ridge)	0.94	0.94	None	0.94	0.60
GenericBooster(MultiTask(TweedieRegressor))	0.93	0.93	None	0.93	14.96
GenericBooster(DecisionTreeRegressor)	0.88	0.88	None	0.88	4.12
GenericBooster(MultiTask(PassiveAggressiveRegressor))	0.79	0.79	None	0.80	12.68
GenericBooster(MultiTask(LinearSVR))	0.37	0.39	None	0.26	294.88
GenericBooster(Lars)	0.20	0.20	None	0.21	19.40
GenericBooster(MultiTask(QuantileRegressor))	0.12	0.10	None	0.03	145.91
GenericBooster(MultiTask(SGDRegressor))	0.10	0.10	None	0.06	10.30
GenericBooster(LassoLars)	0.07	0.10	None	0.01	0.02
GenericBooster(Lasso)	0.07	0.10	None	0.01	0.03
GenericBooster(MultiTaskLasso)	0.07	0.10	None	0.01	0.03
GenericBooster(ElasticNet)	0.07	0.10	None	0.01	0.03
GenericBooster(DummyRegressor)	0.07	0.10	None	0.01	0.02
GenericBooster(MultiTaskElasticNet)	0.07	0.10	None	0.01	0.03

My presentation at ISF 2024 conference (slides with nnetsauce probabilistic forecasting news)

2024-07-03T00:00:00+00:00

Here are thes slides of my presentation at the International Symposium on Forecasting today:

https://www.researchgate.net/publication/381957724_Probabilistic_Forecasting_with_RandomizedQuasi-Randomized_networks_presentation_at_the_International_Symposium_on_Forecasting_2024

I discussed probabilistic Forecasting with quasi-randomized networks, sequential split conformal prediction, and GPU computation in Python package nnetsauce (thanks to JAX). Bonus slides: Automated Forecasting (sort of) with nnetsauce’s LazyMTS and time series cross-validation with nnetsauce’s TimeSeriesSplit.

Thanks to the organizers for allowing me to present my independent work on nnetsauce. These conferences are always a great opportunity to meet new people (or people we’ve heard about for a long time), and share our passion.

10 uncertainty quantification methods in nnetsauce forecasting

2024-07-01T00:00:00+00:00

This week, I released (Python) version 0.22.4 of nnetsauce. nnetsauce now contains 10 uncertainty quantification methods for time series forecasting:

gaussian: simple, fast, but: assumes stationarity of Gaussian in-sample residuals and independence in the multivariate case
kde: based on Kernel Density Estimation of in-sample residuals
bootstrap: based on independent bootstrap of in-sample residuals
block-bootstrap: based on moving block bootstrap of in-sample residuals
scp-kde: Split conformal prediction with Kernel Density Estimation of calibrated residuals
scp-bootstrap: Split conformal prediction with independent bootstrap of calibrated residuals
scp-block-bootstrap: Split conformal prediction with moving block bootstrap of calibrated residuals
scp2-kde: Split conformal prediction with Kernel Density Estimation of standardized calibrated residuals
scp2-bootstrap: Split conformal prediction with independent bootstrap of standardized calibrated residuals
scp2-block-bootstrap: Split conformal prediction with moving block bootstrap of standardized calibrated residuals

The release is available on Github, Conda and PyPI.

I’ll present nnetsauce and these methods with more details and examples at the 44th International Symposium on Forecasting (ISF) (ISF) 2024 (on Wednesday). I hope to see you there, with your questions, remarks and suggestions.

Next week, I’ll release a stable (and documented) version of learningmachine.

Forecasting with XGBoost embedded in Quasi-Randomized Neural Networks

2024-06-24T00:00:00+00:00

Next week, I’ll present nnetsauce’s (univariate and multivariate probabilistic) time series forecasting capabilities at the 44th International Symposium on Forecasting (ISF) (ISF) 2024. ISF is the premier forecasting conference, attracting the world’s leading forecasting (I don’t only do forecasting though) researchers, practitioners, and students. I hope to see you there!

In this post, I illustrate how to obtain predictive simulations with nnetsauce’s MTS class using XGBoost as base learner, and I give some intuition behind the method "kde" employed for uncertainty quantification in this case. The time series used here is the USAccDeaths dataset, a univariate time series of monthly deaths from motor vehicle traffic in the United States from 1973 to 1978.

(Command line)

!pip install nnetsauce --upgrade --no-cache-dir

Import Python packages

import nnetsauce as ns
import numpy as np
import pandas as pd
import xgboost as xgb
import matplotlib.pyplot as plt
import seaborn as sns

# import data 
url = "https://raw.githubusercontent.com/Techtonique/datasets/main/time_series/univariate/USAccDeaths.csv"
df = pd.read_csv(url)
df.index = pd.DatetimeIndex(df.date)
df.drop(columns=['date'], inplace=True)

Adjusting XGBoost regressors with different number of estimators

# number of estimators for the base learner
n_estimators_list = [5, 20, 50, 100]
estimators = []
residuals = []

for n_estimators in n_estimators_list:
  # XGBoost regressor as base learner
  regr_xgb = ns.MTS(obj=xgb.XGBRegressor(n_estimators=n_estimators,
                                         learning_rate=0.1),
                    n_hidden_features=50,
                    replications=1000,
                    kernel='gaussian',
                    lags=25)
  regr_xgb.fit(df)
  # in sample residuals 
  residuals.append(regr_xgb.residuals_.ravel())
  # out-of-sample predictions
  regr_xgb.predict(h=30)
  estimators.append(regr_xgb)

residuals_df = pd.DataFrame(np.asarray(residuals).T,
                            columns=["n5", "n20", "n50", "n100"])

100%|██████████| 1/1 [00:00<00:00,  4.55it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1797.56it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1583.30it/s]
100%|██████████| 1/1 [00:00<00:00,  2.30it/s]
100%|██████████| 1000/1000 [00:01<00:00, 582.86it/s]
100%|██████████| 1000/1000 [00:01<00:00, 882.61it/s]
100%|██████████| 1/1 [00:01<00:00,  1.83s/it]
100%|██████████| 1000/1000 [00:01<00:00, 808.35it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1555.30it/s]
100%|██████████| 1/1 [00:02<00:00,  2.37s/it]
100%|██████████| 1000/1000 [00:01<00:00, 578.83it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1436.02it/s]

sns.set_theme(style="darkgrid")

for est in estimators:
  est.plot(type_plot="spaghetti")

for i in range(4):
  sns.kdeplot(residuals_df.iloc[:,i], fill=True, color="red")
  plt.show()

In order to obtain predictive simulations with "kde" method (and as seen last week in #143), a Kernel Density Estimator (KDE) is adjusted to in-sample residuals. The most intuitive piece I found on KDEs is the following presentation: https://scholar.harvard.edu/files/montamat/files/nonparametric_estimation.pdf.

When using a high number of estimators (with the other parameters kept constant), the XGBRegressor base learner will overfit the training set, so that the in-sample residuals will be very small, and the uncertainty can’t be captured/estimated adequately: the predictions will consist of point forecasts. A compromise needs to be found by using cross-validation on the base learner’s hyperparameters, with an uncertainty quantification metric. Other types of predictions intervals/predictive simulation methods will be available in future versions of nnetsauce.

Forecasting Monthly Airline Passenger Numbers with Quasi-Randomized Neural Networks

2024-06-17T00:00:00+00:00

This post is about forecasting airline passenger numbers with quasi-randomized neural networks, and most specifically using nnetsauce’s class MTS. MTS stands for ‘Multivariate Time Series’, but MTS can also be used for univariate time series as shown in this post.

The data used here is the famous AirPassengers dataset, a time series of monthly totals of international airline passengers from 1949 to 1960. AirPassengers has a total number of 144 observations, an upward trend, and a seasonal component. It also has a time-varying volatility, which makes it a bit more challenging and interesting to forecast.

0 - Install and load packages

Install nnetsauce at the command line:

pip install nnetsauce --upgrade --no-cache-dir

Python code for loading the packages:

import nnetsauce as ns
import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge

1 - Load `AirPassengers` data

url = "https://raw.githubusercontent.com/Techtonique/datasets/main/time_series/univariate/AirPassengers.csv"
df = pd.read_csv(url)
df.index = pd.DatetimeIndex(df.date)
df.drop(columns=['date'], inplace=True)
df.plot()

2 - Train a quasi-randomized neural network based on Ridge regression

The quasi-randomized neural network used here relies on a Ridge Regression model, and has 5 hidden nodes by default (that makes it a nonlinear model). The replications parameter below is the number of predictive simulations. The kernel parameter set to ‘gaussian’ means that a Gaussian kernel is used for simulating from the residuals’ density. The lags parameter is set to 15, which means that the quasi-randomized neural network model is run on the 15 previous values of the time series to predict the next value.

regr = ns.MTS(obj=Ridge(),
              replications=250,
              kernel='gaussian',
              lags=15)
regr.fit(df)              

3 - Forecasting 40 steps ahead

regr.predict(h=40)

regr.plot(type_plot="pi")

regr.plot(type_plot="spaghetti")

The prediction interval is a bit narrow on this data set, but the model captures the trend, seasonality and time-varying volatility quite well. Do not hesitate to try the model on other time series data sets, and tune the hyperparameters of the model. Another example of univariate time series can be found in the link appearing at the top of this post. Other types of predictions intervals will be available in future versions of nnetsauce.

Automated hyperparameter tuning using any conformalized surrogate

2024-06-09T00:00:00+00:00

Bayesian optimization(BO) is widely used for Machine Learning hyperparameter tuning. BO relies mainly on a probabilistic model of the objective function (generally a Gaussian process model that approximates the objective function) called the surrogate and improved sequentially, and an acquisition function that allows to select the next point to evaluate in the sequential optimization procedure.

In this post, I will show how to use conformalized surrogates to tune hyperparameters of machine learning models. In this context, instead of using the posterior closed-form distribution of a Gaussian Process, any conformalized surrogate can be used for a probabilistic approximation of the objective function. And since there’s no closed-form expression of the acquisition function (here the Expected Improvement over the current optimum), a monte-carlo approximation of the (expectation) acquisition function is used, based on simulations of the conformalized surrogate. The simulation approach is similar to the one used in this post, except, the sequential ordering doesn’t matter here.

0 - Install and load packages

!pip install nnetsauce

!pip install git+https://github.com/Techtonique/GPopt.git --upgrade --no-cache-dir

import GPopt as gp
import nnetsauce as ns
import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import ElasticNetCV
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn import metrics
from time import time

1 - Cross-validation and hyperparameter tuning

def ridge2_cv(X_train, y_train,
              lambda1 = 0.1,
              lambda2 = 0.1,
              n_hidden_features=5,
              n_clusters=5,
              dropout = 0.8,
              solver="L-BFGS-B"):

  estimator  = ns.Ridge2Classifier(lambda1 = lambda1,
                                   lambda2 = lambda2,
                                   n_hidden_features=n_hidden_features,
                                   n_clusters=n_clusters,
                                   dropout = dropout,
                                   solver=solver)

  return -cross_val_score(estimator, X_train, y_train,
                          scoring='accuracy',
                          cv=5, n_jobs=None,
                          verbose=0).mean()

def optimize_ridge2(X_train, y_train, solver="L-BFGS-B",
                    surrogate="rf"):
  # objective function for hyperparams tuning
  def crossval_objective(x):
    return ridge2_cv(X_train=X_train,
                  y_train=y_train,
                  lambda1 = 10**x[0],
                  lambda2 = 10**x[1],
                  n_hidden_features=int(x[2]),
                  n_clusters=int(x[3]),
                  dropout = x[4],
                  solver = solver)
  if surrogate == "rf":
    gp_opt = gp.GPOpt(objective_func=crossval_objective,
                      lower_bound = np.array([ -10, -10,   3, 2, 0.6]),
                      upper_bound = np.array([  10,  10, 100, 5,   1]),
                      surrogate_obj = ns.CustomRegressor(obj=RandomForestRegressor(), # it's a conformalized quasi-randomized network
                                                        replications=250, # number of simulations for evaluating the expected improvement
                                                        type_pi="kde"), # Kernel Density Estimation is used for simulation
                      acquisition="ei", # expected improvement by simulation
                      params_names=["lambda1", "lambda2", "n_hidden_features", "n_clusters", "dropout"],
                      n_init=10, n_iter=90, seed=3137)
  elif surrogate == "enet":
    gp_opt = gp.GPOpt(objective_func=crossval_objective,
                      lower_bound = np.array([ -10, -10,   3, 2, 0.6]),
                      upper_bound = np.array([  10,  10, 100, 5,   1]),
                      surrogate_obj = ns.CustomRegressor(obj=ElasticNetCV(), # the model is nonlinear, it's a conformalized quasi-randomized network
                                                        replications=250, # number of simulations for evaluating the expected improvement
                                                        type_pi="kde"), # Kernel Density Estimation is used for simulation
                      acquisition="ei", # expected improvement by simulation
                      params_names=["lambda1", "lambda2", "n_hidden_features", "n_clusters", "dropout"],
                      n_init=10, n_iter=90, seed=3137)

  return gp_opt.optimize(method = "mc", verbose=2, abs_tol=1e-3) # monte carlo computation of expected improvement

dataset = load_breast_cancer()
X = dataset.data
y = dataset.target

# split data into training test and test set
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2, random_state=3137)

# hyperparams tuning, surrogate = conformalized Random Forest
res_opt1 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B", surrogate = "rf")
print(res_opt1)

# hyperparams tuning with different starting values for the optimization algorithm, surrogate = conformalized Random Forest
res_opt2 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B-lstsq", surrogate = "rf")
print(res_opt2)

# hyperparams tuning, surrogate = conformalized ElasticNet
res_opt3 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B", surrogate = "enet")
print(res_opt3)

# hyperparams tuning with different starting values for the optimization algorithm, surrogate = conformalized ElasticNet
res_opt4 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B-lstsq", surrogate = "enet")
print(res_opt4)

res_opt1.best_params["lambda1"] = 10**(res_opt1.best_params["lambda1"])
res_opt1.best_params["lambda2"] = 10**(res_opt1.best_params["lambda2"])
res_opt1.best_params["n_hidden_features"] = int(res_opt1.best_params["n_hidden_features"])
res_opt1.best_params["n_clusters"] = int(res_opt1.best_params["n_clusters"])
print(res_opt1.best_params)

res_opt2.best_params["lambda1"] = 10**(res_opt2.best_params["lambda1"])
res_opt2.best_params["lambda2"] = 10**(res_opt2.best_params["lambda2"])
res_opt2.best_params["n_hidden_features"] = int(res_opt2.best_params["n_hidden_features"])
res_opt2.best_params["n_clusters"] = int(res_opt2.best_params["n_clusters"])
print(res_opt2.best_params)

res_opt3.best_params["lambda1"] = 10**(res_opt3.best_params["lambda1"])
res_opt3.best_params["lambda2"] = 10**(res_opt3.best_params["lambda2"])
res_opt3.best_params["n_hidden_features"] = int(res_opt3.best_params["n_hidden_features"])
res_opt3.best_params["n_clusters"] = int(res_opt3.best_params["n_clusters"])
print(res_opt3.best_params)

res_opt4.best_params["lambda1"] = 10**(res_opt4.best_params["lambda1"])
res_opt4.best_params["lambda2"] = 10**(res_opt4.best_params["lambda2"])
res_opt4.best_params["n_hidden_features"] = int(res_opt4.best_params["n_hidden_features"])
res_opt4.best_params["n_clusters"] = int(res_opt4.best_params["n_clusters"])
print(res_opt4.best_params)

{'lambda1': 0.2143160456513889, 'lambda2': 99.32768474363539, 'n_hidden_features': 3, 'n_clusters': 4, 'dropout': 0.80830078125}
{'lambda1': 1.19372502075462e-10, 'lambda2': 0.0003873778332245682, 'n_hidden_features': 5, 'n_clusters': 3, 'dropout': 0.8306396484375}
{'lambda1': 0.03853145684685379, 'lambda2': 0.0020254361391973223, 'n_hidden_features': 91, 'n_clusters': 4, 'dropout': 0.75242919921875}
{'lambda1': 1.19372502075462e-10, 'lambda2': 0.0003873778332245682, 'n_hidden_features': 5, 'n_clusters': 3, 'dropout': 0.8306396484375}

2 - Out-of-sample scores

from time import time


clf1 = ns.Ridge2Classifier(**res_opt1.best_params,
                          solver="L-BFGS-B")
start = time()
clf1.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(f"Test set accuracy: {clf1.score(X_test, y_test)}")


clf2 = ns.Ridge2Classifier(**res_opt2.best_params,
                          solver="L-BFGS-B-lstsq")
start = time()
clf2.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(f"Test set accuracy: {clf2.score(X_test, y_test)}")

clf3 = ns.Ridge2Classifier(**res_opt3.best_params,
                          solver="L-BFGS-B")
start = time()
clf3.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(f"Test set accuracy: {clf3.score(X_test, y_test)}")


clf4 = ns.Ridge2Classifier(**res_opt4.best_params,
                          solver="L-BFGS-B-lstsq")
start = time()
clf4.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(f"Test set accuracy: {clf4.score(X_test, y_test)}")

Elapsed: 1.5195319652557373
Test set accuracy: 0.9736842105263158
Elapsed: 1.8859667778015137
Test set accuracy: 0.9736842105263158
Elapsed: 0.5796549320220947
Test set accuracy: 0.9736842105263158
Elapsed: 0.6930491924285889
Test set accuracy: 0.9736842105263158

# confusion matrix
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.metrics import confusion_matrix

y_pred = clf2.predict(X_test)
cm = confusion_matrix(y_test, y_pred)

fig, ax = plt.subplots(figsize=(10, 8))
ax = sns.heatmap(cm, annot=True, cmap='Blues', fmt='g', xticklabels=np.arange(0, 2), yticklabels=np.arange(0, 2))
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
plt.show()

Recognizing handwritten digits with Ridge2Classifier

2024-06-03T00:00:00+00:00

This post is about Ridge2Classifier, a classifier that I presented 5 years ago in this document. It’s now possible to choose starting values of the (likelihood) optimization algorithm which are solutions from least squares regression. Not always better, but can be seen as a new hyperparameter. Also, Ridge2Classifier used to fail miserably on digits data sets but now, with nnetsauce’s maturity, Ridge2Classifier is doing much better on this type of data, as demonstrated below.

0 - Install and load packages

!pip install nnetsauce

!pip install GPopt

import GPopt as gp
import nnetsauce as ns
import numpy as np

from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn import metrics
from time import time

1 - Cross-validation and hyperparameter tuning

def ridge2_cv(X_train, y_train,
              lambda1 = 0.1,
              lambda2 = 0.1,
              n_hidden_features=5,
              n_clusters=5,
              dropout = 0.8,
              solver="L-BFGS-B"): # 'solver' is the optimization algorithm

  estimator  = ns.Ridge2Classifier(lambda1 = lambda1,
                                   lambda2 = lambda2,
                                   n_hidden_features=n_hidden_features,
                                   n_clusters=n_clusters,
                                   dropout = dropout,
                                   solver=solver)

  return -cross_val_score(estimator, X_train, y_train,
                          scoring='accuracy',
                          cv=5, n_jobs=None,
                          verbose=0).mean()

def optimize_ridge2(X_train, y_train, solver="L-BFGS-B"):
  # objective function for hyperparams tuning
  def crossval_objective(x):
    return ridge2_cv(X_train=X_train,
                  y_train=y_train,
                  lambda1 = 10**x[0],
                  lambda2 = 10**x[1],
                  n_hidden_features=int(x[2]),
                  n_clusters=int(x[3]),
                  dropout = x[4],
                  solver = solver)
  gp_opt = gp.GPOpt(objective_func=crossval_objective,
                    lower_bound = np.array([ -10, -10,   3, 2, 0.6]),
                    upper_bound = np.array([  10,  10, 100, 5,   1]),
                    params_names=["lambda1", "lambda2", "n_hidden_features", "n_clusters", "dropout"],
                    n_init=10, n_iter=90, seed=3137)
  return gp_opt.optimize(verbose=2, abs_tol=1e-3)

dataset = load_digits()
X = dataset.data
y = dataset.target

# split data into training test and test set
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2, random_state=3137)

# hyperparams tuning
res_opt1 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B")
print(res_opt1)

# hyperparams tuning with different starting values for the optimization algorithm
res_opt2 = optimize_ridge2(X_train, y_train, solver="L-BFGS-B-lstsq")
print(res_opt2)

res_opt1.best_params["lambda1"] = 10**(res_opt1.best_params["lambda1"])
res_opt1.best_params["lambda2"] = 10**(res_opt1.best_params["lambda2"])
res_opt1.best_params["n_hidden_features"] = int(res_opt1.best_params["n_hidden_features"])
res_opt1.best_params["n_clusters"] = int(res_opt1.best_params["n_clusters"])
print(res_opt1.best_params)

res_opt2.best_params["lambda1"] = 10**(res_opt2.best_params["lambda1"])
res_opt2.best_params["lambda2"] = 10**(res_opt2.best_params["lambda2"])
res_opt2.best_params["n_hidden_features"] = int(res_opt2.best_params["n_hidden_features"])
res_opt2.best_params["n_clusters"] = int(res_opt2.best_params["n_clusters"])
print(res_opt2.best_params)

{'lambda1': 5.243297406977503e-10, 'lambda2': 1.2433817601870388e-05, 'n_hidden_features': 14, 'n_clusters': 2, 'dropout': 0.94100341796875}
{'lambda1': 1.747558169384434e-08, 'lambda2': 1360.0188315151736, 'n_hidden_features': 14, 'n_clusters': 2, 'dropout': 0.7794189453125}

2 - Out-of-sample scores

from time import time


clf1 = ns.Ridge2Classifier(**res_opt1.best_params,
                          solver="L-BFGS-B")
start = time()
clf1.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(clf1.score(X_test, y_test))


clf2 = ns.Ridge2Classifier(**res_opt2.best_params,
                          solver="L-BFGS-B-lstsq")
start = time()
clf2.fit(X_train, y_train)
print(f"Elapsed: {time()-start}")
print(clf2.score(X_test, y_test))

Elapsed: 2.6086528301239014
0.9138888888888889
Elapsed: 1.2307183742523193
0.9416666666666667

# confusion matrix
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix

y_pred = clf2.predict(X_test)
cm = confusion_matrix(y_test, y_pred)

fig, ax = plt.subplots(figsize=(10, 8))
ax = sns.heatmap(cm, annot=True, cmap='Blues', fmt='g', xticklabels=np.arange(0, 10), yticklabels=np.arange(0, 10))
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
plt.show()

A plethora of datasets at your fingertips Part3: how many times do couples cheat on each other?

2024-02-19T00:00:00+00:00

Remark: I have no affiliation with these projects, but I love using quarto and Visual Studio Code’s extension Jupyter To Markdown for these posts. Just wanted to say that.

Today:

I’ll show you how to use Python’s nnetsauce (version 0.17.0) to download data from the R universe API.
I’ll use nnetsauce’s automated Machine Learning (AutoML) to adjust a model to the Affairs data set also studied in this post from 2023-12-25.
Understand the model’s decisions and obtain prediction intervals on the test set.

0 - Install packages

pip install nnetsauce the-teller imbalanced-learn matplotlib IPython

1- Import data from R universe API

import nnetsauce as ns
import teller as tr 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from IPython.display import display
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import RandomOverSampler
from time import time

downloader = ns.Downloader()

df = downloader.download(pkgname="AER", dataset="Affairs", source = "https://cran.r-universe.dev/")

print(f"===== df: \n {df} \n")
print(f"===== df.dtypes: \n {df.dtypes}")

===== df: 
      affairs  gender   age  yearsmarried children  religiousness  education  \
0          0    male 37.00         10.00       no              3         18   
1          0  female 27.00          4.00       no              4         14   
2          0  female 32.00         15.00      yes              1         12   
3          0    male 57.00         15.00      yes              5         18   
4          0    male 22.00          0.75       no              2         17   
..       ...     ...   ...           ...      ...            ...        ...   
596        1    male 22.00          1.50      yes              1         12   
597        7  female 32.00         10.00      yes              2         18   
598        2    male 32.00         10.00      yes              2         17   
599        2    male 22.00          7.00      yes              3         18   
600        1  female 32.00         15.00      yes              3         14   

     occupation  rating  
0             7       4  
1             6       4  
2             1       4  
3             6       5  
4             6       3  
..          ...     ...  
596           2       5  
597           5       4  
598           6       5  
599           6       2  
600           1       5  

[601 rows x 9 columns] 

===== df.dtypes: 
 affairs            int64
gender            object
age              float64
yearsmarried     float64
children          object
religiousness      int64
education          int64
occupation         int64
rating             int64
dtype: object

The data set is extensively in studied in this post.

2 - Data preprocessing

encoder = OneHotEncoder(sparse_output=False).set_output(transform="pandas")
df_char_encoded = encoder.fit_transform(df[['gender', 'children']])
df_char_encoded.drop(columns = ['gender_female', 'children_no'], axis=1, inplace=True)
df_minus_char = df.drop(columns=['affairs', 'gender', 'children'])

X = pd.concat((df_minus_char, df_char_encoded), axis=1)
y = np.float64(df['affairs'].values)

Upsampling

ros = RandomOverSampler(random_state=42)
X_res, y_res = ros.fit_resample(X, y)

Split data into training test and test set

X_train, X_test, y_train, y_test = train_test_split(X_res, y_res,
                                                    test_size=0.2, 
                                                    random_state=13)

3 - Automated Machine Learning with `nnetsauce`

# build the AutoML regressor 
clf = ns.LazyRegressor(verbose=0, ignore_warnings=True, custom_metric=None, 
n_hidden_features=10)

# adjust on training and evaluate on test data
models, predictions = clf.fit(X_train, X_test, y_train, y_test)

  0%|          | 0/32 [00:00<?, ?it/s]  3%|▎         | 1/32 [00:00<00:03,  7.76it/s]  9%|▉         | 3/32 [00:00<00:02, 13.29it/s] 22%|██▏       | 7/32 [00:00<00:01, 16.00it/s] 28%|██▊       | 9/32 [00:00<00:02,  9.88it/s] 38%|███▊      | 12/32 [00:00<00:01, 13.48it/s] 50%|█████     | 16/32 [00:01<00:01, 14.66it/s] 62%|██████▎   | 20/32 [00:01<00:00, 18.59it/s] 72%|███████▏  | 23/32 [00:02<00:01,  6.32it/s] 78%|███████▊  | 25/32 [00:02<00:01,  6.57it/s] 84%|████████▍ | 27/32 [00:03<00:01,  4.74it/s] 97%|█████████▋| 31/32 [00:03<00:00,  7.31it/s]100%|██████████| 32/32 [00:03<00:00,  8.57it/s]

Models leaderboard

display(models)

4 - Explaining best model’s “decisions”

Notice the best here.

model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
regr = model_dictionary['CustomRegressor(ExtraTreesRegressor)']
regr

CustomRegressor(n_hidden_features=10, obj=ExtraTreesRegressor(random_state=42))

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# creating the explainer
expr0 = tr.Explainer(obj=regr)
expr0

Explainer(obj=CustomRegressor(n_hidden_features=10,
                              obj=ExtraTreesRegressor(random_state=42)))

start = time()
# creating an Explainer for the fitted object `regr1`
expr = tr.Explainer(obj=regr)
# covariates' effects
expr.fit(X_test.values, y_test, X_names=X_test.columns.to_list(), method="ci")
# summary of results
expr.summary()
# timing
print(f"\n Elapsed: {time()-start}")

Score (rmse): 
 1.083


Residuals: 
   Min    1Q  Median   3Q  Max
-12.00 -0.00    0.00 0.00 1.33


Tests on marginal effects (Jackknife): 
              Estimate Std. Error 95% lbound 95% ubound Pr(>|t|)     
gender_male       4.24       2.16      -0.00       8.49     0.05    .
yearsmarried      0.18       0.03       0.12       0.24     0.00  ***
occupation        0.11       0.07      -0.02       0.25     0.10    -
education         0.08       0.04      -0.00       0.17     0.06    .
age              -0.06       0.01      -0.09      -0.03     0.00  ***
religiousness    -0.45       0.10      -0.65      -0.25     0.00  ***
children_yes     -0.48       0.27      -1.01       0.05     0.08    .
rating           -0.48       0.09      -0.66      -0.30     0.00  ***


Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘-’ 1


Multiple R-squared:  0.929, Adjusted R-squared:  0.928

 Elapsed: 118.04078912734985

The results are quite similar obtained in the previous study, with the same factors being associated with the same responses.

5 - Prediction interval on test set

pi = tr.PredictionInterval(regr, method="splitconformal", level=0.95)
pi.fit(X_train, y_train)
preds = pi.predict(X_test, return_pi=True)

pred = preds[0]
y_lower = preds[1]
y_upper = preds[2]

# compute and display the average coverage
print(f"coverage rate = {np.mean((y_test >= y_lower) & (y_test <= y_upper))}")

coverage rate = 0.9575645756457565

import warnings
warnings.filterwarnings('ignore')

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

import numpy as np
np.warnings.filterwarnings('ignore')


split_color = 'green'
local_color = 'gray'

np.random.seed(1)

def plot_func(x,
              y,
              y_u=None,
              y_l=None,
              pred=None,
              shade_color="",
              method_name="",
              title=""):

    fig = plt.figure()

    plt.plot(x, y, 'k.', alpha=.3, markersize=10,
             fillstyle='full', label=u'Test set observations')

    if (y_u is not None) and (y_l is not None):
        plt.fill(np.concatenate([x, x[::-1]]),
                 np.concatenate([y_u, y_l[::-1]]),
                 alpha=.3, fc=shade_color, ec='None',
                 label = method_name + ' Prediction interval')

    if pred is not None:
        plt.plot(x, pred, 'k--', lw=2, alpha=0.9,
                 label=u'Predicted value')

    #plt.ylim([-2.5, 7])
    plt.xlabel('$X$')
    plt.ylabel('$Y$')
    plt.legend(loc='upper right')
    plt.title(title)

    plt.show()

max_idx = 50
plot_func(x = range(max_idx),
          y = y_test[0:max_idx],
          y_u = y_upper[0:max_idx],
          y_l = y_lower[0:max_idx],
          pred = pred[0:max_idx],
          shade_color=split_color,
          title = f"Split conformal ({max_idx} first points in test set)")

nnetsauce's introduction as of 2024-02-11 (new version 0.17.0)

2024-02-11T00:00:00+00:00

0 - What is `nnetsauce`?

nnetsauce is a general-purpose computational tool for statistical/Machine Learning (ML). As of 2024-02-11, nnetsauce is available for R and Python. It contains various implementations of – shallow and deep – regression, classification, univariate and multivariate probabilistic time series forecasting models.

The specificity of nnetsauce resides on the fact that the great majority of these ML models are powered by Quasi-Randomized Networks (QRNNs, see Moudiki, Planchet, and Cousin (2018) and Moudiki (2019-2024)). Indeed, whereas traditional neural networks (NNs, see Goodfellow, Bengio, and Courville (2016)) are trained by using gradient-based optimization algorithms, QRNNs do actually engineer features by creating new, randomized or deterministic ones.

The package lives in and relies on a well-supported and documented Python ecosystem; the one introduced by Pedregosa et al. (2011):

ML models are trained by using a method called fit
predictions on unseen data are made by a method called predict
thanks to object-oriented programming’s inheritance, the models are a 100% compatible with scikit-learn’s Pipelines, GridSearchCVs, RandomizedSearchCVs, and cross_val_scores (tools for model calibration).

Changes in version 0.17.0 (2024-02-11) include:

Attribute estimators (a list of Estimator’s as strings) for LazyClassifier, LazyRegressor, LazyDeepClassifier, LazyDeepRegressor, LazyMTS, and LazyDeepMTS
New documentation for the package, using pdoc (highly customizable, based on the excellent Jinja2, HTML and CSS):

https://techtonique.github.io/nnetsauce/

Remove external regressors xreg at inference time for time series forecasting classes MTS and DeepMTS
New class Downloader: querying the R universe API for datasets (see https://thierrymoudiki.github.io/blog/2023/12/25/python/r/misc/mlsauce/runiverse-api2 for similar example in mlsauce)
Add custom metrics to Lazy*
Rename Deep regressors and classifiers to Deep* in Lazy*
Add attribute sort_by to Lazy* – sort the data frame output by a given metric
Add attribute classes_ to classifiers (ensure consistency with scikit-learn)

The best way to illustrate the use of nnetsauce is to showcase its integrated automated machine learning (AutoML) functionalities for solving a real-world problem. This will be done by applying a classification task on the Wisconsin breast cancer data set available in scikit-learn (Pedregosa et al. (2011)). The same functionality is also available for regression and time series forecasting tasks.

Contents

1 - Installing nnetsauce for Python
2 - Automated Machine Learning (AutoML) with nnetsauce classifiers on Wisconsin breast cancer data set
dependencies

1 - Installing `nnetsauce` for Python

There are three ways to install nnetsauce for Python:

1st method: from PyPI by using pip at the command line (stable version):

pip install nnetsauce

2nd method: using conda (Linux and macOS only for now)

conda install -c conda-forge nnetsauce

3rd method: from Github (development version)

pip install git+https://github.com/Techtonique/nnetsauce.git

git clone https://github.com/Techtonique/nnetsauce.git
cd nnetsauce
make install

One way to run all the examples available in the repository after cloning it (as done in this 3rd installation option) is to navigate into it and run:

cd nnetsauce
make install
make run-examples 

There are also:

Several Jupyter notebooks available in: https://github.com/Techtonique/nnetsauce/tree/master/nnetsauce/demo
Blog posts available at: https://thierrymoudiki.github.io/blog#QuasiRandomizedNN

2 - Automated Machine Learning (AutoML) with `nnetsauce` classifiers on Wisconsin breast cancer data set

Import the packages required for the examples:

import nnetsauce as ns
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import  load_breast_cancer
from sklearn.ensemble import ExtraTreesRegressor, RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics
from time import time

Run AutoML with nnetsauce classifiers on Wisconsin breast cancer (identify and benign or malignant) toy data set:

# load the whole data set 
dataset = load_breast_cancer()
X = dataset.data
y = dataset.target

# splitting data in a training set (80%) and a test set (20%)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=123)

Build the AutoML classifier

There are more than 90 classifiers available in nnetsauce as of 2024-02-11. The LazyClassifier is a meta-estimator that fits a large number of classifiers of different types, and provides the best model. It is a good starting point for solving a classification problem.

# build the AutoML classifier
clf = ns.LazyClassifier(verbose=0, ignore_warnings=True, custom_metric=None, 
n_hidden_features=10, col_sample=0.9)

# adjust on training and evaluate on test data
models, predictions = clf.fit(X_train, X_test, y_train, y_test)

  0%|          | 0/92 [00:00<?, ?it/s]
  1%|1         | 1/92 [00:00<01:06,  1.38it/s]
  2%|2         | 2/92 [00:00<00:37,  2.41it/s]
  4%|4         | 4/92 [00:01<00:18,  4.72it/s]
  8%|7         | 7/92 [00:01<00:09,  8.75it/s]
 10%|9         | 9/92 [00:01<00:10,  7.74it/s]
 12%|#1        | 11/92 [00:01<00:08,  9.47it/s]
 15%|#5        | 14/92 [00:01<00:06, 11.41it/s]
 18%|#8        | 17/92 [00:02<00:07,  9.84it/s]
 21%|##        | 19/92 [00:02<00:12,  5.75it/s]
 24%|##3       | 22/92 [00:03<00:08,  7.90it/s]
 26%|##6       | 24/92 [00:03<00:10,  6.36it/s]
 28%|##8       | 26/92 [00:03<00:08,  7.62it/s]
 30%|###       | 28/92 [00:03<00:07,  9.11it/s]
 33%|###2      | 30/92 [00:04<00:14,  4.14it/s]
 35%|###4      | 32/92 [00:05<00:12,  4.91it/s]
 37%|###6      | 34/92 [00:05<00:09,  5.92it/s]
 39%|###9      | 36/92 [00:06<00:18,  3.06it/s]
 40%|####      | 37/92 [00:07<00:21,  2.55it/s]
 42%|####2     | 39/92 [00:07<00:16,  3.24it/s]
 45%|####4     | 41/92 [00:08<00:13,  3.65it/s]
 46%|####5     | 42/92 [00:08<00:14,  3.45it/s]
 47%|####6     | 43/92 [00:08<00:12,  3.93it/s]
 48%|####7     | 44/92 [00:10<00:24,  1.96it/s]
 49%|####8     | 45/92 [00:10<00:19,  2.39it/s]
 50%|#####     | 46/92 [00:10<00:18,  2.55it/s]
 51%|#####1    | 47/92 [00:10<00:14,  3.07it/s]
 52%|#####2    | 48/92 [00:10<00:11,  3.79it/s]
 53%|#####3    | 49/92 [00:11<00:12,  3.58it/s]
 54%|#####4    | 50/92 [00:12<00:22,  1.90it/s]
 57%|#####6    | 52/92 [00:12<00:12,  3.11it/s]
 58%|#####7    | 53/92 [00:12<00:11,  3.49it/s]
 59%|#####8    | 54/92 [00:12<00:11,  3.44it/s]
 60%|#####9    | 55/92 [00:14<00:21,  1.69it/s]
 62%|######1   | 57/92 [00:14<00:12,  2.77it/s]
 64%|######4   | 59/92 [00:14<00:08,  4.09it/s]
 66%|######6   | 61/92 [00:14<00:07,  4.06it/s]
 67%|######7   | 62/92 [00:15<00:07,  3.82it/s]
 70%|######9   | 64/92 [00:15<00:05,  5.32it/s]
 73%|#######2  | 67/92 [00:16<00:04,  5.08it/s]
 75%|#######5  | 69/92 [00:16<00:04,  4.62it/s]
 77%|#######7  | 71/92 [00:16<00:03,  5.78it/s]
 80%|########  | 74/92 [00:16<00:02,  7.62it/s]
 83%|########2 | 76/92 [00:17<00:03,  4.52it/s]
 85%|########4 | 78/92 [00:18<00:02,  5.09it/s]
 87%|########6 | 80/92 [00:18<00:01,  6.28it/s]
 89%|########9 | 82/92 [00:19<00:03,  3.12it/s]
 92%|#########2| 85/92 [00:19<00:01,  4.35it/s]
 93%|#########3| 86/92 [00:20<00:01,  4.13it/s]
 95%|#########4| 87/92 [00:21<00:02,  2.35it/s]
100%|##########| 92/92 [00:21<00:00,  4.27it/s]

Print the models’ leaderboard

print(models)

                                                Accuracy  ...  Time Taken
Model                                                     ...            
SimpleMultitaskClassifier(ExtraTreesRegressor)      0.99  ...        0.50
CustomClassifier(RandomForestClassifier)            0.99  ...        0.43
CustomClassifier(MLPClassifier)                     0.99  ...        0.73
CustomClassifier(Perceptron)                        0.99  ...        0.04
CustomClassifier(LogisticRegression)                0.99  ...        0.05
...                                                  ...  ...         ...
MultitaskClassifier(DummyRegressor)                 0.64  ...        0.08
SimpleMultitaskClassifier(LassoLars)                0.64  ...        0.05
SimpleMultitaskClassifier(Lasso)                    0.64  ...        0.03
SimpleMultitaskClassifier(Lars)                     0.41  ...        0.05
MultitaskClassifier(Lars)                           0.39  ...        0.31

[89 rows x 5 columns]

Provide the best model

model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
fit_obj = model_dictionary['SimpleMultitaskClassifier(ExtraTreesRegressor)']
fit_obj

CustomClassifier(col_sample=0.9, n_hidden_features=10,
                 obj=SimpleMultitaskClassifier(obj=ExtraTreesRegressor()))

Classification report

# Classification report
start = time()
preds = fit_obj.predict(X_test)
print(f"Elapsed {time() - start}")

Elapsed 0.04414701461791992              precision    recall  f1-score   support

           0       0.98      1.00      0.99        40
           1       1.00      0.99      0.99        74

    accuracy                           0.99       114
   macro avg       0.99      0.99      0.99       114
weighted avg       0.99      0.99      0.99       114

Confusion matrix

metrics.ConfusionMatrixDisplay.from_estimator(fit_obj, X_test, y_test)

<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay object at 0x12ccf3050>

dependencies

python3 -m pip freeze

anyio==4.2.0
appnope==0.1.3
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.14.0
beautifulsoup4==4.12.2
binaryornot==0.4.4
bleach==6.1.0
certifi==2023.11.17
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
comm==0.2.0
contourpy==1.2.0
cookiecutter==2.5.0
cycler==0.12.1
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
distlib==0.3.8
docutils==0.20.1
executing==2.0.1
fastjsonschema==2.19.1
filelock==3.13.1
fonttools==4.47.2
fqdn==1.5.1
fsspec==2023.12.2
idna==3.6
importlib-metadata==7.0.1
ipykernel==6.28.0
ipython==8.19.0
ipywidgets==8.1.1
isoduration==20.11.0
jaraco.classes==3.3.0
jax==0.4.23
jaxlib==0.4.23
jedi==0.19.1
Jinja2==3.1.2
joblib==1.3.2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.20.0
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.9.0
jupyter-lsp==2.2.1
jupyter_client==8.6.0
jupyter_core==5.5.1
jupyter_server==2.12.1
jupyter_server_terminals==0.5.1
jupyterlab==4.0.10
jupyterlab-widgets==3.0.9
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.2
keyring==24.3.0
kiwisolver==1.4.5
liac-arff==2.5.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.2
matplotlib-inline==0.1.6
mdurl==0.1.2
minio==7.2.3
mistune==3.0.2
ml-dtypes==0.3.2
more-itertools==10.1.0
mpmath==1.3.0
nbclient==0.9.0
nbconvert==7.13.1
nbformat==5.9.2
nest-asyncio==1.5.8
networkx==3.2.1
nh3==0.2.15
nnetsauce @ file:///Users/t/Documents/Python_Packages/nnetsauce
notebook==7.0.6
notebook_shim==0.2.3
numpy==1.26.3
openml==0.14.2
opt-einsum==3.3.0
overrides==7.4.0
packaging==23.2
pandas==2.2.0
pandocfilters==1.5.0
parso==0.8.3
patsy==0.5.6
pexpect==4.9.0
pillow==10.2.0
pkginfo==1.9.6
platformdirs==4.1.0
prometheus-client==0.19.0
prompt-toolkit==3.0.43
psutil==5.9.7
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==15.0.0
pycparser==2.21
pycryptodome==3.20.0
Pygments==2.17.2
pyparsing==3.1.1
python-dateutil==2.8.2
python-json-logger==2.0.7
python-slugify==8.0.1
pytz==2023.4
PyYAML==6.0.1
pyzmq==25.1.2
qtconsole==5.5.1
QtPy==2.4.1
readme-renderer==42.0
referencing==0.32.0
requests==2.31.0
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986==2.0.0
rfc3986-validator==0.1.1
rich==13.7.0
rpds-py==0.16.2
scikit-learn==1.4.0
scipy==1.12.0
Send2Trash==1.8.2
six==1.16.0
sniffio==1.3.0
soupsieve==2.5
stack-data==0.6.3
statsmodels==0.14.1
sympy==1.12
tabpfn==0.1.9
terminado==0.18.0
text-unidecode==1.3
threadpoolctl==3.2.0
tinycss2==1.2.1
torch==2.2.0
tornado==6.4
tqdm==4.66.1
traitlets==5.14.0
twine==4.0.2
types-python-dateutil==2.8.19.14
typing_extensions==4.9.0
tzdata==2023.4
uri-template==1.3.0
urllib3==2.1.0
virtualenv==20.25.0
wcwidth==0.2.12
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
widgetsnbextension==4.0.9
xmltodict==0.13.0
zipp==3.17.0

sessionInfo()

R version 4.3.2 (2023-10-31)
Platform: x86_64-apple-darwin20 (64-bit)
Running under: macOS Sonoma 14.2

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Paris
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] digest_0.6.34     fastmap_1.1.1     xfun_0.41         Matrix_1.6-1.1   
 [5] lattice_0.21-9    reticulate_1.34.0 knitr_1.45        htmltools_0.5.7  
 [9] png_0.1-8         rmarkdown_2.25    cli_3.6.2         grid_4.3.2       
[13] compiler_4.3.2    rprojroot_2.0.4   here_1.0.1        rstudioapi_0.15.0
[17] tools_4.3.2       evaluate_0.23     Rcpp_1.0.12       yaml_2.3.8       
[21] rlang_1.1.3       jsonlite_1.8.8   

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT press.

Moudiki, T. 2019-2024. “Nnetsauce, A Package for Statistical/Machine Learning Using Randomized and Quasi-Randomized (Neural) Networks.” https://github.com/thierrymoudiki/nnetsauce.

Moudiki, T., Frédéric Planchet, and Areski Cousin. 2018. “Multiple Time Series Forecasting Using Quasi-Randomized Functional Link Neural Networks.” Risks 6 (1): 22.

Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, et al. 2011. “Scikit-Learn: Machine Learning in Python.” The Journal of Machine Learning Research 12: 2825–30.

DeepMTS, a Deep Learning Model for Multivariate Time Series

2024-01-15T00:00:00+00:00

New in nnetsauce v0.16.3:

add robust scaler to type_scaling in all models
relatively faster scaling in preprocessing
Regression-based classifiers (see https://www.researchgate.net/publication/377227280_Regression-based_machine_learning_classifiers)
DeepMTS (multivariate time series forecasting with deep quasi-random layers): see https://thierrymoudiki.github.io/blog/2024/01/15/python/quasirandomizednn/forecasting/DeepMTS
AutoML for MTS (multivariate time series forecasting): see https://thierrymoudiki.github.io/blog/2023/10/29/python/quasirandomizednn/MTS-LazyPredict
AutoML for DeepMTS (multivariate time series forecasting): see https://github.com/Techtonique/nnetsauce/blob/master/nnetsauce/demo/thierrymoudiki_20240106_LazyDeepMTS.ipynb
Spaghetti plots for MTS and DeepMTS (multivariate time series forecasting): see below
Subsample continuous and discrete responses

`DeepMTS` in nnetsauce v0.16.3 for Multivariate time series (MTS)

Contents

1 - Install
2 - DeepMTS

1 - Install

!pip uninstall nnetsauce --yes

Found existing installation: nnetsauce 0.16.3
Uninstalling nnetsauce-0.16.3:
  Successfully uninstalled nnetsauce-0.16.3

!pip install git+https://github.com/Techtonique/nnetsauce.git --upgrade --no-cache-dir

Collecting git+https://github.com/Techtonique/nnetsauce.git
  Cloning https://github.com/Techtonique/nnetsauce.git to /tmp/pip-req-build-2fy08xrz
  Running command git clone --filter=blob:none --quiet https://github.com/Techtonique/nnetsauce.git /tmp/pip-req-build-2fy08xrz
  Resolved https://github.com/Techtonique/nnetsauce.git to commit e99ea1404604dc282576abc610b44c490cd8b598
  Preparing metadata (setup.py) ... [?25l[?25hdone
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (1.3.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (3.7.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (1.23.5)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (1.5.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (1.11.4)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (1.2.2)
Requirement already satisfied: threadpoolctl in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (3.2.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (4.66.1)
Requirement already satisfied: jax in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (0.4.23)
Requirement already satisfied: jaxlib in /usr/local/lib/python3.10/dist-packages (from nnetsauce==0.16.3) (0.4.23+cuda12.cudnn89)
Requirement already satisfied: ml-dtypes>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from jax->nnetsauce==0.16.3) (0.2.0)
Requirement already satisfied: opt-einsum in /usr/local/lib/python3.10/dist-packages (from jax->nnetsauce==0.16.3) (3.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (4.47.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce==0.16.3) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->nnetsauce==0.16.3) (2023.3.post1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->nnetsauce==0.16.3) (1.16.0)
Building wheels for collected packages: nnetsauce
  Building wheel for nnetsauce (setup.py) ... [?25l[?25hdone
  Created wheel for nnetsauce: filename=nnetsauce-0.16.3-py2.py3-none-any.whl size=152402 sha256=10d081174d14ad5b6af07273a895e85fa0ff28527ec2a27db90aff43102e47f5
  Stored in directory: /tmp/pip-ephem-wheel-cache-a9o14nt9/wheels/18/d7/31/2518e2b1957d1fbc99b30e79e99976579d956e031b45f61794
Successfully built nnetsauce
Installing collected packages: nnetsauce
Successfully installed nnetsauce-0.16.3

#!pip install nnetsauce==0.16.2 --upgrade --no-cache-dir

!pip install statsmodels

Requirement already satisfied: statsmodels in /usr/local/lib/python3.10/dist-packages (0.14.1)
Requirement already satisfied: numpy<2,>=1.18 in /usr/local/lib/python3.10/dist-packages (from statsmodels) (1.23.5)
Requirement already satisfied: scipy!=1.9.2,>=1.4 in /usr/local/lib/python3.10/dist-packages (from statsmodels) (1.11.4)
Requirement already satisfied: pandas!=2.1.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from statsmodels) (1.5.3)
Requirement already satisfied: patsy>=0.5.4 in /usr/local/lib/python3.10/dist-packages (from statsmodels) (0.5.6)
Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.10/dist-packages (from statsmodels) (23.2)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas!=2.1.0,>=1.0->statsmodels) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas!=2.1.0,>=1.0->statsmodels) (2023.3.post1)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from patsy>=0.5.4->statsmodels) (1.16.0)

import nnetsauce as ns
import numpy as np
import pandas as pd
from sklearn.linear_model import ElasticNetCV, LassoCV
from sklearn.metrics import mean_squared_error
import statsmodels.api as sm
from statsmodels.tsa.base.datetools import dates_from_str

2 - `DeepMTS`

Macro data

# some example data
mdata = sm.datasets.macrodata.load_pandas().data

# prepare the dates index
dates = mdata[['year', 'quarter']].astype(int).astype(str)

quarterly = dates["year"] + "Q" + dates["quarter"]

quarterly = dates_from_str(quarterly)

print(mdata.head())

#mdata = mdata[['realgdp','realcons','realinv', 'realgovt',
#               'realdpi', 'cpi', 'm1', 'tbilrate', 'unemp',
#               'pop']]

mdata = mdata[['realgovt', 'tbilrate', 'cpi']]

mdata.index = pd.DatetimeIndex(quarterly)

data = np.log(mdata).diff().dropna()

#data = mdata

display(data)

     year  quarter  realgdp  realcons  realinv  realgovt  realdpi   cpi  \
1959.00     1.00  2710.35   1707.40   286.90    470.05  1886.90 28.98   
1959.00     2.00  2778.80   1733.70   310.86    481.30  1919.70 29.15   
1959.00     3.00  2775.49   1751.80   289.23    491.26  1916.40 29.35   
1959.00     4.00  2785.20   1753.70   299.36    484.05  1931.30 29.37   
1960.00     1.00  2847.70   1770.50   331.72    462.20  1955.50 29.54   

      m1  tbilrate  unemp    pop  infl  realint  
139.70      2.82   5.80 177.15  0.00     0.00  
141.70      3.08   5.10 177.83  2.34     0.74  
140.50      3.82   5.30 178.66  2.74     1.09  
140.00      4.33   5.60 179.39  0.27     4.06  
139.60      3.50   5.20 180.01  2.31     1.19  

	realgovt	tbilrate	cpi
1959-06-30	0.02	0.09	0.01
1959-09-30	0.02	0.22	0.01
1959-12-31	-0.01	0.13	0.00
1960-03-31	-0.05	-0.21	0.01
1960-06-30	-0.00	-0.27	0.00
...	...	...	...
2008-09-30	0.03	-0.40	-0.01
2008-12-31	0.02	-2.28	-0.02
2009-03-31	-0.01	0.61	0.00
2009-06-30	0.03	-0.20	0.01
2009-09-30	0.02	-0.41	0.01

202 rows × 3 columns

n = data.shape[0]
max_idx_train = np.floor(n*0.8)
training_index = np.arange(0, max_idx_train)
testing_index = np.arange(max_idx_train, n)
df_train = data.iloc[training_index,:]
df_test = data.iloc[testing_index,:]

# Adjust ElasticNetCV
regr7 = ElasticNetCV()
obj_MTS2 = ns.DeepMTS(regr7,
                     n_layers=3,
                     lags = 4,
                     n_hidden_features=5,
                     replications=10,
                     kernel='gaussian',
                     verbose = 1)
obj_MTS2.fit(df_train)
res4 = obj_MTS2.predict(h=len(testing_index))

 Adjusting DeepRegressor to multivariate time series... 
 
100%|██████████| 3/3 [00:02<00:00,  1.02it/s]

 Simulate residuals using gaussian kernel... 

 Best parameters for gaussian kernel: {'bandwidth': 0.022335377063851233} 

100%|██████████| 10/10 [00:00<00:00, 1582.64it/s]
100%|██████████| 10/10 [00:00<00:00, 3664.43it/s]

obj_MTS2.plot("realgovt", type_plot="pi")
obj_MTS2.plot("tbilrate", type_plot="pi")
obj_MTS2.plot("cpi", type_plot="pi")

obj_MTS2.plot("realgovt", type_plot = "spaghetti")
obj_MTS2.plot("tbilrate", type_plot = "spaghetti")
obj_MTS2.plot("cpi", type_plot = "spaghetti")

A classifier that's very accurate (and deep) Pt.2: there are > 90 classifiers in nnetsauce

2024-01-08T00:00:00+00:00

Version v0.16.2 of nnetsauce is now available on your favorite platforms: PyPI, conda and GitHub for Python; R universe and GitHub for R. The updated documentation will be available next week (I’ve been using keras-autodoc so far for this purpose; it is now discontinued).

For Windows users: if you run into issues when trying to install nnetsauce (but you shouldn’t): remember that you can use the Windows Subsystem for Linux (WSL).

The changes in this new version are:

Regression-based classifiers
Automated Machine Learning (AutoML) for MTS (multivariate time series forecasting) (not in R yet)
AutoML for DeepMTS (multivariate time series forecasting) (not in R yet)
Subsample continuous and discrete responses

This post is about AutoML in nnetsauce using its 90 base classifiers, thus including regression-based classifiers. It’s a mix of these resources:

This PDF document describing the regression-based classifiers available in nnetsauce
Classification using linear regression
A classifier that’s very accurate (and deep)

The goal is to show that, with nnetsauce, it’s possible to calibrate a large number of classifiers and select the best one, using a few lines of code. AutoML for MTS (multivariate time series forecasting) and DeepMTS will be presented next week.

0 - Install packages

The notebook can be found on GitHub for reproducibility of the following results: https://github.com/Techtonique/nnetsauce/blob/master/nnetsauce/demo/thierrymoudiki_080124_lazy_classifier_deep_classifier.ipynb.

!pip install nnetsauce

Collecting nnetsauce
  Downloading nnetsauce-0.16.2-py2.py3-none-any.whl (152 kB)
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m152.2/152.2 kB[0m [31m2.3 MB/s[0m eta [36m0:00:00[0m
[?25hRequirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (1.3.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (3.7.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (1.23.5)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (1.5.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (1.11.4)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (1.2.2)
Requirement already satisfied: threadpoolctl in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (3.2.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (4.66.1)
Requirement already satisfied: jax in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (0.4.23)
Requirement already satisfied: jaxlib in /usr/local/lib/python3.10/dist-packages (from nnetsauce) (0.4.23+cuda12.cudnn89)
Requirement already satisfied: ml-dtypes>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from jax->nnetsauce) (0.2.0)
Requirement already satisfied: opt-einsum in /usr/local/lib/python3.10/dist-packages (from jax->nnetsauce) (3.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (4.47.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->nnetsauce) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->nnetsauce) (2023.3.post1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->nnetsauce) (1.16.0)
Installing collected packages: nnetsauce
Successfully installed nnetsauce-0.16.2

import os
import nnetsauce as ns
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
from time import time

1 - Lazy classifiers

1 - 1 breast cancer data

data = load_breast_cancer()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyClassifier(verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)

display(models)

100%|██████████| 90/90 [01:21<00:00,  1.10it/s]

 Elapsed: 81.99524259567261 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
CustomClassifier(RandomForestClassifier)	0.99	0.99	0.99	0.99	1.96
CustomClassifier(SGDClassifier)	0.99	0.99	0.99	0.99	0.24
CustomClassifier(QuadraticDiscriminantAnalysis)	0.99	0.99	0.99	0.99	1.20
CustomClassifier(Perceptron)	0.99	0.99	0.99	0.99	1.38
SimpleMultitaskClassifier(BaggingRegressor)	0.99	0.99	0.99	0.99	0.58
...	...	...	...	...	...
SimpleMultitaskClassifier(Lasso)	0.64	0.50	0.50	0.50	0.56
SimpleMultitaskClassifier(LarsCV)	0.64	0.50	0.50	0.50	0.76
MultitaskClassifier(LassoLars)	0.64	0.50	0.50	0.50	2.11
SimpleMultitaskClassifier(Lars)	0.55	0.57	0.57	0.56	0.40
MultitaskClassifier(Lars)	0.54	0.56	0.56	0.55	0.53

85 rows × 5 columns

model_dictionary["CustomClassifier(RandomForestClassifier)"]

CustomClassifier(obj=RandomForestClassifier(random_state=42))

print(classification_report(y_test, model_dictionary["CustomClassifier(RandomForestClassifier)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      0.98      0.99        41
           1       0.99      1.00      0.99        73

    accuracy                           0.99       114
   macro avg       0.99      0.99      0.99       114
weighted avg       0.99      0.99      0.99       114

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(RandomForestClassifier)"], X_test, y_test)
plt.show()

1 - 2 - iris data

data = load_iris()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyClassifier(verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [00:10<00:00,  8.95it/s]

 Elapsed: 10.071635246276855 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
MultitaskClassifier(AdaBoostRegressor)	0.97	0.97	None	0.97	0.16
CustomClassifier(SGDClassifier)	0.97	0.97	None	0.97	0.02
MultitaskClassifier(BayesianRidge)	0.97	0.97	None	0.97	0.04
MultitaskClassifier(DecisionTreeRegressor)	0.97	0.97	None	0.97	0.05
SimpleMultitaskClassifier(LinearRegression)	0.97	0.97	None	0.97	0.02
...	...	...	...	...	...
MultitaskClassifier(Lasso)	0.20	0.33	None	0.07	0.07
SimpleMultitaskClassifier(ElasticNet)	0.20	0.33	None	0.07	0.02
SimpleMultitaskClassifier(DummyRegressor)	0.20	0.33	None	0.07	0.02
CustomClassifier(DummyClassifier)	0.20	0.33	None	0.07	0.02
MultitaskClassifier(LassoLars)	0.20	0.33	None	0.07	0.06

85 rows × 5 columns

model_dictionary["MultitaskClassifier(AdaBoostRegressor)"]

CustomClassifier(obj=MultitaskClassifier(obj=AdaBoostRegressor()))

print(classification_report(y_test, model_dictionary["MultitaskClassifier(AdaBoostRegressor)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        13
           1       0.86      1.00      0.92         6
           2       1.00      0.91      0.95        11

    accuracy                           0.97        30
   macro avg       0.95      0.97      0.96        30
weighted avg       0.97      0.97      0.97        30

ConfusionMatrixDisplay.from_estimator(model_dictionary["MultitaskClassifier(AdaBoostRegressor)"], X_test, y_test)
plt.show()

1 - 3 - wine data

data = load_wine()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyClassifier(verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [00:13<00:00,  6.66it/s]

 Elapsed: 13.523086071014404 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
MultitaskClassifier(AdaBoostRegressor)	1.00	1.00	None	1.00	0.51
SimpleMultitaskClassifier(ExtraTreesRegressor)	1.00	1.00	None	1.00	0.70
MultitaskClassifier(ElasticNetCV)	1.00	1.00	None	1.00	0.56
SimpleMultitaskClassifier(LassoLarsCV)	1.00	1.00	None	1.00	0.09
SimpleMultitaskClassifier(HuberRegressor)	1.00	1.00	None	1.00	0.18
...	...	...	...	...	...
MultitaskClassifier(Lars)	0.31	0.28	None	0.31	0.07
CustomClassifier(DummyClassifier)	0.31	0.33	None	0.14	0.02
MultitaskClassifier(Lasso)	0.31	0.33	None	0.14	0.06
MultitaskClassifier(LassoLars)	0.31	0.33	None	0.14	0.06
MultitaskClassifier(DummyRegressor)	0.31	0.33	None	0.14	0.05

83 rows × 5 columns

model_dictionary["MultitaskClassifier(AdaBoostRegressor)"]

CustomClassifier(obj=MultitaskClassifier(obj=AdaBoostRegressor()))

print(classification_report(y_test, model_dictionary["MultitaskClassifier(AdaBoostRegressor)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         8
           1       1.00      1.00      1.00        11
           2       1.00      1.00      1.00        17

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36

ConfusionMatrixDisplay.from_estimator(model_dictionary["MultitaskClassifier(AdaBoostRegressor)"], X_test, y_test)
plt.show()

1 - 4 - digits data

data = load_digits()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyClassifier(verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [03:47<00:00,  2.53s/it]

 Elapsed: 227.8047935962677 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
CustomClassifier(SVC)	0.99	0.99	None	0.99	0.99
CustomClassifier(ExtraTreesClassifier)	0.98	0.98	None	0.98	1.26
MultitaskClassifier(ExtraTreesRegressor)	0.98	0.98	None	0.98	7.42
SimpleMultitaskClassifier(ExtraTreesRegressor)	0.98	0.98	None	0.98	6.63
CustomClassifier(LogisticRegressionCV)	0.98	0.98	None	0.98	5.03
...	...	...	...	...	...
MultitaskClassifier(LassoLars)	0.08	0.10	None	0.01	2.20
MultitaskClassifier(Lasso)	0.08	0.10	None	0.01	2.07
MultitaskClassifier(DummyRegressor)	0.08	0.10	None	0.01	2.39
MultitaskClassifier(ElasticNet)	0.08	0.10	None	0.01	1.89
SimpleMultitaskClassifier(DummyRegressor)	0.08	0.10	None	0.01	1.19

83 rows × 5 columns

model_dictionary["CustomClassifier(SVC)"]

CustomClassifier(obj=SVC(random_state=42))

print(classification_report(y_test, model_dictionary["CustomClassifier(SVC)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        39
           1       0.94      1.00      0.97        34
           2       1.00      0.97      0.99        36
           3       1.00      1.00      1.00        33
           4       0.95      1.00      0.98        42
           5       1.00      0.95      0.97        37
           6       1.00      1.00      1.00        43
           7       1.00      1.00      1.00        31
           8       1.00      0.95      0.97        37
           9       0.97      1.00      0.98        28

    accuracy                           0.99       360
   macro avg       0.99      0.99      0.99       360
weighted avg       0.99      0.99      0.99       360

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(SVC)"], X_test, y_test)
plt.show()

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(LogisticRegressionCV)"], X_test, y_test)
plt.show()

2 - Lazy Deep classifiers

2 - 1 breast cancer data

data = load_breast_cancer()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)

display(models)

100%|██████████| 90/90 [03:43<00:00,  2.48s/it]

 Elapsed: 223.3978340625763 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
SimpleMultitaskClassifier(MLPRegressor)	0.99	0.99	0.99	0.99	3.77
MultitaskClassifier(BaggingRegressor)	0.99	0.99	0.99	0.99	3.45
SimpleMultitaskClassifier(BaggingRegressor)	0.99	0.99	0.99	0.99	1.94
CustomClassifier(Perceptron)	0.99	0.99	0.99	0.99	1.54
CustomClassifier(MLPClassifier)	0.99	0.99	0.99	0.99	3.19
...	...	...	...	...	...
MultitaskClassifier(LassoLars)	0.64	0.50	0.50	0.50	2.25
SimpleMultitaskClassifier(Lars)	0.54	0.54	0.54	0.55	3.24
MultitaskClassifier(Lars)	0.51	0.51	0.51	0.52	2.02
SimpleMultitaskClassifier(RANSACRegressor)	0.39	0.52	0.52	0.26	2.86
MultitaskClassifier(RANSACRegressor)	0.36	0.50	0.50	0.19	5.04

85 rows × 5 columns

model_dictionary["SimpleMultitaskClassifier(MLPRegressor)"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SimpleMultitaskClassifier(obj=MLPRegressor()))))

print(classification_report(y_test, model_dictionary["SimpleMultitaskClassifier(MLPRegressor)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      0.98      0.99        41
           1       0.99      1.00      0.99        73

    accuracy                           0.99       114
   macro avg       0.99      0.99      0.99       114
weighted avg       0.99      0.99      0.99       114

ConfusionMatrixDisplay.from_estimator(model_dictionary["SimpleMultitaskClassifier(MLPRegressor)"], X_test, y_test)
plt.show()

2 - 2 - iris data

data = load_iris()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [00:25<00:00,  3.54it/s]

 Elapsed: 25.443820476531982 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
CustomClassifier(BaggingClassifier)	1.00	1.00	None	1.00	0.12
CustomClassifier(DecisionTreeClassifier)	1.00	1.00	None	1.00	0.08
MultitaskClassifier(MLPRegressor)	1.00	1.00	None	1.00	0.85
MultitaskClassifier(BaggingRegressor)	0.97	0.97	None	0.97	0.25
MultitaskClassifier(ElasticNetCV)	0.97	0.97	None	0.97	1.34
...	...	...	...	...	...
SimpleMultitaskClassifier(Lasso)	0.20	0.33	None	0.07	0.09
MultitaskClassifier(Lasso)	0.20	0.33	None	0.07	0.15
SimpleMultitaskClassifier(ElasticNet)	0.20	0.33	None	0.07	0.10
SimpleMultitaskClassifier(DummyRegressor)	0.20	0.33	None	0.07	0.08
MultitaskClassifier(LassoLars)	0.20	0.33	None	0.07	0.13

85 rows × 5 columns

model_dictionary["CustomClassifier(BaggingClassifier)"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))

print(classification_report(y_test, model_dictionary["CustomClassifier(BaggingClassifier)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        13
           1       1.00      1.00      1.00         6
           2       1.00      1.00      1.00        11

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(BaggingClassifier)"], X_test, y_test)
plt.show()

2 - 3 - wine data

data = load_wine()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [00:27<00:00,  3.26it/s]

 Elapsed: 27.655547857284546 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
MultitaskClassifier(LassoCV)	1.00	1.00	None	1.00	1.36
SimpleMultitaskClassifier(LinearRegression)	1.00	1.00	None	1.00	0.10
MultitaskClassifier(PassiveAggressiveRegressor)	1.00	1.00	None	1.00	0.19
MultitaskClassifier(SGDRegressor)	1.00	1.00	None	1.00	0.20
SimpleMultitaskClassifier(AdaBoostRegressor)	1.00	1.00	None	1.00	0.52
...	...	...	...	...	...
MultitaskClassifier(Lars)	0.31	0.38	None	0.27	0.21
CustomClassifier(DummyClassifier)	0.31	0.33	None	0.14	0.10
MultitaskClassifier(Lasso)	0.31	0.33	None	0.14	0.16
MultitaskClassifier(LassoLars)	0.31	0.33	None	0.14	0.18
MultitaskClassifier(DummyRegressor)	0.31	0.33	None	0.14	0.45

83 rows × 5 columns

model_dictionary["MultitaskClassifier(LassoCV)"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=MultitaskClassifier(obj=LassoCV()))))

print(classification_report(y_test, model_dictionary["MultitaskClassifier(LassoCV)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         8
           1       1.00      1.00      1.00        11
           2       1.00      1.00      1.00        17

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36

ConfusionMatrixDisplay.from_estimator(model_dictionary["MultitaskClassifier(LassoCV)"], X_test, y_test)
plt.show()

2 - 4 - digits data

data = load_digits()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 90/90 [10:05<00:00,  6.72s/it]

 Elapsed: 605.1929349899292 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
CustomClassifier(SVC)	0.99	0.99	None	0.99	4.41
SimpleMultitaskClassifier(AdaBoostRegressor)	0.98	0.98	None	0.98	12.88
MultitaskClassifier(ExtraTreesRegressor)	0.98	0.98	None	0.98	17.08
SimpleMultitaskClassifier(ExtraTreesRegressor)	0.98	0.98	None	0.98	19.48
CustomClassifier(LogisticRegressionCV)	0.98	0.98	None	0.98	15.02
...	...	...	...	...	...
SimpleMultitaskClassifier(DummyRegressor)	0.08	0.10	None	0.01	3.48
MultitaskClassifier(DummyRegressor)	0.08	0.10	None	0.01	6.17
MultitaskClassifier(ElasticNet)	0.08	0.10	None	0.01	5.31
CustomClassifier(DummyClassifier)	0.08	0.10	None	0.01	4.52
MultitaskClassifier(LassoLars)	0.08	0.10	None	0.01	4.03

83 rows × 5 columns

model_dictionary["CustomClassifier(SVC)"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))

print(classification_report(y_test, model_dictionary["CustomClassifier(SVC)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        39
           1       0.94      1.00      0.97        34
           2       1.00      0.97      0.99        36
           3       1.00      1.00      1.00        33
           4       0.95      1.00      0.98        42
           5       1.00      0.95      0.97        37
           6       1.00      1.00      1.00        43
           7       1.00      1.00      1.00        31
           8       1.00      0.95      0.97        37
           9       0.97      1.00      0.98        28

    accuracy                           0.99       360
   macro avg       0.99      0.99      0.99       360
weighted avg       0.99      0.99      0.99       360

print(classification_report(y_test, model_dictionary["CustomClassifier(LogisticRegressionCV)"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        39
           1       0.87      1.00      0.93        34
           2       1.00      1.00      1.00        36
           3       1.00      1.00      1.00        33
           4       0.98      0.98      0.98        42
           5       1.00      0.97      0.99        37
           6       1.00      0.95      0.98        43
           7       0.97      1.00      0.98        31
           8       1.00      0.89      0.94        37
           9       0.97      1.00      0.98        28

    accuracy                           0.98       360
   macro avg       0.98      0.98      0.98       360
weighted avg       0.98      0.98      0.98       360

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(SVC)"], X_test, y_test)
plt.show()

ConfusionMatrixDisplay.from_estimator(model_dictionary["CustomClassifier(LogisticRegressionCV)"], X_test, y_test)
plt.show()

Quasi-randomized nnetworks in Julia, Python and R

2023-11-27T00:00:00+00:00

nnetsauce, a package for quasi-randomized supervised learning (classification and regression), is currently available for R and Python. For more details on nnetsauce, you can read these posts.

I’ve always wanted to port nnetsauce to the Julia language. However, in the past few years, there was a little timing overhead (more precisely, a lag) when I tried to do that with Julia’s PyCall, based on my Python source code. This overhead seems to have ‘disappeared’.

Julia language’s nnetsauce is not a package yet, but you can already use nnetsauce in Julia.

Here’s how I did it on Ubuntu Linux:

Contents

1 - Install Julia
2 - Example using a nnetsauce classifier in Julia language

1 - Install Julia

Run (terminal):

wget https://julialang-s3.julialang.org/bin/linux/x64/1.9/julia-1.9.4-linux-x86_64.tar.gz

Run (terminal):

tar zxvf julia-1.9.4-linux-x86_64.tar.gz

Run (terminal)(This is VSCode, but use your favorite editor here):

code ~/.bashrc

Add to .bashrc (last line):

export PATH="$PATH:julia-1.9.4/bin"

Run (terminal):

source ~/.bashrc

Run (terminal):

julia nnetsauce_example.jl

2 - Example using a nnetsauce classifier in Julia language

For Python users, notice that this is basically… Python ^^

using Pkg
ENV["PYTHON"] = ""  # replace with your Python path
Pkg.add("PyCall")
Pkg.build("PyCall")
Pkg.add("Conda")
Pkg.build("Conda")

using PyCall
using Conda

Conda.add("pip")  # Ensure pip is installed
Conda.pip_interop(true)  # Enable pip interop
Conda.pip("install", "scikit-learn")  # Install scikit-learn
Conda.pip("install", "jax")  # /!\ Only on Linux or macOS: Install jax
Conda.pip("install", "jaxlib")  # /!\ Only on Linux or macOS: Install jaxlib
Conda.pip("install", "nnetsauce")  # Install nnetsauce
Conda.add("numpy")

np = pyimport("numpy")
ns = pyimport("nnetsauce")
sklearn = pyimport("sklearn")


# 1 - breast cancer dataset

dataset = sklearn.datasets.load_breast_cancer()

X = dataset["data"]
y = dataset["target"]

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, 
test_size=0.2, random_state=123)

clf = ns.Ridge2MultitaskClassifier(n_hidden_features=9, dropout=0.43, n_clusters=1, 
lambda1=1.24023438e+01, lambda2=7.30263672e+03)

@time clf.fit(X=X_train, y=y_train) # timing?

print("\n\n Model parameters: \n\n")
print(clf.get_params())

print("\n\n Testing score: \n\n") # Classifier's accuracy
print(clf.score(X_test, y_test)) # Must be: 0.9824561403508771
print("\n\n")

# 2 - wine dataset

dataset = sklearn.datasets.load_wine()

X = dataset["data"]
y = dataset["target"]

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, 
test_size=0.2, random_state=123)

clf = ns.Ridge2MultitaskClassifier(n_hidden_features=15,
dropout=0.1, n_clusters=3, 
type_clust="gmm")

@time clf.fit(X=X_train, y=y_train) # timing?

print("\n\n Model parameters: \n\n")
print(clf.get_params())

print("\n\n Testing score: \n\n") # Classifier's accuracy
print(clf.score(X_test, y_test)) # Must be 1.0
print("\n\n")

A classifier that's very accurate (and deep)

2023-11-12T00:00:00+00:00

Version v0.15.0 of nnetsauce is now available on your favorite platforms: PyPI, conda and GitHub for Python; R universe and GitHub for R.

The changes in this version are mostly related to Automated Machine learning (AutoML):

lazy prediction for classification and regression: see this post for more details on the subject (and remember to use pip install nnetsauce directly, instead of installing from a GitHub branch named lazy-predict)
lazy prediction for multivariate time series (MTS): see this post for more details on the subject (and remember to use pip install nnetsauce directly, instead of installing from a GitHub branch named lazy-predict)
lazy prediction with deep quasi-randomized nnetworks will be described in this post

Note that in the example below, for the offically released version (v0.15.0), Gradient boosting classifiers are available. This doesn’t change the best model chosen by the algorithm (which is never Gradient boosting, because the deep model is already doing “too much”). Not sure if Gradient boosting will be kept as a default model here because, in this context, it’s relatively slow.

To finish, for Windows users, if you run into issues when trying to install nnetsauce (but you shouldn’t): remember that you can use the Windows Subsystem for Linux (WSL).

0 - Install and import packages
1 - breast cancer data
2 - iris data
3 - wine data
4 - digits data
5 - R examples

Here is a jupyter notebook allowing you to reproduce these results. Do not hesitate to modify these examples by choosing – in LazyDeepClassifier – a different number of layers n_layers, or the number of engineered features per layer, n_hidden_features.

0 - Install and import packages

top

!pip install nnetsauce --upgrade

import os
import nnetsauce as ns
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
from time import time

1 - breast cancer data

top

data = load_breast_cancer()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)

display(models)

100%|██████████| 27/27 [00:44<00:00,  1.65s/it]

 Elapsed: 44.49836850166321 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
Perceptron	0.99	0.99	0.99	0.99	2.15
SGDClassifier	0.98	0.98	0.98	0.98	1.99
RandomForestClassifier	0.98	0.98	0.98	0.98	2.76
PassiveAggressiveClassifier	0.98	0.98	0.98	0.98	1.89
LogisticRegression	0.98	0.98	0.98	0.98	1.14
ExtraTreesClassifier	0.98	0.98	0.98	0.98	3.13
BaggingClassifier	0.97	0.97	0.97	0.97	1.62
LabelPropagation	0.97	0.97	0.97	0.97	1.69
LabelSpreading	0.97	0.97	0.97	0.97	1.08
AdaBoostClassifier	0.97	0.97	0.97	0.97	3.42
LinearSVC	0.97	0.97	0.97	0.97	1.65
KNeighborsClassifier	0.97	0.97	0.97	0.97	1.07
SVC	0.97	0.96	0.96	0.97	1.07
CalibratedClassifierCV	0.97	0.96	0.96	0.97	1.82
DecisionTreeClassifier	0.96	0.96	0.96	0.96	1.50
QuadraticDiscriminantAnalysis	0.96	0.95	0.95	0.96	1.16
LinearDiscriminantAnalysis	0.96	0.95	0.95	0.96	1.39
ExtraTreeClassifier	0.96	0.94	0.94	0.96	0.83
RidgeClassifier	0.96	0.94	0.94	0.96	3.15
RidgeClassifierCV	0.96	0.94	0.94	0.96	1.51
GaussianNB	0.93	0.90	0.90	0.93	1.50
NearestCentroid	0.93	0.90	0.90	0.93	3.00
NuSVC	0.93	0.90	0.90	0.93	1.27
BernoulliNB	0.91	0.89	0.89	0.91	1.15
DummyClassifier	0.64	0.50	0.50	0.50	0.96

model_dictionary["Perceptron"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42))))

print(classification_report(y_test, model_dictionary["Perceptron"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      0.98      0.99        41
           1       0.99      1.00      0.99        73

    accuracy                           0.99       114
   macro avg       0.99      0.99      0.99       114
weighted avg       0.99      0.99      0.99       114

ConfusionMatrixDisplay.from_estimator(model_dictionary["Perceptron"], X_test, y_test)
plt.show()

2 - iris data

top

data = load_iris()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [00:05<00:00,  4.51it/s]

 Elapsed: 5.992894172668457 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
BaggingClassifier	1.00	1.00	None	1.00	0.18
DecisionTreeClassifier	1.00	1.00	None	1.00	0.11
KNeighborsClassifier	0.97	0.97	None	0.97	0.10
CalibratedClassifierCV	0.97	0.97	None	0.97	0.24
SGDClassifier	0.97	0.97	None	0.97	0.21
ExtraTreeClassifier	0.97	0.97	None	0.97	0.10
RidgeClassifier	0.97	0.97	None	0.97	1.05
RidgeClassifierCV	0.97	0.97	None	0.97	0.21
RandomForestClassifier	0.97	0.97	None	0.97	0.88
LogisticRegression	0.97	0.97	None	0.97	0.14
LinearSVC	0.97	0.97	None	0.97	0.11
PassiveAggressiveClassifier	0.93	0.94	None	0.93	0.11
Perceptron	0.93	0.94	None	0.93	0.11
BernoulliNB	0.93	0.94	None	0.93	0.10
LabelSpreading	0.93	0.91	None	0.93	0.31
LabelPropagation	0.93	0.91	None	0.93	0.20
ExtraTreesClassifier	0.93	0.91	None	0.93	0.45
GaussianNB	0.90	0.91	None	0.90	0.10
AdaBoostClassifier	0.90	0.91	None	0.90	0.38
SVC	0.87	0.88	None	0.87	0.19
NuSVC	0.87	0.88	None	0.87	0.12
NearestCentroid	0.87	0.88	None	0.87	0.10
LinearDiscriminantAnalysis	0.63	0.67	None	0.54	0.20
QuadraticDiscriminantAnalysis	0.20	0.33	None	0.07	0.11
DummyClassifier	0.20	0.33	None	0.07	0.10

model_dictionary["BaggingClassifier"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))

print(classification_report(y_test, model_dictionary["BaggingClassifier"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        13
           1       1.00      1.00      1.00         6
           2       1.00      1.00      1.00        11

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

ConfusionMatrixDisplay.from_estimator(model_dictionary["BaggingClassifier"], X_test, y_test)
plt.show()

3 - wine data

top

data = load_wine()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [00:05<00:00,  5.09it/s]

 Elapsed: 5.312330007553101 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
RidgeClassifierCV	1.00	1.00	None	1.00	0.15
RidgeClassifier	1.00	1.00	None	1.00	0.13
Perceptron	1.00	1.00	None	1.00	0.15
SVC	0.97	0.98	None	0.97	0.13
SGDClassifier	0.97	0.98	None	0.97	0.13
CalibratedClassifierCV	0.97	0.98	None	0.97	0.27
PassiveAggressiveClassifier	0.97	0.98	None	0.97	0.14
LogisticRegression	0.97	0.98	None	0.97	0.16
LinearSVC	0.97	0.98	None	0.97	0.14
BaggingClassifier	0.97	0.97	None	0.97	0.21
RandomForestClassifier	0.97	0.97	None	0.97	0.61
LinearDiscriminantAnalysis	0.97	0.97	None	0.97	0.16
LabelSpreading	0.94	0.94	None	0.94	0.44
LabelPropagation	0.94	0.94	None	0.94	0.26
ExtraTreesClassifier	0.94	0.94	None	0.94	0.50
KNeighborsClassifier	0.92	0.92	None	0.92	0.13
ExtraTreeClassifier	0.89	0.88	None	0.89	0.13
DecisionTreeClassifier	0.83	0.87	None	0.83	0.12
GaussianNB	0.86	0.85	None	0.85	0.13
NearestCentroid	0.86	0.81	None	0.86	0.13
NuSVC	0.83	0.79	None	0.84	0.13
AdaBoostClassifier	0.81	0.78	None	0.81	0.43
BernoulliNB	0.81	0.75	None	0.80	0.15
QuadraticDiscriminantAnalysis	0.53	0.58	None	0.52	0.13
DummyClassifier	0.31	0.33	None	0.14	0.13

model_dictionary["RidgeClassifierCV"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV())))

print(classification_report(y_test, model_dictionary["RidgeClassifierCV"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         8
           1       1.00      1.00      1.00        11
           2       1.00      1.00      1.00        17

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36

ConfusionMatrixDisplay.from_estimator(model_dictionary["RidgeClassifierCV"], X_test, y_test)
plt.show()

4 - digits data

top

data = load_digits()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [01:41<00:00,  3.75s/it]

 Elapsed: 101.15918207168579 seconds

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
SVC	0.99	0.99	None	0.99	3.47
LogisticRegression	0.97	0.97	None	0.97	3.67
CalibratedClassifierCV	0.97	0.97	None	0.97	8.78
RandomForestClassifier	0.97	0.97	None	0.97	4.46
LinearDiscriminantAnalysis	0.96	0.96	None	0.96	4.78
LinearSVC	0.96	0.96	None	0.96	4.18
ExtraTreesClassifier	0.96	0.96	None	0.96	3.50
PassiveAggressiveClassifier	0.96	0.96	None	0.96	2.89
KNeighborsClassifier	0.96	0.96	None	0.96	3.60
SGDClassifier	0.96	0.96	None	0.96	2.62
RidgeClassifier	0.94	0.94	None	0.94	4.45
Perceptron	0.94	0.94	None	0.94	4.38
RidgeClassifierCV	0.94	0.94	None	0.93	3.02
NuSVC	0.93	0.93	None	0.93	4.55
BaggingClassifier	0.92	0.92	None	0.92	4.84
LabelPropagation	0.91	0.91	None	0.92	2.63
LabelSpreading	0.91	0.91	None	0.92	3.64
QuadraticDiscriminantAnalysis	0.88	0.88	None	0.88	2.39
NearestCentroid	0.85	0.85	None	0.85	4.27
DecisionTreeClassifier	0.83	0.83	None	0.83	3.36
BernoulliNB	0.82	0.82	None	0.82	2.85
GaussianNB	0.81	0.81	None	0.80	4.63
ExtraTreeClassifier	0.81	0.80	None	0.81	3.35
AdaBoostClassifier	0.39	0.37	None	0.32	5.05
DummyClassifier	0.08	0.10	None	0.01	4.98

model_dictionary["SVC"]

CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))

print(classification_report(y_test, model_dictionary["SVC"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        39
           1       0.94      1.00      0.97        34
           2       1.00      0.97      0.99        36
           3       1.00      1.00      1.00        33
           4       0.95      1.00      0.98        42
           5       1.00      0.95      0.97        37
           6       1.00      1.00      1.00        43
           7       1.00      1.00      1.00        31
           8       1.00      0.95      0.97        37
           9       0.97      1.00      0.98        28

    accuracy                           0.99       360
   macro avg       0.99      0.99      0.99       360
weighted avg       0.99      0.99      0.99       360

ConfusionMatrixDisplay.from_estimator(model_dictionary["SVC"], X_test, y_test)
plt.show()

ConfusionMatrixDisplay.from_estimator(model_dictionary["LogisticRegression"], X_test, y_test)
plt.show()

5 - R examples

top

I hope this is not too technical: please note that installing the R version of nnetsauce will create an environment in the current directory (more precisely, a folder named r-reticulate will be created where you’ll install the package).

Classification

remotes::install_github("Techtonique/nnetsauce_r") # install nnetsauce from GitHub

library(datasets)

set.seed(123)
X <- as.matrix(iris[, 1:4])
y <- as.integer(iris$Species) - 1L

(index_train <- base::sample.int(n = nrow(X),
                                 size = floor(0.8*nrow(X)),
                                 replace = FALSE))
X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train]

obj <- nnetsauce::LazyDeepClassifier(n_layers = 3L)
res <- obj$fit(X_train, X_test, y_train, y_test)
print(res[[1]]) # best accuracy must be 1.0

Regression

X <- MASS::Boston[,-14] # dataset has an ethical problem
y <- MASS::Boston$medv

set.seed(13)
(index_train <- base::sample.int(n = nrow(X),
                                 size = floor(0.8*nrow(X)),
                                 replace = FALSE))
X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train]

obj <- nnetsauce::LazyDeepRegressor(n_layers = 3L, n_clusters=2L)
res <- obj$fit(X_train, X_test, y_train, y_test)
print(res[[1]]) # best RMSE must be close to 2.8, try n_clusters=3L for 2.69

AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Pt.2: multivariate time series forecasting

2023-10-29T00:00:00+00:00

Last week, I talked about an AutoML method for regression and classification implemented in Python package nnetsauce. This week, my post is about the same AutoML method, applied this time to multivariate time series (MTS) forecasting.

In the examples below, keep in mind that VAR (Vector Autoregression) and VECM (Vector Error Correction Model) forecasting models aren’t thoroughly trained. nnetsauce.MTS isn’t really tuned either; this is just a demo. To finish, a probabilistic error metric (instead of the Root Mean Squared Error, RMSE) is better suited for models capturing forecasting uncertainty.

Contents

1 - Install
2 - MTS
2 - 1 nnetsauce.MTS
2 - 2 statsmodels VAR
2 - 3 statsmodels VECM

1 - Install

!pip install git+https://github.com/Techtonique/nnetsauce.git@lazy-predict

import nnetsauce as ns
import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.tsa.api import VAR
from statsmodels.tsa.base.datetools import dates_from_str
from sklearn.linear_model import LassoCV
from statsmodels.tsa.api import VAR
from sklearn.metrics import mean_squared_error
from statsmodels.tsa.vector_ar.vecm import VECM, select_order
from statsmodels.tsa.base.datetools import dates_from_str

2 - MTS

Macro data

# some example data
mdata = sm.datasets.macrodata.load_pandas().data

# prepare the dates index
dates = mdata[['year', 'quarter']].astype(int).astype(str)

quarterly = dates["year"] + "Q" + dates["quarter"]

quarterly = dates_from_str(quarterly)

mdata = mdata[['realgovt', 'tbilrate']]

mdata.index = pd.DatetimeIndex(quarterly)

data = np.log(mdata).diff().dropna()

display(data)

df = data

df.index.rename('date')

idx_train = int(df.shape[0]*0.8)
idx_end = df.shape[0]
df_train = df.iloc[0:idx_train,]
df_test = df.iloc[idx_train:idx_end,]

regr_mts = ns.LazyMTS(verbose=1, ignore_warnings=True, custom_metric=None,
                      lags = 1, n_hidden_features=3, n_clusters=0, random_state=1)
models, predictions = regr_mts.fit(df_train, df_test)
model_dictionary = regr_mts.provide_models(df_train, df_test)

display(models)

	RMSE	MAE	MPL	Time Taken
Model
LassoCV	0.22	0.12	0.06	0.20
ElasticNetCV	0.22	0.12	0.06	0.19
LassoLarsCV	0.22	0.12	0.06	0.08
LarsCV	0.22	0.12	0.06	0.08
DummyRegressor	0.22	0.12	0.06	0.06
ElasticNet	0.22	0.12	0.06	0.07
LassoLars	0.22	0.12	0.06	0.06
Lasso	0.22	0.12	0.06	0.07
ExtraTreeRegressor	0.22	0.14	0.07	0.12
KNeighborsRegressor	0.22	0.12	0.06	0.09
SVR	0.22	0.12	0.06	0.13
HistGradientBoostingRegressor	0.23	0.13	0.06	0.79
NuSVR	0.23	0.13	0.06	0.20
ExtraTreesRegressor	0.24	0.13	0.07	0.87
GradientBoostingRegressor	0.24	0.13	0.07	0.25
RandomForestRegressor	0.26	0.16	0.08	2.06
AdaBoostRegressor	0.28	0.19	0.10	0.45
DecisionTreeRegressor	0.28	0.18	0.09	0.06
BaggingRegressor	0.28	0.19	0.10	0.20
GaussianProcessRegressor	8.26	5.90	2.95	0.17
BayesianRidge	11774168792.68	3129885640.50	1564942820.25	0.08
TweedieRegressor	1066305878860.67	263521546472.00	131760773236.00	0.12
LassoLarsIC	10841414830181.57	2665022282527.50	1332511141263.75	0.08
PassiveAggressiveRegressor	200205325611502239744.00	40689888595970097152.00	20344944297985048576.00	0.17
SGDRegressor	1383750703550277812748288.00	269310062772019343130624.00	134655031386009671565312.00	0.13
LinearSVR	6205416599219790202011648.00	1189414936788171753521152.00	594707468394085876760576.00	0.06
OrthogonalMatchingPursuitCV	18588484112627753604349952.00	3542235944300533382119424.00	1771117972150266691059712.00	0.23
OrthogonalMatchingPursuit	18588484112627753604349952.00	3542235944300533382119424.00	1771117972150266691059712.00	0.20
HuberRegressor	50554040814422644093913571262464.00	9061839427591544042390898606080.00	4530919713795772021195449303040.00	0.09
RidgeCV	1788858960353426286932811384356864.00	317940467527547291488891451736064.00	158970233763773645744445725868032.00	0.23
RANSACRegressor	352805899757804849079011831705501696.00	61914238966205227684888230708117504.00	30957119483102613842444115354058752.00	1.44
LinearRegression	13408548756595947978849418193194188800.00	2316276205868561893698967459810246656.00	1158138102934280946849483729905123328.00	0.06
TransformedTargetRegressor	13408548756595947978849418193194188800.00	2316276205868561893698967459810246656.00	1158138102934280946849483729905123328.00	0.11
Lars	13408548756596845228481163425784791040.00	2316276205868715960905471081985343488.00	1158138102934357980452735540992671744.00	0.08
Ridge	27935786184657480745080678989281886208.00	4824713257018197525713060327109689344.00	2412356628509098762856530163554844672.00	0.12
KernelRidge	27935786184685139645570846501298503680.00	4824713257022931107816326787730767872.00	2412356628511465553908163393865383936.00	0.09
MLPRegressor	64247413650209509837810706524366567768365621314...	10088348458681313437051396009759695398571807517...	50441742293406567185256980048798476992859037587...	0.42

model_dictionary['LassoCV']

MTS(n_clusters=0, n_hidden_features=3, obj=LassoCV(random_state=1), seed='mean')

2 - 1 - `nnetsauce.MTS`

regr = ns.MTS(obj = LassoCV(random_state=1),
              lags = 1, n_hidden_features=3,
              n_clusters=0, replications = 250,
              kernel = "gaussian", verbose = 1)

regr.fit(df_train)

 Adjusting LassoCV to multivariate time series... 
 
100%|██████████| 2/2 [00:00<00:00,  6.22it/s]

 Simulate residuals using gaussian kernel... 

 Best parameters for gaussian kernel: {'bandwidth': 0.04037017258596558}

MTS(kernel='gaussian', n_clusters=0, n_hidden_features=3,
    obj=LassoCV(random_state=1), replications=250, verbose=1)

res = regr.predict(h=df_test.shape[0], level=95)

100%|██████████| 250/250 [00:00<00:00, 3686.16it/s]
100%|██████████| 250/250 [00:00<00:00, 6971.82it/s]

regr.plot("realgovt")
regr.plot("tbilrate")

2 - 2 - VAR

model = VAR(df_train)
results = model.fit(maxlags=5, ic='aic')
lag_order = results.k_ar
VAR_preds = results.forecast(df_train.values[-lag_order:], df_test.shape[0])

results.plot_forecast(steps = df_test.shape[0]);

2 - 3 - VECM

model = VECM(df_train, k_ar_diff=2, coint_rank=2)
vecm_res = model.fit()
vecm_res.gamma.round(4)
vecm_res.summary()
vecm_res.predict(steps=df_test.shape[0])
forecast, lower, upper = vecm_res.predict(df_test.shape[0], 0.05)

vecm_res.plot_forecast(steps = df_test.shape[0])

out-of-sample errors

display([("nnetsauce.MTS+"+models.index[i], models["RMSE"].iloc[i]) for i in range(3)])
display(('VAR', mean_squared_error(df_test.values, VAR_preds, squared=False)))
display(('VECM', mean_squared_error(df_test.values, forecast, squared=False)))

[('nnetsauce.MTS+LassoCV', 0.22102547609924011),
 ('nnetsauce.MTS+ElasticNetCV', 0.22103106562991648),
 ('nnetsauce.MTS+LassoLarsCV', 0.22103468506703655)]
('VAR', 0.22128770514262763)
('VECM', 0.22170093788693065)

AutoML in nnetsauce (randomized and quasi-randomized nnetworks)

2023-10-22T00:00:00+00:00

Content:

Installing nnetsauce for Python
Classification
Regression

Disclaimer: I have no affiliation with the lazypredict project.

A few days ago, I stumbled accross a cool Python package called lazypredict. Pretty well-designed, working, and relying on scikit-learn’s design.

With lazypredict, you can rapidly have an idea of which scikit-learn model (can also work with xgboost’s and lightgbm’s scikit-learn-like interfaces) performs best on a given data set, with a little preprocessing, and without hyperparameters’ tuning (this is important to note).

I thought something similar could be beneficial to nnetsauce’s classes CustomClassifier, CustomRegressor (see detailed examples below, and interact with the graphs) and MTS. For now.

So far, in nnetsauce (Python version), I adapted the lazy prediction feature to regression (CustomRegressor) and classification (CustomClassifier). Not for univariate and multivariate time series forecasting (MTS) yet. You can try it from a GitHub branch.

2 - Installation

!pip install git+https://github.com/Techtonique/nnetsauce.git@lazy-predict

2 - Classification

2 - 1 Loading the Dataset

import nnetsauce as ns
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X = data.data
y= data.target

2 - 2 Building the classification model using LazyPredict

from sklearn.model_selection import train_test_split

# split the data
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=123)

# build the lazyclassifier
clf = ns.LazyClassifier(verbose=0, ignore_warnings=True,
                        custom_metric=None,
                        n_hidden_features=10,
                        col_sample=0.9)

# fit it
models, predictions = clf.fit(X_train, X_test, y_train, y_test)

100%|██████████| 27/27 [00:09<00:00,  2.71it/s]

# print the best models
display(models)

	Accuracy	Balanced Accuracy	ROC AUC	F1 Score	Time Taken
Model
LogisticRegression	0.99	0.99	0.99	0.99	0.69
LinearSVC	0.98	0.98	0.98	0.98	0.33
SGDClassifier	0.98	0.98	0.98	0.98	0.19
Perceptron	0.98	0.98	0.98	0.98	0.15
LabelPropagation	0.98	0.98	0.98	0.98	0.33
LabelSpreading	0.98	0.98	0.98	0.98	0.43
SVC	0.98	0.98	0.98	0.98	0.16
RandomForestClassifier	0.98	0.98	0.98	0.98	0.66
ExtraTreesClassifier	0.98	0.98	0.98	0.98	0.40
KNeighborsClassifier	0.98	0.98	0.98	0.98	0.34
DecisionTreeClassifier	0.97	0.97	0.97	0.97	0.53
PassiveAggressiveClassifier	0.97	0.97	0.97	0.97	0.21
LinearDiscriminantAnalysis	0.97	0.96	0.96	0.97	0.19
CalibratedClassifierCV	0.97	0.96	0.96	0.97	0.24
AdaBoostClassifier	0.96	0.96	0.96	0.96	1.31
BaggingClassifier	0.95	0.95	0.95	0.95	0.63
RidgeClassifier	0.96	0.94	0.94	0.96	0.27
RidgeClassifierCV	0.96	0.94	0.94	0.96	0.18
QuadraticDiscriminantAnalysis	0.95	0.94	0.94	0.95	0.81
ExtraTreeClassifier	0.94	0.93	0.93	0.94	0.12
NuSVC	0.94	0.91	0.91	0.94	0.29
GaussianNB	0.93	0.91	0.91	0.93	0.17
BernoulliNB	0.92	0.90	0.90	0.92	0.31
NearestCentroid	0.92	0.89	0.89	0.92	0.24
DummyClassifier	0.64	0.50	0.50	0.50	0.27

model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)

model_dictionary['LogisticRegression']

model_dictionary['LogisticRegression'].get_params()

{'memory': None,
 'steps': [('preprocessor',
   ColumnTransformer(transformers=[('numeric',
                                    Pipeline(steps=[('imputer', SimpleImputer()),
                                                    ('scaler', StandardScaler())]),
                                    Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
               17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
              dtype='int64')),
                                   ('categorical_low',
                                    Pipeline(steps=[('imputer',
                                                     SimpleImputer(fill_value='missing',
                                                                   strategy='constant')),
                                                    ('encoding',
                                                     OneHotEncoder(handle_unknown='ignore',
                                                                   sparse=False))]),
                                    Int64Index([], dtype='int64')),
                                   ('categorical_high',
                                    Pipeline(steps=[('imputer',
                                                     SimpleImputer(fill_value='missing',
                                                                   strategy='constant')),
                                                    ('encoding',
                                                     OrdinalEncoder())]),
                                    Int64Index([], dtype='int64'))])),
  ('classifier',
   CustomClassifier(col_sample=0.9, n_hidden_features=10,
                    obj=LogisticRegression(random_state=42)))],
 'verbose': False,
 'preprocessor': ColumnTransformer(transformers=[('numeric',
                                  Pipeline(steps=[('imputer', SimpleImputer()),
                                                  ('scaler', StandardScaler())]),
                                  Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
             17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
            dtype='int64')),
                                 ('categorical_low',
                                  Pipeline(steps=[('imputer',
                                                   SimpleImputer(fill_value='missing',
                                                                 strategy='constant')),
                                                  ('encoding',
                                                   OneHotEncoder(handle_unknown='ignore',
                                                                 sparse=False))]),
                                  Int64Index([], dtype='int64')),
                                 ('categorical_high',
                                  Pipeline(steps=[('imputer',
                                                   SimpleImputer(fill_value='missing',
                                                                 strategy='constant')),
                                                  ('encoding',
                                                   OrdinalEncoder())]),
                                  Int64Index([], dtype='int64'))]),
 'classifier': CustomClassifier(col_sample=0.9, n_hidden_features=10,
                  obj=LogisticRegression(random_state=42)),
 'preprocessor__n_jobs': None,
 'preprocessor__remainder': 'drop',
 'preprocessor__sparse_threshold': 0.3,
 'preprocessor__transformer_weights': None,
 'preprocessor__transformers': [('numeric',
   Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler())]),
   Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
               17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
              dtype='int64')),
  ('categorical_low',
   Pipeline(steps=[('imputer',
                    SimpleImputer(fill_value='missing', strategy='constant')),
                   ('encoding',
                    OneHotEncoder(handle_unknown='ignore', sparse=False))]),
   Int64Index([], dtype='int64')),
  ('categorical_high',
   Pipeline(steps=[('imputer',
                    SimpleImputer(fill_value='missing', strategy='constant')),
                   ('encoding', OrdinalEncoder())]),
   Int64Index([], dtype='int64'))],
 'preprocessor__verbose': False,
 'preprocessor__verbose_feature_names_out': True,
 'preprocessor__numeric': Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler())]),
 'preprocessor__categorical_low': Pipeline(steps=[('imputer',
                  SimpleImputer(fill_value='missing', strategy='constant')),
                 ('encoding',
                  OneHotEncoder(handle_unknown='ignore', sparse=False))]),
 'preprocessor__categorical_high': Pipeline(steps=[('imputer',
                  SimpleImputer(fill_value='missing', strategy='constant')),
                 ('encoding', OrdinalEncoder())]),
 'preprocessor__numeric__memory': None,
 'preprocessor__numeric__steps': [('imputer', SimpleImputer()),
  ('scaler', StandardScaler())],
 'preprocessor__numeric__verbose': False,
 'preprocessor__numeric__imputer': SimpleImputer(),
 'preprocessor__numeric__scaler': StandardScaler(),
 'preprocessor__numeric__imputer__add_indicator': False,
 'preprocessor__numeric__imputer__copy': True,
 'preprocessor__numeric__imputer__fill_value': None,
 'preprocessor__numeric__imputer__keep_empty_features': False,
 'preprocessor__numeric__imputer__missing_values': nan,
 'preprocessor__numeric__imputer__strategy': 'mean',
 'preprocessor__numeric__imputer__verbose': 'deprecated',
 'preprocessor__numeric__scaler__copy': True,
 'preprocessor__numeric__scaler__with_mean': True,
 'preprocessor__numeric__scaler__with_std': True,
 'preprocessor__categorical_low__memory': None,
 'preprocessor__categorical_low__steps': [('imputer',
   SimpleImputer(fill_value='missing', strategy='constant')),
  ('encoding', OneHotEncoder(handle_unknown='ignore', sparse=False))],
 'preprocessor__categorical_low__verbose': False,
 'preprocessor__categorical_low__imputer': SimpleImputer(fill_value='missing', strategy='constant'),
 'preprocessor__categorical_low__encoding': OneHotEncoder(handle_unknown='ignore', sparse=False),
 'preprocessor__categorical_low__imputer__add_indicator': False,
 'preprocessor__categorical_low__imputer__copy': True,
 'preprocessor__categorical_low__imputer__fill_value': 'missing',
 'preprocessor__categorical_low__imputer__keep_empty_features': False,
 'preprocessor__categorical_low__imputer__missing_values': nan,
 'preprocessor__categorical_low__imputer__strategy': 'constant',
 'preprocessor__categorical_low__imputer__verbose': 'deprecated',
 'preprocessor__categorical_low__encoding__categories': 'auto',
 'preprocessor__categorical_low__encoding__drop': None,
 'preprocessor__categorical_low__encoding__dtype': numpy.float64,
 'preprocessor__categorical_low__encoding__handle_unknown': 'ignore',
 'preprocessor__categorical_low__encoding__max_categories': None,
 'preprocessor__categorical_low__encoding__min_frequency': None,
 'preprocessor__categorical_low__encoding__sparse': False,
 'preprocessor__categorical_low__encoding__sparse_output': True,
 'preprocessor__categorical_high__memory': None,
 'preprocessor__categorical_high__steps': [('imputer',
   SimpleImputer(fill_value='missing', strategy='constant')),
  ('encoding', OrdinalEncoder())],
 'preprocessor__categorical_high__verbose': False,
 'preprocessor__categorical_high__imputer': SimpleImputer(fill_value='missing', strategy='constant'),
 'preprocessor__categorical_high__encoding': OrdinalEncoder(),
 'preprocessor__categorical_high__imputer__add_indicator': False,
 'preprocessor__categorical_high__imputer__copy': True,
 'preprocessor__categorical_high__imputer__fill_value': 'missing',
 'preprocessor__categorical_high__imputer__keep_empty_features': False,
 'preprocessor__categorical_high__imputer__missing_values': nan,
 'preprocessor__categorical_high__imputer__strategy': 'constant',
 'preprocessor__categorical_high__imputer__verbose': 'deprecated',
 'preprocessor__categorical_high__encoding__categories': 'auto',
 'preprocessor__categorical_high__encoding__dtype': numpy.float64,
 'preprocessor__categorical_high__encoding__encoded_missing_value': nan,
 'preprocessor__categorical_high__encoding__handle_unknown': 'error',
 'preprocessor__categorical_high__encoding__unknown_value': None,
 'classifier__a': 0.01,
 'classifier__activation_name': 'relu',
 'classifier__backend': 'cpu',
 'classifier__bias': True,
 'classifier__cluster_encode': True,
 'classifier__col_sample': 0.9,
 'classifier__direct_link': True,
 'classifier__dropout': 0,
 'classifier__n_clusters': 2,
 'classifier__n_hidden_features': 10,
 'classifier__nodes_sim': 'sobol',
 'classifier__obj__C': 1.0,
 'classifier__obj__class_weight': None,
 'classifier__obj__dual': False,
 'classifier__obj__fit_intercept': True,
 'classifier__obj__intercept_scaling': 1,
 'classifier__obj__l1_ratio': None,
 'classifier__obj__max_iter': 100,
 'classifier__obj__multi_class': 'auto',
 'classifier__obj__n_jobs': None,
 'classifier__obj__penalty': 'l2',
 'classifier__obj__random_state': 42,
 'classifier__obj__solver': 'lbfgs',
 'classifier__obj__tol': 0.0001,
 'classifier__obj__verbose': 0,
 'classifier__obj__warm_start': False,
 'classifier__obj': LogisticRegression(random_state=42),
 'classifier__row_sample': 1,
 'classifier__seed': 123,
 'classifier__type_clust': 'kmeans',
 'classifier__type_scaling': ('std', 'std', 'std')}

3 - Regression

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

data = load_diabetes()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

regr = ns.LazyRegressor(verbose=0, ignore_warnings=True, custom_metric=None)
models, predictions = regr.fit(X_train, X_test, y_train, y_test)
model_dictionary = regr.provide_models(X_train, X_test, y_train, y_test)

100%|██████████| 40/40 [00:03<00:00, 12.38it/s]

display(models)

	Adjusted R-Squared	R-Squared	RMSE	Time Taken
Model
LassoLarsIC	0.53	0.59	51.11	0.03
SGDRegressor	0.53	0.58	51.24	0.03
HuberRegressor	0.53	0.58	51.26	0.05
Ridge	0.53	0.58	51.37	0.03
KernelRidge	0.53	0.58	51.37	0.03
RidgeCV	0.53	0.58	51.37	0.03
Lasso	0.52	0.58	51.52	0.03
LassoLars	0.52	0.58	51.52	0.03
LassoCV	0.52	0.58	51.58	0.12
LassoLarsCV	0.52	0.58	51.58	0.05
TransformedTargetRegressor	0.52	0.58	51.62	0.03
LinearRegression	0.52	0.58	51.62	0.03
OrthogonalMatchingPursuitCV	0.52	0.58	51.69	0.05
BayesianRidge	0.52	0.57	51.77	0.03
LinearSVR	0.51	0.57	52.04	0.02
ElasticNetCV	0.51	0.56	52.49	0.08
LarsCV	0.50	0.56	52.79	0.05
PassiveAggressiveRegressor	0.49	0.55	53.39	0.03
GradientBoostingRegressor	0.48	0.54	54.00	0.26
ElasticNet	0.46	0.52	54.92	0.03
BaggingRegressor	0.46	0.52	54.92	0.07
RandomForestRegressor	0.46	0.52	55.07	0.37
HistGradientBoostingRegressor	0.45	0.51	55.42	0.20
ExtraTreesRegressor	0.44	0.51	55.71	0.24
AdaBoostRegressor	0.44	0.51	55.75	0.14
MLPRegressor	0.43	0.50	56.38	0.45
TweedieRegressor	0.42	0.48	57.03	0.03
RANSACRegressor	0.42	0.48	57.14	0.16
KNeighborsRegressor	0.31	0.39	62.10	0.05
OrthogonalMatchingPursuit	0.31	0.38	62.27	0.04
GaussianProcessRegressor	0.19	0.28	67.13	0.05
ExtraTreeRegressor	0.15	0.24	69.09	0.03
SVR	0.12	0.22	69.98	0.04
NuSVR	0.12	0.22	70.14	0.04
DummyRegressor	-0.13	-0.00	79.39	0.03
DecisionTreeRegressor	-0.26	-0.11	83.75	0.03
Lars	-1.95	-1.61	128.28	0.14

model_dictionary["LassoLarsIC"]

model_dictionary["LassoLarsIC"].get_params()

{'memory': None,
 'steps': [('preprocessor',
   ColumnTransformer(transformers=[('numeric',
                                    Pipeline(steps=[('imputer', SimpleImputer()),
                                                    ('scaler', StandardScaler())]),
                                    Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')),
                                   ('categorical_low',
                                    Pipeline(steps=[('imputer',
                                                     SimpleImputer(fill_value='missing',
                                                                   strategy='constant')),
                                                    ('encoding',
                                                     OneHotEncoder(handle_unknown='ignore',
                                                                   sparse=False))]),
                                    Int64Index([], dtype='int64')),
                                   ('categorical_high',
                                    Pipeline(steps=[('imputer',
                                                     SimpleImputer(fill_value='missing',
                                                                   strategy='constant')),
                                                    ('encoding',
                                                     OrdinalEncoder())]),
                                    Int64Index([], dtype='int64'))])),
  ('regressor', CustomRegressor(obj=LassoLarsIC()))],
 'verbose': False,
 'preprocessor': ColumnTransformer(transformers=[('numeric',
                                  Pipeline(steps=[('imputer', SimpleImputer()),
                                                  ('scaler', StandardScaler())]),
                                  Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')),
                                 ('categorical_low',
                                  Pipeline(steps=[('imputer',
                                                   SimpleImputer(fill_value='missing',
                                                                 strategy='constant')),
                                                  ('encoding',
                                                   OneHotEncoder(handle_unknown='ignore',
                                                                 sparse=False))]),
                                  Int64Index([], dtype='int64')),
                                 ('categorical_high',
                                  Pipeline(steps=[('imputer',
                                                   SimpleImputer(fill_value='missing',
                                                                 strategy='constant')),
                                                  ('encoding',
                                                   OrdinalEncoder())]),
                                  Int64Index([], dtype='int64'))]),
 'regressor': CustomRegressor(obj=LassoLarsIC()),
 'preprocessor__n_jobs': None,
 'preprocessor__remainder': 'drop',
 'preprocessor__sparse_threshold': 0.3,
 'preprocessor__transformer_weights': None,
 'preprocessor__transformers': [('numeric',
   Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler())]),
   Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')),
  ('categorical_low',
   Pipeline(steps=[('imputer',
                    SimpleImputer(fill_value='missing', strategy='constant')),
                   ('encoding',
                    OneHotEncoder(handle_unknown='ignore', sparse=False))]),
   Int64Index([], dtype='int64')),
  ('categorical_high',
   Pipeline(steps=[('imputer',
                    SimpleImputer(fill_value='missing', strategy='constant')),
                   ('encoding', OrdinalEncoder())]),
   Int64Index([], dtype='int64'))],
 'preprocessor__verbose': False,
 'preprocessor__verbose_feature_names_out': True,
 'preprocessor__numeric': Pipeline(steps=[('imputer', SimpleImputer()), ('scaler', StandardScaler())]),
 'preprocessor__categorical_low': Pipeline(steps=[('imputer',
                  SimpleImputer(fill_value='missing', strategy='constant')),
                 ('encoding',
                  OneHotEncoder(handle_unknown='ignore', sparse=False))]),
 'preprocessor__categorical_high': Pipeline(steps=[('imputer',
                  SimpleImputer(fill_value='missing', strategy='constant')),
                 ('encoding', OrdinalEncoder())]),
 'preprocessor__numeric__memory': None,
 'preprocessor__numeric__steps': [('imputer', SimpleImputer()),
  ('scaler', StandardScaler())],
 'preprocessor__numeric__verbose': False,
 'preprocessor__numeric__imputer': SimpleImputer(),
 'preprocessor__numeric__scaler': StandardScaler(),
 'preprocessor__numeric__imputer__add_indicator': False,
 'preprocessor__numeric__imputer__copy': True,
 'preprocessor__numeric__imputer__fill_value': None,
 'preprocessor__numeric__imputer__keep_empty_features': False,
 'preprocessor__numeric__imputer__missing_values': nan,
 'preprocessor__numeric__imputer__strategy': 'mean',
 'preprocessor__numeric__imputer__verbose': 'deprecated',
 'preprocessor__numeric__scaler__copy': True,
 'preprocessor__numeric__scaler__with_mean': True,
 'preprocessor__numeric__scaler__with_std': True,
 'preprocessor__categorical_low__memory': None,
 'preprocessor__categorical_low__steps': [('imputer',
   SimpleImputer(fill_value='missing', strategy='constant')),
  ('encoding', OneHotEncoder(handle_unknown='ignore', sparse=False))],
 'preprocessor__categorical_low__verbose': False,
 'preprocessor__categorical_low__imputer': SimpleImputer(fill_value='missing', strategy='constant'),
 'preprocessor__categorical_low__encoding': OneHotEncoder(handle_unknown='ignore', sparse=False),
 'preprocessor__categorical_low__imputer__add_indicator': False,
 'preprocessor__categorical_low__imputer__copy': True,
 'preprocessor__categorical_low__imputer__fill_value': 'missing',
 'preprocessor__categorical_low__imputer__keep_empty_features': False,
 'preprocessor__categorical_low__imputer__missing_values': nan,
 'preprocessor__categorical_low__imputer__strategy': 'constant',
 'preprocessor__categorical_low__imputer__verbose': 'deprecated',
 'preprocessor__categorical_low__encoding__categories': 'auto',
 'preprocessor__categorical_low__encoding__drop': None,
 'preprocessor__categorical_low__encoding__dtype': numpy.float64,
 'preprocessor__categorical_low__encoding__handle_unknown': 'ignore',
 'preprocessor__categorical_low__encoding__max_categories': None,
 'preprocessor__categorical_low__encoding__min_frequency': None,
 'preprocessor__categorical_low__encoding__sparse': False,
 'preprocessor__categorical_low__encoding__sparse_output': True,
 'preprocessor__categorical_high__memory': None,
 'preprocessor__categorical_high__steps': [('imputer',
   SimpleImputer(fill_value='missing', strategy='constant')),
  ('encoding', OrdinalEncoder())],
 'preprocessor__categorical_high__verbose': False,
 'preprocessor__categorical_high__imputer': SimpleImputer(fill_value='missing', strategy='constant'),
 'preprocessor__categorical_high__encoding': OrdinalEncoder(),
 'preprocessor__categorical_high__imputer__add_indicator': False,
 'preprocessor__categorical_high__imputer__copy': True,
 'preprocessor__categorical_high__imputer__fill_value': 'missing',
 'preprocessor__categorical_high__imputer__keep_empty_features': False,
 'preprocessor__categorical_high__imputer__missing_values': nan,
 'preprocessor__categorical_high__imputer__strategy': 'constant',
 'preprocessor__categorical_high__imputer__verbose': 'deprecated',
 'preprocessor__categorical_high__encoding__categories': 'auto',
 'preprocessor__categorical_high__encoding__dtype': numpy.float64,
 'preprocessor__categorical_high__encoding__encoded_missing_value': nan,
 'preprocessor__categorical_high__encoding__handle_unknown': 'error',
 'preprocessor__categorical_high__encoding__unknown_value': None,
 'regressor__a': 0.01,
 'regressor__activation_name': 'relu',
 'regressor__backend': 'cpu',
 'regressor__bias': True,
 'regressor__cluster_encode': True,
 'regressor__col_sample': 1,
 'regressor__direct_link': True,
 'regressor__dropout': 0,
 'regressor__n_clusters': 2,
 'regressor__n_hidden_features': 5,
 'regressor__nodes_sim': 'sobol',
 'regressor__obj__copy_X': True,
 'regressor__obj__criterion': 'aic',
 'regressor__obj__eps': 2.220446049250313e-16,
 'regressor__obj__fit_intercept': True,
 'regressor__obj__max_iter': 500,
 'regressor__obj__noise_variance': None,
 'regressor__obj__normalize': 'deprecated',
 'regressor__obj__positive': False,
 'regressor__obj__precompute': 'auto',
 'regressor__obj__verbose': False,
 'regressor__obj': LassoLarsIC(),
 'regressor__row_sample': 1,
 'regressor__seed': 123,
 'regressor__type_clust': 'kmeans',
 'regressor__type_scaling': ('std', 'std', 'std')}

Version v0.14.0 of nnetsauce for R and Python

2023-10-16T00:00:00+00:00

Version v0.14.0 of nnetsauce is now available for R (hopefully a rapid installation) and Python on GitHub, PyPI and conda. It’s been mainly tested on Linux and macOS. For Windows users, you can try to install of course, but if it doesn’t work, please use WSL2.

NEWS

update and align as much as possible with R version (new plotting function for multivariate time series (MTS), plot.MTS, is not S3, but it’s complicated)

# 0 - install packages ----------------------------------------------------

#utils::install.packages("remotes")
remotes::install_github("Techtonique/nnetsauce/R-package", force = TRUE)

# 1 - ENET simulations ----------------------------------------------------

obj <- nnetsauce::sklearn$linear_model$ElasticNet()
obj2 <- nnetsauce::MTS(obj, 
                       start_input = start(fpp::vn), 
                       frequency_input = frequency(fpp::vn),
                       kernel = "gaussian", replications = 100L)
X <- data.frame(fpp::vn)
obj2$fit(X)
obj2$predict(h = 10L)
typeof(obj2)

par(mfrow=c(2, 2))
plot.MTS(obj2, selected_series = "Sydney")
plot.MTS(obj2, selected_series = "Melbourne")
plot.MTS(obj2, selected_series = "NSW")
plot.MTS(obj2, selected_series = "BrisbaneGC")

# 2 - Bayesian Ridge ----------------------------------------------------

obj <- nnetsauce::sklearn$linear_model$BayesianRidge()
obj2 <- nnetsauce::MTS(obj,
                       start_input = start(fpp::vn), 
                       frequency_input = frequency(fpp::vn))
X <- data.frame(fpp::vn)
obj2$fit(X)
obj2$predict(h = 10L, return_std = TRUE)

par(mfrow=c(2, 2))
plot.MTS(obj2, selected_series = "Sydney")
plot.MTS(obj2, selected_series = "Melbourne")
plot.MTS(obj2, selected_series = "NSW")
plot.MTS(obj2, selected_series = "BrisbaneGC")

colored graphics for Python class MTS

# !pip install nnetsauce —upgrade

import nnetsauce as ns
import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge, BayesianRidge
from sklearn.ensemble import RandomForestRegressor
from time import time

url = "https://raw.githubusercontent.com/thierrymoudiki/mts-data/master/heater-ice-cream/ice_cream_vs_heater.csv"

df = pd.read_csv(url)

# ice cream vs heater (I don't own the copyright)
df.set_index('Month', inplace=True)
df.index.rename('date')

df = df.pct_change().dropna()

idx_train = int(df.shape[0]*0.8)
idx_end = df.shape[0]
df_train = df.iloc[0:idx_train,]

regr3 = Ridge()
obj_MTS3 = ns.MTS(regr3, lags = 4, n_hidden_features=7, #IRL, must be tuned
                  replications=50, kernel='gaussian',
                  seed=24, verbose = 1)
start = time()
obj_MTS3.fit(df_train)
print(f"Elapsed {time()-start} s")

# predict 
obj_MTS3.predict()

obj_MTS3.plot("heater")
obj_MTS3.plot("ice cream")

An infinity of time series forecasting models in nnetsauce (Part 2 with uncertainty quantification)

2023-09-25T00:00:00+00:00

Update 2023-09-27: conda users, v0.13.0 of nnetsauce is now available for Linux and macOS. For Windows, please use WSL2.

As I said a few years ago, nnetsauce.MTS is a family of univariate/multivariate time series forecasting models that I was supposed to present at R/Finance 2020 (this post is 100% Python) in Chicago, IL. But the COVID-19 decided differently.

The more I thought about it, namely nnetsauce.MTS (still doesn’t have a more glamorous name), the more I thought ‘It’s kind of weird…‘. Why? Because in the statistical learning procedure, all the input time series models share the same hyperparameters. Today, I think nnetsauce.MTS it’s not quite different from a multi-output regression (regression models for predicting multiple responses, based on covariates), and it seems to be working well empirically, as shown below. No grandiose state-of-the-art (SOTA for the snobs) claims here, but I think that with the high number of possible model inputs (actually, any regression Estimator having fit and predict methods), you could cover a lot of space.

You can read this post if you want to understand how it works (but avoid the ugly graph at the end, the ones presented here are hopefully more compelling). Pull requests and (constructive) discussions are welcome as usual.

In the examples presented here, I focus on uncertainty quantification:

simulation-based, using Kernel Density Estimation of the residuals
a Bayesian approach, even though ‘Bayesianism’ is in hot water these days. Its subjectivity? I must admit that choosing a prior distribution is quite an interesting (interpret ‘interesting’ here as you want, I mean both good and bad) experiment. But ‘Bayesianism’, Gaussian Processes in particular, works quite well in settings such as hyperparameters tuning (I hope the code still works) for example

Conformal prediction, the new cool kid on the uncertainty quantification block, will certainly be included in future versions of the tool.

Contents

0 - Install and import packages + get data
1 - Simulation-based forecasting using Kernel Density Estimation
1 - 1 With Ridge regression
1 - 2 With Random Forest
2 - Bayesian Forecasting
Appendix

You can also download this notebook from GitHub, which follows the same plan.

0 - Install and import packages + get data

Installing nnetsauce (v0.13.0) with pip:

pip install nnetsauce

Installing nnetsauce (v0.13.0) using conda:

conda install -c conda-forge nnetsauce

Installing from GitHub:

pip install git+https://github.com/Techtonique/nnetsauce.git

Import the packages in Python:

import nnetsauce as ns
import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge, BayesianRidge
from sklearn.ensemble import RandomForestRegressor
from time import time

Get data:

url = "https://raw.githubusercontent.com/thierrymoudiki/mts-data/master/heater-ice-cream/ice_cream_vs_heater.csv"

df = pd.read_csv(url)

# ice cream vs heater (I don't own the copyright)
df.set_index('Month', inplace=True) 
df.index.rename('date')

df = df.pct_change().dropna()

idx_train = int(df.shape[0]*0.8)
idx_end = df.shape[0]
df_train = df.iloc[0:idx_train,]

1 - Simulation-based forecasting using Kernel Density Estimation

1 - 1 With Ridge regression

regr3 = Ridge()
obj_MTS3 = ns.MTS(regr3, lags = 3, n_hidden_features=7, #IRL, must be tuned
                  replications=50, kernel='gaussian',
                  seed=24, verbose = 1)
start = time()
obj_MTS3.fit(df_train)
print(f"Elapsed {time()-start} s")

res = obj_MTS3.predict(h=15)
print("\n")
print(f" Predictive simulations #10: \n{obj_MTS3.sims_[9]}")
print("\n")
print(f" Predictive simulations #25: \n{obj_MTS3.sims_[24]}")

obj_MTS3.plot("heater")
obj_MTS3.plot("ice cream")

1 - 2 With Random Forest

regr3 = RandomForestRegressor(n_estimators=250)
obj_MTS3 = ns.MTS(regr3, lags = 3, n_hidden_features=7, #IRL, must be tuned
                  replications=50, kernel='gaussian',
                  seed=24, verbose = 1)
start = time()
obj_MTS3.fit(df_train)
print(f"Elapsed {time()-start} s")

res = obj_MTS3.predict(h=15)
print("\n")
print(f" Predictive simulations #10: \n{obj_MTS3.sims_[9]}")
print("\n")
print(f" Predictive simulations #25: \n{obj_MTS3.sims_[24]}")

obj_MTS3.plot("heater")
obj_MTS3.plot("ice cream")

2 - Bayesian Forecasting

regr4 = BayesianRidge()
obj_MTS4 = ns.MTS(regr4, lags = 3, n_hidden_features=7, #IRL, must be tuned
                  seed=24)
start = time()
obj_MTS4.fit(df_train)
print(f"\n\n Elapsed {time()-start} s")

res = obj_MTS4.predict(h=15, return_std=True)

obj_MTS4.plot("heater")
obj_MTS4.plot("ice cream")

Appendix

How does this family of time series forecasting models works?

A new version of nnetsauce (randomized and quasi-randomized 'neural' networks)

2023-04-02T00:00:00+00:00

Content:

nnetsauce's new version
Installing nnetsauce for Python
About nnetsauce for R

nnetsauce’s new version

A new version of nnetsauce, v0.12.0, is available on PyPI and for conda. It’s been mostly tested on Linux and macOS platforms. For Windows users: you can use the Windows Subsystem for Linux in case it doesn’t work directly on your computer.

As a reminder, nnetsauce does Statistical/Machine Learning (regression, classification, and time series forecasting for now) using randomized and quasi-randomized neural networks layers. More precisely, every model in nnetsauce is based on components g(XW + b), where:

X is a matrix containing explanatory variables and optional clustering information. Clustering the inputs helps in taking into account data’s heterogeneity before model fitting.
W creates new, additional explanatory variables from X. W can be drawn from various random and quasi-random sequences.
b is an optional bias parameter.
g is an activation function such as ReLU or the hyperbolic tangent, that makes the combination of explanatory variables – through W – nonlinear.

Examples of use of nnetsauce are available on GitHub, here (including R Markdown examples) and here.

v0.12.0 is an important release, because it’s totally written in Python (using numpy, scipy, jax, and scikit-learn), and doesn’t use C++ nor Cython anymore. Because of this, nnetsauce is faster to install, and easier to maintain.

If you like using nnetsauce, do not hesitate to star the repo or submit a pull request!

Installing nnetsauce for Python

1st method: by using pip at the command line for the stable version

pip install nnetsauce

2nd method: using conda (Linux and macOS only for now)

conda install -c conda-forge nnetsauce

3rd method: from Github, for the development version

pip install git+https://github.com/Techtonique/nnetsauce.git

or in a virtual environment:

git clone https://github.com/Techtonique/nnetsauce.git
cd nnetsauce
make install

About nnetsauce for R

The R version is discontinued. Well, ‘discontinued’ until I finally wrap my head around it… If you’re interested in solving this issue, and therefore, using nnetsauce for R, everything happens in this R script. You can submit a pull request (and star the repo ;) )!

New version of nnetsauce -- various quasi-randomized networks

2022-02-12T00:00:00+00:00

A new version of nnetsauce, v0.10.0, is available on Pypi (for Python) and GitHub (for R). To those who’ve never heard about nnetsauce: it’s a package for supervised learning (as of February 2022, you can solve regression, classification, and time series forecasting problems with nnetsauce) based on various combinations of components $g(XW+b)$, with:

$X$, a matrix of explanatory variables or multivariate (univariate works too, but hasn’t been tested enough yet) time series
$W$, a matrix which contains quasirandom numbers, that help in achieving a kind of automated feature engineering ($XW$)
$b$, a bias term
$g$, an activation function, used for model nonlinearity (otherwise, without it, the model would be linear)

For example, here is how nnetsauce can be used to create a nonlinear model from a linear model:

In nnetsauce v0.10.0, the most important change is a – potentially breaking – change in the API: classes’ attributes (mostly, computed in method fit) which do not belong to the interface have been renamed with a suffix “_”. As in scikit-learn.

Multiple Python examples can be found on GitHub, along with notebooks. Here is an example of use of the package in R (on Ubuntu):

library(devtools)
devtools::install_github("Techtonique/nnetsauce/R-package")
library(nnetsauce)

set.seed(123)
(n <- nrow(iris))
(index_train <- sample.int(n, size = floor(0.8*n), replace = FALSE))

X_train <- as.matrix(iris[index_train, 1:4])
y_train <- as.integer(iris[index_train, 5]) - 1L
X_test <- as.matrix(iris[-index_train, 1:4])
y_test <- as.integer(iris[-index_train, 5]) - 1L

obj <- nnetsauce::Ridge2MultitaskClassifier()
print(obj$get_params())
obj$fit(X_train, y_train)
print(obj$score(X_test, y_test)) # accuracy

R session info:

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] nnetsauce_0.10.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8       magrittr_2.0.2   rappdirs_0.3.3   munsell_0.5.0    colorspace_2.0-2
 [6] here_1.0.1       lattice_0.20-45  R6_2.5.1         rlang_1.0.1      fansi_1.0.2     
[11] tools_4.1.2      grid_4.1.2       gtable_0.3.0     png_0.1-7        utf8_1.2.2      
[16] cli_3.1.1        ellipsis_0.3.2   rprojroot_2.0.2  tibble_3.1.6     lifecycle_1.0.1 
[21] crayon_1.4.2     Matrix_1.3-4     ggplot2_3.3.5    vctrs_0.3.8      glue_1.6.1      
[26] compiler_4.1.2   pillar_1.7.0     scales_1.1.1     reticulate_1.24  jsonlite_1.7.3  
[31] pkgconfig_2.0.3 

Tuning and interpreting LSBoost

2021-11-15T00:00:00+00:00

There is a plethora of Automated Machine Learning tools in the wild, implementing Machine Learning (ML) pipelines from data cleaning to model validation. In this post, the input data set is already cleaned and pre-processed (diabetes dataset); the ML model is already chosen too, mlsauce’s LSBoost. We are going to focus on two important steps of a ML pipeline:

LSBoost’s hyperparameter tuning with GPopt on diabetes data
Interpretation of LSBoost’s output using the-teller’s new version, 0.7.0. It’s worth mentioning that LSBoost, which is nonlinear, is interpretable as a linear model wherever its activation functions can be differentiated. This requires some calculus (but no calculus today, hence the-teller :) ).

Installing and importing packages

Install packages from PyPI:

pip install mlsauce
pip install GPopt
pip install the-teller==0.7.0
pip install matplotlib==3.1.3

Python packages for the demo:

import GPopt as gp 
import mlsauce as ms
import numpy as np
import pandas as pd
import seaborn as sns
import teller as tr
import matplotlib.pyplot as plt
import matplotlib.style as style
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split, cross_val_score
from time import time

Objective function to be minimized (for hyperparameter tuning)

# Number of boosting iterations (global variable, dangerous)
n_estimators = 250

def lsboost_cv(X_train, y_train, learning_rate=0.1, 
               n_hidden_features=5, reg_lambda=0.1, 
               dropout=0, tolerance=1e-4, 
               col_sample=1, seed=123):

  estimator = ms.LSBoostRegressor(n_estimators=n_estimators, 
                                   learning_rate=learning_rate,
                                   n_hidden_features=np.int(n_hidden_features), 
                                   reg_lambda=reg_lambda,
                                   dropout=dropout,
                                   tolerance=tolerance,
                                   col_sample=col_sample,
                                   seed=seed, verbose=0)

  return -cross_val_score(estimator, X_train, y_train,
                          scoring='neg_root_mean_squared_error', cv=5, n_jobs=-1).mean()

def optimize_lsboost(X_train, y_train):

  def crossval_objective(x):

    return lsboost_cv(            
      X_train=X_train, 
      y_train=y_train,
      learning_rate=x[0],
      n_hidden_features=np.int(x[1]), 
      reg_lambda=x[2], 
      dropout=x[3],
      tolerance=x[4],
      col_sample=x[5])

  gp_opt = gp.GPOpt(objective_func=crossval_objective, 
                      lower_bound = np.array([0.001, 5, 1e-2, 0.1, 1e-6, 0.5]), 
                      upper_bound = np.array([0.4, 250, 1e4, 0.8, 0.1, 0.999]),
                      n_init=10, n_iter=190, seed=123)    
  return {'parameters': gp_opt.optimize(verbose=2, abs_tol=1e-3), 'opt_object':  gp_opt}

Hyperparameter tuning on diabetes data

In the diabetes dataset, the response is “a quantitative measure of disease progression one year after baseline”. The explanatory variables are:

age: age in years
sex
bmi: body mass index
bp: average blood pressure
s1: tc, total serum cholesterol
s2: ldl, low-density lipoproteins
s3: hdl, high-density lipoproteins
s4: tch, total cholesterol / HDL
s5: ltg, possibly log of serum triglycerides level
s6: glu, blood sugar level

# load dataset
dataset = load_diabetes()
X = dataset.data
y = dataset.target

# split data into training test and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    test_size=0.1, random_state=13)

# Bayesian optimization for hyperparameters tuning                                                    
res = optimize_lsboost(X_train, y_train)  
res

{'opt_object': <GPopt.GPOpt.GPOpt.GPOpt at 0x7f550e5c5f50>,
 'parameters': (array([1.53620422e-01, 6.20779419e+01, 8.39242559e+02, 1.74212646e-01, 5.48527464e-02, 7.15906433e-01]),
  53.61909741088658)}

Adjusting LSBoost to diabetes data (training set) and obtaining predictions

# _best_ hyperparameters
parameters = res["parameters"][0]

# Adjusting LSBoost to diabetes data (training set)
estimator = ms.LSBoostRegressor(n_estimators=n_estimators,
                                   learning_rate=parameters[0],
                                   n_hidden_features=np.int(parameters[1]), 
                                   reg_lambda=parameters[2], 
                                   dropout=parameters[3],  
                                   tolerance=parameters[4], 
                                   col_sample=parameters[5],
                                   seed=123, verbose=1).fit(X_train, y_train)

# predict on test set
err = estimator.predict(X_test) - y_test
print(f"\n\n Test set RMSE: {np.sqrt(np.mean(np.square(err)))}")

100%|██████████| 250/250 [00:01<00:00, 132.50it/s]

 Test set RMSE: 55.92500853500942

Create an Explainer object in order to understand `LSBoost` decisions

As a reminder, the-teller computes changes (effects) in the response (variable to be explained), consecutive to a small change in an explanatory variable.

# creating an Explainer object
explainer = tr.Explainer(obj=estimator)

# fitting the Explainer to unseen data
explainer.fit(X_test, y_test, X_names=dataset.feature_names, method="avg")

Heterogeneity of marginal effects:

# heterogeneity because 45 patients in test set => a distribution of effects
explainer.summary() 

           mean         std      median         min         max
bmi  556.001858  198.440761  498.042418  295.134632  877.900389
s5   502.361989   56.518532  488.352521  423.339630  663.398877
bp   256.974826  121.099501  245.205494   83.019164  495.913721
s4   190.995503   69.881801  185.163689   49.870049  356.093240
s6    72.047634  100.701186   76.269634  -68.037669  229.263444
age   55.482125  185.000373   61.218433 -174.677003  329.485983
s2    -8.097623   49.166848  -10.127223  -78.075175  104.572880
s1  -141.735836   72.327037 -115.976202 -292.320955   -6.694544
s3  -146.470803  164.826337 -196.285307 -357.895526  132.102133
sex -234.702770  162.564859 -314.707386 -415.665287   24.017851

Visualizing the average effects (new in version 0.7.0):

explainer.plot(what="average_effects")

Visualizing the distribution (heterogeneity) of effects (new in version 0.7.0):

explainer.plot(what="hetero_effects")

If you’re interested in obtaining all the individual effects, for each patient, then type:

print(explainer.get_individual_effects())

Classification using linear regression

2021-09-26T00:00:00+00:00

In this post, I illustrate classification using linear regression, as implemented in Python/R package nnetsauce, and more precisely, in nnetsauce’s MultitaskClassifier. If you’re not interested in reading about the model description, you can jump directly to the 2nd section, “Two examples in Python”. In addition, the source code is relatively self-explanatory.

Model description

Chapter 4 of Elements of Statistical Learning (ESL), at section 4.2 Linear Regression of an Indicator Matrix, describes classification using linear regression pretty well. Let $K \in \mathbb{N}$ be the number of classes and $y \in \mathbb{N}^n$ with values in $\lbrace 1, \ldots, K \rbrace$ be the variable to be explained. An indicator response matrix $\textbf{Y} \in \mathbb{N}^{n \times K }$, containing only 0’s and 1’s, can be obtained from $y$. Each row of $\textbf{Y}$ shall contain a single 1 – in the column corresponding to the class where the example belongs, and 0’s elsewhere.

Now, let $\textbf{X} \in \mathbb{R}^{n \times p }$ be the set of explanatory variables for $y$ and $\textbf{Y}$, with examples in rows, and characteristics in columns. ESL applies $K$ least squares models to $\textbf{X}$, for each column of $\textbf{Y}$. The regression’s predicted values can be interpreted as raw estimates of probabilities, because the least squares’ solution is a conditional expectation. And for $G$, a random variable describing the class, we have:

\[\mathbb{E} \left[ \mathbb{1}_{ G = k } \vert X = x \right] = \mathbb{P} \left[ G = k \vert X = x \right]\]

The difference between nnetsauce’s MultitaskClassifier and the model described in ESL is:

Any model possessing methods fit and predict can be used in lieu of a linear regression of $\textbf{Y}$ on $\textbf{X}$
the set of covariates include the original covariates, $\textbf{X}$, plus nonlinear transformations of $\textbf{X}$, $h(\textbf{X})$, as done in Quasi-Randomized Networks. Having $h(\textbf{X})$ as additional explanatory variables enhances the models’ flexibility; the model is no longer linear.
If for each $k \in \lbrace 1, \ldots, K \rbrace$, $\hat{f}_k(x)$ is the regression’s predicted value for class $k$ and an observation characterized by $x$, nnetsauce’s MultitaskClassifier obtains probabilities that an observation characterized by $x$ belongs to class $k$ as:

\[\hat{p}_k(x) := \frac{expit \left( \hat{f}_k(x) \right)}{\sum_{i=1}^K expit \left( \hat{f}_k(x) \right)}\]

Where we have $expit := \frac{1}{1 + exp(-x)}$. $x \mapsto expit(x)$ is strictly increasing, hence it preserves the ordering of linear regression’s predictions. $x \mapsto expit(x)$ is also bounded in $[0, 1]$, which helps in avoiding overflows. I divide $expit \left( \hat{f}_k(x) \right)$ by $\sum_{i=1}^K expit \left( \hat{f}_k(x) \right)$, so that the probabilities add up to 1. And to finish, the class predicted for an example characterized by $x$ is:

\[argmax_{k \in \lbrace 1, \ldots, K \rbrace} \hat{p}_k(x)\]

Two examples in Python

Currently, installing nnetsauce from Pypi doesn’t work – and I’m working on fixing it. However, you can install nnetsauce from GitHub as follows:

pip install git+https://github.com/Techtonique/nnetsauce.git

Import the packages required for the 2 examples.

import nnetsauce as ns
import numpy as np
from sklearn.datasets import  load_wine, load_iris
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics
from time import time

1. Classification of iris dataset:

dataset = load_iris()
Z = dataset.data
t = dataset.target

# training set (80%) and test set (20%)
X_train, X_test, y_train, y_test = train_test_split(Z, t, test_size=0.2, 
                                                    random_state=143)

# Linear Regression is used here
regr3 = LinearRegression()

# `n_hidden_features` makes the model nonlinear
# `n_clusters` takes into account heterogeneity
fit_obj3 = ns.MultitaskClassifier(regr3, n_hidden_features=5, 
                                  n_clusters=2, type_clust="gmm")

# Adjust the model
start = time()
fit_obj3.fit(X_train, y_train)
print(f"Elapsed {time() - start}") 

# Classification report
start = time()
preds = fit_obj3.predict(X_test)
print(f"Elapsed {time() - start}") 
print(metrics.classification_report(preds, y_test))

Elapsed 0.021012067794799805
Elapsed 0.0010943412780761719
              precision    recall  f1-score   support

           0       1.00      1.00      1.00        12
           1       1.00      1.00      1.00         5
           2       1.00      1.00      1.00        13

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

2. Classification of wine dataset:

dataset = load_wine()
Z = dataset.data
t = dataset.target

# training set (80%) and test set (20%)
X_train, X_test, y_train, y_test = train_test_split(Z, t, test_size=0.2, 
                                                    random_state=143)

# Linear Regression is used here 
regr4 = LinearRegression()

# `n_hidden_features` makes the model nonlinear
# `n_clusters` takes into account heterogeneity
fit_obj4 = ns.MultitaskClassifier(regr4, n_hidden_features=5, 
                                  n_clusters=2, type_clust="gmm")

# Adjust the model
start = time()
fit_obj4.fit(X_train, y_train)
print(f"Elapsed {time() - start}") 

# Classification report
start = time()
preds = fit_obj4.predict(X_test)
print(f"Elapsed {time() - start}") 
print(metrics.classification_report(preds, y_test))

Elapsed 0.019229650497436523
Elapsed 0.001451253890991211
              precision    recall  f1-score   support

           0       1.00      1.00      1.00        16
           1       1.00      1.00      1.00        11
           2       1.00      1.00      1.00         9

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36

An infinity of time series models in nnetsauce

2021-03-06T00:00:00+00:00

I was selected and invited to present this family of univariate/multivariate time series models at R/Finance 2020 (in Chicago, IL). However, the COVID-19 pandemics decided differently. It’s still a work in progress; comments, remarks, pull requests are welcome as usual (in nnetsauce). But the general philosophy of model construction in this framework is already set and will be explained in this post. An R example, with less details about the implementation (so, R users read on) can also be found in this post: Bayesian forecasting for uni/multivariate time series.

How does this family of time series models work?

By doing what’s shown on the graph before, I make some relatively strong hypotheses:

I assume that there’s a bidirectional relationship between the series, as in Vector Autoregressive (VAR) models, i.e each series is influencing – and is influenced by – the others.
I assume that all the successive time series sub-models share the same architecture, the same ML model, hence the same hyperparameters, for learning from the lags.

At testing time, for ML prediction, recursive forecasts are implemented so far. One improvement could be to obtain direct forecasts too, in the future.

In the example presented here, the shared ML model is a Bayesian Ridge regression. It could be any other regression model, such as Random Forests, Gradient Boosting Machines, or else, as long as the method’s implementation contains methods fit and predict. Hence, “infinity” in the title. No hyperparameter tuning is used here, so, some better results could certainly be obtained.

Installing packages

pip install git+https://github.com/Techtonique/nnetsauce.git

pip install matplotlib==3.1.3

Obtaining predictions from nnetsauce’s MTS

import nnetsauce as ns
import numpy as np      
import matplotlib.pyplot as plt
from sklearn import datasets, metrics
from sklearn import linear_model


# a simple example with 3 made-up multivariate time series
np.random.seed(123)
M = np.random.rand(10, 3)
M[:,0] = 10*M[:,0]
M[:,2] = 25*M[:,2]
M[:,1] = M[:,2]/23 + 0.5

print("Initial series")
print(M)
print("\n")

# using a Bayesian model along with nnetsauce's MTS 
regr = linear_model.BayesianRidge()
obj_MTS = ns.MTS(regr, lags = 2, n_hidden_features=5)
obj_MTS.fit(M)

print("mean forecast -----")
preds = obj_MTS.predict(h=10)
print(preds)
print("\n")

# confidence level = 80%
# makes the assumption of Gaussian uncertainty and works 
# only if the shared supervised learning model has an argument `return_std`
# in method `predict`
print("predict with credible intervals and confidence level = 80% -----")
preds_80 = obj_MTS.predict(h=10, return_std=True, level=80)
print(preds_80)
print("\n")

# confidence level = 95%
# makes the assumption of Gaussian uncertainty and works 
# only if the shared supervised learning model has an argument `return_std`
# in method `predict`
print("predict with credible intervals and confidence level = 95% -----")
preds_95 = obj_MTS.predict(h=10, return_std=True, level=95)
print(preds_95)
print("\n")

Initial series
[[ 6.96469186  0.74657767  5.67128634]
 [ 5.51314769  0.95989833 10.5776615 ]
 [ 9.80764198  1.02275207 12.02329754]
 [ 3.92117518  1.29244533 18.22624268]
 [ 4.38572245  0.9326568   9.95110638]
 [ 7.37995406  0.69070843  4.3862939 ]
 [ 5.31551374  1.18956626 15.86002396]
 [ 8.49431794  1.16415599 15.27558777]
 [ 7.22443383  0.89324854  9.04471639]
 [ 2.28263231  1.18584361 15.7744031 ]]


mean forecast -----
[[ 6.09958391  1.0445821  12.56586894]
 [ 6.09858969  1.04358789 12.56487472]
 [ 6.10102293  1.04602113 12.56730796]
 [ 6.10101234  1.04601054 12.56729738]
 [ 6.10097759  1.04597578 12.56726262]
 [ 6.10097809  1.04597629 12.56726312]
 [ 6.10097859  1.04597678 12.56726362]
 [ 6.10097857  1.04597677 12.5672636 ]
 [ 6.10097857  1.04597676 12.5672636 ]
 [ 6.10097857  1.04597676 12.5672636 ]]


predict with credible intervals and confidence level = 80% -----
{'mean': array([[ 6.09958391,  1.0445821 , 12.56586894],
       [ 6.09858969,  1.04358789, 12.56487472],
       [ 6.10102293,  1.04602113, 12.56730796],
       [ 6.10101234,  1.04601054, 12.56729738],
       [ 6.10097759,  1.04597578, 12.56726262],
       [ 6.10097809,  1.04597629, 12.56726312],
       [ 6.10097859,  1.04597678, 12.56726362],
       [ 6.10097857,  1.04597677, 12.5672636 ],
       [ 6.10097857,  1.04597676, 12.5672636 ],
       [ 6.10097857,  1.04597676, 12.5672636 ]]), 'std': array([[4.28508739, 4.28508739, 4.28508739],
       [4.28508739, 4.28508739, 4.28508739],
       [4.28508739, 4.28508739, 4.28508739],
       [4.2850874 , 4.2850874 , 4.2850874 ],
       [4.28508736, 4.28508736, 4.28508736],
       [4.2850873 , 4.2850873 , 4.2850873 ],
       [4.28509003, 4.28509003, 4.28509003],
       [4.28509202, 4.28509202, 4.28509202],
       [4.28630194, 4.28630194, 4.28630194],
       [4.28696269, 4.28696269, 4.28696269]]), 'lower': array([[ 0.60802345, -4.44697835,  7.07430848],
       [ 0.60702924, -4.44797257,  7.07331427],
       [ 0.60946248, -4.44553933,  7.07574751],
       [ 0.60945189, -4.44554992,  7.07573692],
       [ 0.60941718, -4.44558463,  7.07570221],
       [ 0.60941775, -4.44558405,  7.07570278],
       [ 0.60941475, -4.44558705,  7.07569978],
       [ 0.60941219, -4.44558962,  7.07569722],
       [ 0.6078616 , -4.4471402 ,  7.07414663],
       [ 0.60701482, -4.44798698,  7.07329985]]), 'upper': array([[11.59114437,  6.53614256, 18.0574294 ],
       [11.59015015,  6.53514835, 18.05643518],
       [11.59258339,  6.53758159, 18.05886842],
       [11.5925728 ,  6.537571  , 18.05885783],
       [11.592538  ,  6.53753619, 18.05882303],
       [11.59253843,  6.53753663, 18.05882347],
       [11.59254242,  6.53754061, 18.05882745],
       [11.59254496,  6.53754315, 18.05882999],
       [11.59409553,  6.53909373, 18.06038056],
       [11.59494231,  6.53994051, 18.06122734]])}


predict with credible intervals and confidence level = 95% -----
{'mean': array([[ 6.09958391,  1.0445821 , 12.56586894],
       [ 6.09858969,  1.04358789, 12.56487472],
       [ 6.10102293,  1.04602113, 12.56730796],
       [ 6.10101234,  1.04601054, 12.56729738],
       [ 6.10097759,  1.04597578, 12.56726262],
       [ 6.10097809,  1.04597629, 12.56726312],
       [ 6.10097859,  1.04597678, 12.56726362],
       [ 6.10097857,  1.04597677, 12.5672636 ],
       [ 6.10097857,  1.04597676, 12.5672636 ],
       [ 6.10097857,  1.04597676, 12.5672636 ]]), 'std': array([[4.28508739, 4.28508739, 4.28508739],
       [4.28508739, 4.28508739, 4.28508739],
       [4.28508739, 4.28508739, 4.28508739],
       [4.2850874 , 4.2850874 , 4.2850874 ],
       [4.28508736, 4.28508736, 4.28508736],
       [4.2850873 , 4.2850873 , 4.2850873 ],
       [4.28509003, 4.28509003, 4.28509003],
       [4.28509202, 4.28509202, 4.28509202],
       [4.28630194, 4.28630194, 4.28630194],
       [4.28696269, 4.28696269, 4.28696269]]), 'lower': array([[-2.29903305, -7.35403486,  4.16725198],
       [-2.30002727, -7.35502907,  4.16625776],
       [-2.29759403, -7.35259583,  4.168691  ],
       [-2.29760462, -7.35260643,  4.16868041],
       [-2.2976393 , -7.35264111,  4.16864573],
       [-2.29763869, -7.3526405 ,  4.16864634],
       [-2.29764354, -7.35264535,  4.16864149],
       [-2.29764745, -7.35264926,  4.16863758],
       [-2.30001887, -7.35502067,  4.16626616],
       [-2.30131391, -7.35631571,  4.16497112]]), 'upper': array([[14.49820087,  9.44319907, 20.9644859 ],
       [14.49720666,  9.44220485, 20.96349169],
       [14.4996399 ,  9.44463809, 20.96592493],
       [14.49962931,  9.4446275 , 20.96591434],
       [14.49959448,  9.44459267, 20.96587951],
       [14.49959488,  9.44459307, 20.96587991],
       [14.49960071,  9.4445989 , 20.96588574],
       [14.4996046 ,  9.44460279, 20.96588963],
       [14.501976  ,  9.44697419, 20.96826103],
       [14.50327104,  9.44826923, 20.96955607]])}

Visualizing credible (because Bayesian, “credible”) intervals

j0 = 1 # choice of time series index

x = np.linspace(0, 9, 10)
y_est = preds_95['mean'][:,j0]
y_est_upper = preds_95['upper'][:,j0]
y_est_lower = preds_95['lower'][:,j0]
y_est_upper2 = preds_80['upper'][:,j0]
y_est_lower2 = preds_80['lower'][:,j0]


fig, ax = plt.subplots()
ax.plot(x, y_est, '-')
ax.fill_between(x, y_est_lower, y_est_upper, alpha=0.4)
ax.fill_between(x, y_est_lower2, y_est_upper2, alpha=0.2)

A deeper learning architecture in nnetsauce

2020-12-18T00:00:00+00:00

As you may know already (or not), nnetsauce contains CustomClassifier and CustomRegressor models, which allow to derive a new statistical/Machine Learning (ML) model from another one.

Creating this new ML model is achieved by doing some feature engineering. That is, by augmenting the original set of explanatory variables, using random or quasirandom numbers and nonlinear activation functions such as hyperbolic tangent or ReLU.

nnetsauce’s Custom*’s can also be chained together, to create a multilayered learning architecture. An example of a 3-layered stack is depicted below, and then demonstrated in an example in Python.

Example in Python

This example is based on a dataset of handwritten digits (0 to 9). The goal is to recognize these digits with nnetsauce, using a multilayered learning architecture. We start by installing nnetsauce and importing useful packages for the demo:

Import packages

!pip install git+https://github.com/Techtonique/nnetsauce.git

import nnetsauce as ns
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits
from time import time
from sklearn.metrics import classification_report

Split data into training and testing sets

digits = load_digits()
X = digits.data
y = digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=123)

Layer 1

Layer 1 is a Random Forest classifier on (X_train, y_train).

layer1_regr = RandomForestClassifier(n_estimators=10, random_state=123)

start = time() 

layer1_regr.fit(X_train, y_train)

# Accuracy in layer 1
print(layer1_regr.score(X_test, y_test))

Layer 2 using layer 1

Layer2 is a Random Forest classifier on ([X_train, h_train_1, clusters_1], y_train), where h_train1 = g(X_train W_1 + b_1), and W_1 is simulated (node_sim) from a uniform distribution. h_train_1 creates 5 new features (n_hidden_features), and 2 indicators of data clusters (n_clusters) are added to the set of explanatory variables (clusters_1).

layer2_regr = ns.CustomClassifier(obj = layer1_regr, n_hidden_features=5, 
                        direct_link=True, bias=True, 
                        nodes_sim='uniform', activation_name='relu', 
                        n_clusters=2, seed=123)
layer2_regr.fit(X_train, y_train)

# Accuracy in layer 2
print(layer2_regr.score(X_test, y_test))

Layer 3 using layer 2

Layer3 is a Random Forest classifier on ([X_train, h_train_1, clusters_1, h_train_2, clusters_2], y_train) where h_train2 = g(X_train W_2 + b_2). As in the previous layer, W_2 is simulated from a uniform distribution h_train2 creates 10 new features and there are 2 additional features (clusters_2), indicators of data clusters. Plus, 70% of nodes in the hidden layer h_train2 are dropped.

layer3_regr = ns.CustomClassifier(obj = layer2_regr, n_hidden_features=10, 
                        direct_link=True, bias=True, dropout=0.7,
                        nodes_sim='uniform', activation_name='relu', 
                        n_clusters=2, seed=123)
layer3_regr.fit(X_train, y_train)

# Accuracy in layer 3
print(layer3_regr.score(X_test, y_test))

The accuracy in layer1, layer2, layer3, are respectively 93.88%, 93.05% and 94.17%. So, in this particular case, if accuracy is the error metric of interest, stacking layers can be useful. Here is a confusion matrix for the whole model, in layer3:

import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
from sklearn.metrics import confusion_matrix

# model predictions on test set 
preds_layer3 = layer3_regr.predict(X_test)

# confusion matrix 
mat = confusion_matrix(y_test, preds_layer3)
print(mat)

Classify penguins with nnetsauce's MultitaskClassifier

2020-12-11T00:00:00+00:00

I’ve recently heard and read about iris dataset’s retirement. iris had been, for years, a go-to dataset for testing classifiers. The new iris is a dataset of palmer penguins, available in R through the package palmerpenguins.

In this blog post, after data preparation, I adjust a classifier – nnetsauce’s MultitaskClassifier – to the palmer penguins dataset.

0 - Import data and packages

Install palmerpenguins R package:

library(palmerpenguins) 

Install nnetsauce’s R package:

library(devtools)
devtools::install_github("Techtonique/nnetsauce/R-package")
library(nnetsauce)

1 - Data preparation

penguins_ below, is a temporary dataset which will contain palmer penguins data after imputation of missing values (NAs).

penguins_ <- as.data.frame(palmerpenguins::penguins)

In numerical variables, NAs are replaced by the median of the column excluding NAs. In categorical variables, NAs are replaced by the most frequent value. These choices have an impact on the result. For example, if NAs are replaced by the mean instead of the median, the results could be quite different.

# replacing NA's by the median

replacement <- median(palmerpenguins::penguins$bill_length_mm, na.rm = TRUE)
penguins_$bill_length_mm[is.na(palmerpenguins::penguins$bill_length_mm)] <- replacement

replacement <- median(palmerpenguins::penguins$bill_depth_mm, na.rm = TRUE)
penguins_$bill_depth_mm[is.na(palmerpenguins::penguins$bill_depth_mm)] <- replacement

replacement <- median(palmerpenguins::penguins$flipper_length_mm, na.rm = TRUE)
penguins_$flipper_length_mm[is.na(palmerpenguins::penguins$flipper_length_mm)] <- replacement

replacement <- median(palmerpenguins::penguins$body_mass_g, na.rm = TRUE)
penguins_$body_mass_g[is.na(palmerpenguins::penguins$body_mass_g)] <- replacement

# replacing NA's by the most frequent occurence
penguins_$sex[is.na(palmerpenguins::penguins$sex)] <- "male" # most frequent

Check: any NA remaining in penguins_?

print(sum(is.na(penguins_)))

The data frame penguins_mat below will contain all the penguins data, with each categorical explanatory variable present in penguins_ transformed into a numerical one (otherwise, no Statistical/Machine learning model can be trained):

# one-hot encoding
penguins_mat <- model.matrix(species ~., data=penguins_)[,-1]
penguins_mat <- cbind(penguins$species, penguins_mat)
penguins_mat <- as.data.frame(penguins_mat)
colnames(penguins_mat)[1] <- "species"

print(head(penguins_mat))
print(tail(penguins_mat))

2 - Model training and testing

The model used here to identify penguins species is nnetsauce’s MultitaskClassifier (the R version here, but there’s a Python version too). Instead of solving the whole problem of classifying these species directly, nnetsauce’s MultitaskClassifier considers three different questions separately: is this an Adelie or not? Is this a Chinstrap or not? Is this a Gentoo or not?

Each one of these binary classification problems is solved by an embedded regression (regression meaning here, a learning model for continuous outputs) model, on augmented data. The relatively strong hypothesis made in this setup is that: each one of these binary classification problems is solved by the same embedded regression model.

2 - 1 First attempt: with feature selection.

At first, only a few features are selected to explain the response: the most positively correlated feature flipper_length_mm

and another an interesting feature: the penguin’s location:

table(palmerpenguins::penguins$species, palmerpenguins::penguins$island)

Splitting the data into a training set and a testing set

y <- as.integer(penguins_mat$species) - 1L
X <- as.matrix(penguins_mat[,2:ncol(penguins_mat)])

n <- nrow(X)
p <- ncol(X)

set.seed(123)
index_train <- sample(1:n, size=floor(0.8*n))

X_train2 <- X[index_train, c("islandDream", "islandTorgersen", "flipper_length_mm")]
y_train2 <- y[index_train]
X_test2 <- X[-index_train, c("islandDream", "islandTorgersen", "flipper_length_mm") ]
y_test2 <- y[-index_train]

obj3 <- nnetsauce::sklearn$linear_model$LinearRegression()
obj4 <- nnetsauce::MultitaskClassifier(obj3)

print(obj4$get_params())

Fit and predict on test set:

obj4$fit(X_train2, y_train2)

# accuracy on test set
print(obj4$score(X_test2, y_test2))

Not bad, an accuracy of 9 penguins out of 10 recognized by the classifier, with manually selected features. Can we do better with the entire dataset (all the features).

2 - 2 Second attempt: the entire dataset.

X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train]

obj <- nnetsauce::sklearn$linear_model$LinearRegression()
obj2 <- nnetsauce::MultitaskClassifier(obj)

obj2$fit(X_train, y_train)

# accuracy on test set
print(obj2$score(X_test, y_test))

By using all the explanatory variables, 100% of the 69 test set penguins are now recognized, thanks to nnetsauce’s MultitaskClassifier.

Bayesian forecasting for uni/multivariate time series

2020-12-04T00:00:00+00:00

This post is about Bayesian forecasting of univariate/multivariate time series in nnetsauce.

For each statistical/machine learning (ML) presented below, its default hyperparameters are used. A further tuning of their respective hyperparameters could, of course, result in a much better performance than what’s showcased here.

1 - univariate time series

The Nile dataset is used as univariate time series. It contains measurements of the annual flow of the river Nile at Aswan (formerly Assuan), 1871–1970, in 10^8 m^3, “with apparent changepoint near 1898” (Cobb(1978), Table 1, p.249).

library(datasets)
plot(Nile)

Split dataset into training/testing sets:

X <- matrix(Nile, ncol=1)
index_train <- 1:floor(nrow(X)*0.8)
X_train <- matrix(X[index_train, ], ncol=1)
X_test <- matrix(X[-index_train, ], ncol=1)

sklearn’s BayesianRidge() is the workhorse here, for nnetsauce’s MTS. It could actually be any Bayesian ML model possessing methods fit and predict (there’s literally an infinity of possibilities here for class MTS).

obj <- nnetsauce::sklearn$linear_model$BayesianRidge()
print(obj$get_params())

Fit and predict using obj:

fit_obj <- nnetsauce::MTS(obj = obj) 
fit_obj$fit(X_train)
preds <- fit_obj$predict(h = nrow(X_test), level=95L,
                          return_std=TRUE)

95% credible intervals:

n_test <- nrow(X_test)
xx <- c(1:n_test, n_test:1)
yy <- c(preds$lower, rev(preds$upper))
plot(1:n_test, drop(X_test), type='l', main="Nile",
     ylim = c(500, 1200))
polygon(xx, yy, col = "gray", border = "gray")
points(1:n_test, drop(X_test), pch=19)
lines(1:n_test, drop(X_test))
lines(1:n_test, drop(preds$mean), col="blue", lwd=2)

2 - multivariate time series

The usconsumption dataset is used as an example of multivariate time series. It contains percentage changes in quarterly personal consumption expenditure and personal disposable income for the US, 1970 to 2010. (Federal Reserve Bank of St Louis. http://data.is/AnVtzB. http://data.is/wQPcjU.)

library(fpp)
plot(fpp::usconsumption)

Split dataset into training/testing sets:

X <- as.matrix(fpp::usconsumption)
index_train <- 1:floor(nrow(X)*0.8)
X_train <- X[index_train, ]
X_test <- X[-index_train, ]

Fit and predict:

obj <- nnetsauce::sklearn$linear_model$BayesianRidge()
fit_obj2 <- nnetsauce::MTS(obj = obj)

fit_obj2$fit(X_train)
preds <- fit_obj2$predict(h = nrow(X_test), level=95L,
                          return_std=TRUE) # standardize output+#plot against X_test

95% credible intervals:

n_test <- nrow(X_test)

xx <- c(1:n_test, n_test:1)
yy <- c(preds$lower[,1], rev(preds$upper[,1]))
yy2 <- c(preds$lower[,2], rev(preds$upper[,2]))

par(mfrow=c(1, 2))
# 95% credible intervals
plot(1:n_test, X_test[,1], type='l', ylim=c(-2.5, 3),
     main="consumption")
polygon(xx, yy, col = "gray", border = "gray")
points(1:n_test, X_test[,1], pch=19)
lines(1:n_test, X_test[,1])
lines(1:n_test, preds$mean[,1], col="blue", lwd=2)

plot(1:n_test, X_test[,2], type='l', ylim=c(-2.5, 3),
     main="income")
polygon(xx, yy2, col = "gray", border = "gray")
points(1:n_test, X_test[,2], pch=19)
lines(1:n_test, X_test[,2])
lines(1:n_test, preds$mean[,2], col="blue", lwd=2)

New nnetsauce

2020-09-25T00:00:00+00:00

A new version of nnetsauce has been released. The best way to use it right now, is to install from GitHub:

pip install git+https://github.com/Techtonique/nnetsauce.git

The main change is the cythonization of RandomBagClassifier. I didn’t see a noticeable change, and I think it’s mainly because in my code, there are a lot of interactions between Python and Cython.

Technical documentation

2020-09-18T00:00:00+00:00

All the Python packages presented in this blog:

nnetsauce Statistical/Machine Learning using Randomized and Quasi-Randomized (neural) networks | feedback form
querier A query language for Python Data Frames | feedback form
mlsauce Miscellaneous Statistical/Machine Learning stuff | feedback form
teller Model-agnostic Statistical/Machine Learning explainability | feedback form

Now have a common home for their documentation, available on Techtonique website (it’s a work in progress).

Figure: New Techtonique Website

For this documentation, I’m using MkDocs in conjunction with keras-autodoc. With MkDocs, I found out that you can create a static website rapidly using Markdown. Regarding package technical documentation in particular, one thing that I find useful and that I’ve been searching for a while, is the ability for a tool to loop and read Python docstrings: that’s what keras-autodoc allowed me to do. I’ve heard of a way of doing such a thing for R documentation recently:

So, cross_val and ESGtoolkit, two R members of Techtonique, are next in line for Techtonique website. I may also give pkgdown a try, depending on how I’m able to tweak the styles, and all.

A new version of nnetsauce, and a new Techtonique website

2020-09-11T00:00:00+00:00

As a reminder nnetsauce, mlsauce, the querier and the teller are now stored under Techtonique – for those who have already starred, you can report your stars there! A new Techtonique website is also out know; it contains documentation + examples for nnetsauce, mlsauce, the querier and the teller, and is a work in progress.

Figure: New Techtonique Website

In addition, a new version of nnetsauce including nonlinear Generalized Linear Models (GLM) has been released, both on Pypi and GitHub.

Installing by using Pypi:

pip install nnetsauce

Installing from Github:

pip install git+https://github.com/Techtonique/nnetsauce.git

I’ve been experiencing some issues when installing from Pypi lately. Please, feel free to report any issue to me, if you’re experiencing some, or use the GitHub version instead. You can still execute the following jupyter notebook – at the bottom for nonlinear GLMs in particular`– to see what’s changed:

https://github.com/Techtonique/nnetsauce/blob/master/nnetsauce/demo/thierrymoudiki_040920_examples.ipynb

How do these nonlinear GLMs work? Let’s say (a highly simplified example) that we want to predict the future, final height of a basketball player, using his current height and his age. We’d use historical information of heights, ages (covariates), and final heights (response) of multiple players:

1 - Input data, the covariates, are transformed into new covariates, as we’ve already seen it before, for nnetsauce. You can visit this page for a refresher: References.

2 - A loss function, a function measuring the quality of adjustment of the model to the observed data, is optimized as represented in the figure below. Currently, two optimization methods are used: Stochastic gradient descent and Stochastic coordinate descent. With, notably and amongst other hyperparameters, early stopping criteria and regularization to prevent overfitting.

Figure: GLM loss function (from notebook)

I’ll write down a more formal description of these algorithms in the future.

Back next week, and a few announcements

2020-09-04T00:00:00+00:00

nnetsauce, mlsauce, the querier and the teller are now stored under Techtonique, and not on my personal GitHub account anymore. I’ll be back next week, with:

1-A new, cython-ready version of nnetsauce including nonlinear Generalized Linear Models (GLM). If you want a sneak peek of this new version, you can execute the following jupyter notebook – at the bottom for nonlinear GLMs in particular, but it contains a general review of the nnetsauce:

https://github.com/Techtonique/nnetsauce/blob/master/nnetsauce/demo/thierrymoudiki_040920_examples.ipynb

Note that AdaBoost and RandomBag classifiers will be faster eventually (ensemble’s loops not cythonized yet, as I’m writing this).

2-A description of the nonlinear Generalized Linear Models.

3-A new Techtonique website containing documentation + examples for nnetsauce, mlsauce, the querier and the teller, all in the same place.

Stay tuned!

Figure: GLM loss function (from notebook)

nnetsauce version 0.5.0, randomized neural networks on GPU

2020-07-17T00:00:00+00:00

Disclaimer: I have no affiliation with Google (cf. JAX).

nnetsauce is a general purpose tool for Statistical/Machine Learning, in which pattern recognition is achieved by using quasi-randomized networks. A new version, 0.5.0, is out on Pypi and for R:

Install by using pip (stable version):

pip install nnetsauce --upgrade

Install from Github (development version):

pip install git+https://github.com/Techtonique/nnetsauce.git --upgrade

Install from Github, in R console:

library(devtools)
devtools::install_github("thierrymoudiki/nnetsauce/R-package")
library(nnetsauce)

This could be the occasion for you to re-read all the previous posts about nnetsauce, or to play with various examples in Python or R. Here are a few other ways to interact with the nnetsauce:

1) Forms

If you’re not comfortable with version control yet: a feedback form.

2) Submit Pull Requests on GitHub

As detailed in this post. Raising issues is another constructive way to interact. You can also contribute examples to this demo repo, using the following naming convention:

yourgithubname_ddmmyy_shortdescriptionofdemo.[ipynb|Rmd]

If it’s a jupyter notebook written in R, then just add _R to the suffix.

3) Reaching out directly via email

Use the address: thierry dot moudiki at pm dot me

To those who are contacting me through LinkedIn: no, I’m not declining, please, add a short message to your request, so that I’d know a bit more about who you are, and/or how we can envisage to work together.

This new version, 0.5.0:

contains a refactorized code for the Base class, and for many other utilities.
makes use of randtoolbox for a faster, more scalable generation of quasi-random numbers.
contains a (work in progress) implementation of most algorithms on GPUs, using JAX. Most of the nnetsauce’s changes related to GPUs are currently made on potentially time consuming operations such as matrices multiplications and matrices inversions. Though, to see a GPU effect, you need to have loads of data at hand, and a relatively high n_hidden_features parameter. How do you try it out? By instantiating a class with the option:

backend = "gpu"

backend = "tpu"

An example can be found in this notebook, on GitHub.

nnetsauce’s future release is planned to be much faster on CPU, due the use of Cython, as with mlsauce. There are indeed a lot of nnetsauce’s parts which can be cythonized. If you’ve ever considered joining the project, now is the right time. For example, among other things, I’m looking for a volunteer to do some testing in R+Python on Microsoft Windows. Envisage a smooth onboarding, even if you don’t have a lot of experience.

R notebooks for nnetsauce

2020-03-06T00:00:00+00:00

English version / Version en français

English version

nnetsauce is a general purpose tool for Statistical/Machine Learning, in which pattern recognition is reliant on Quasi-Randomized networks. Current implementations are Python and R, and a specific RSS feed related to it can be found through this link. This RSS feed will help you to stay up-to-date with everything nnetsauce.

New notebooks for R are now available in the package’s repository, along with multiple other Python notebooks. Notebooks are a great way to create reproducible data analyses, with a mix of code, text and graphs. Popular types of notebooks include jupyter notebooks and R Markdown notebooks.

1) Ridge2Classifier’s R notebook: thierrymoudiki_060320_Ridge2Classifier.Rmd

2) RandomBagClassifier’s R notebook: thierrymoudiki_060320_RandomBagClassifier.Rmd

You can contribute to this repo, by using the following naming convention: yourgithubname_ddmmyy_shortdescriptionofdemo.[ipynb|Rmd].

If it’s a jupyter notebook in R, then just add _R to the suffix.

Note: I am currently looking for a gig. You can hire me on Malt or send me an email: thierry dot moudiki at pm dot me. I can do descriptive statistics, data preparation, feature engineering, model calibration, training and validation, and model outputs’ interpretation. I am fluent in Python, R, SQL, Microsoft Excel, Visual Basic (among others) and French. My résumé? Here!

French version

nnetsauce est un outil d’apprentissage statistique/machine basé sur des réseaux de neurones aléatoires ou quasi-aléatoires. Cet outil est actuellement disponible pour Python et R, et pour suivre les actualités qui y sont liées, vous pouvez souscrire à ce flux RSS spécifique.

De nouveaux notebooks écrits en langage R sont maintenant disponibles dans le répertoire GitHub. Les notebooks constituent un excellent moyen de réaliser des analyses statistiques reproductibles, mêlant du code, du texte et des graphiques. Les choix les plus populaires incluent: les notebooks jupyter et les notebooks R Markdown.

1) Notebook R lié à Ridge2Classifier: thierrymoudiki_060320_Ridge2Classifier.Rmd

2) Notebook R lié à RandomBagClassifier: thierrymoudiki_060320_RandomBagClassifier.Rmd

Pour contribuer à ce répertoire, la règle de nommage est la suivante: yourgithubname_ddmmyy_shortdescriptionofdemo.[ipynb|Rmd].

Si le contenu d’un jupyter notebook est écrit en R, ajoutez un _R au suffixe de la description.

Version 0.4.0 of nnetsauce, with fruits and breast cancer classification

2020-02-28T00:00:00+00:00

English version / Version en français

English version

A new version of nnetsauce, version 0.4.0, is now available on Pypi and for R. As usual, you can install it on Python by using the following commands (command line):

pip install nnetsauce

And if you’re using R, it’s still (R console):

library(devtools)
devtools::install_github("thierrymoudiki/nnetsauce/R-package")
library(nnetsauce)

The R version may be slightly lagging behind on some features; feel free to signal it on GitHub or contact me directly. This new release is accompanied by a few goodies:

1) New features, detailed in the changelog.

2) A refreshed web page containing all the information about package installation, use, interface’s work-in-progress documentation, and contribution to package development.

3) A specific RSS feed related to nnetsauce on this blog (there’s still a general feed containing everything here).

4) A working paper related to Bayesianrvfl2Regressor, Ridge2Regressor, Ridge2Classifier, and Ridge2MultitaskClassifier : Quasi-randomized networks for regression and classification, with two shrinkage parameters. About Ridge2Classifier specifically, you can also consult this other paper.

Among nnetsauce’s new features, there’s a new model class called MultitaskClassifier, briefly described in the first paper from point 4). It’s a multitask classification model based on regression models, with shared covariates. What does that mean? We use the figure below to start the explanation:

Imagine that we have 4 fruits at our disposal, and we would like to classify them as avocados (is an avocado a fruit?), apples or tomatoes, by looking at their color and shapes. What we called covariates before in model description are color and shape, also known as explanatory variables or predictors. The column containing fruit names in the figure – on the left – is a so-called response; a variable that MultitaskClassifier must learn to classify (which is typically much larger). This raw response is transformed into a one-hot encoded one – on the right.

Instead of one response vector, we now have three different responses. And instead of one classification problem on one response, three different two-class classification problems on three responses: is this fruit an apple or not? Is this fruit a tomato or not? Is this fruit an avocado or not? All these three problems share the same covariates: color and shape.

MultitaskClassifier can use any regressor (meaning, a statistical learning model for continuous responses) to solve these three problems; with the same regressor being used for all three of them – which is a priori a relatively strong hypothesis. Regressor’s predictions on each response are interpreted as raw probabilities that the fruit is either one of them or not.

We now use MultitaskClassifier on breast cancer data, as we did in this post for AdaBoostClassifier, to illustrate how it works. The R version for this code would be almost identical, replacing “.”’s by “$”’s.

Import packages:

import nnetsauce as ns
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics
from time import time

Model fitting on training set:

breast_cancer = load_breast_cancer()
Z = breast_cancer.data
t = breast_cancer.target

# Training/testing datasets (using reproducibility seed)
np.random.seed(123)
X_train, X_test, y_train, y_test = train_test_split(Z, t, test_size=0.2)

# Linear Regression is used (can be anything, but must be a regression model)
regr = LinearRegression()
fit_obj = ns.MultitaskClassifier(regr, n_hidden_features=5, 
                             n_clusters=2, type_clust="gmm")

# Model fitting on training set 
start = time()
fit_obj.fit(X_train, y_train)
print(time() - start)

# Model accuracy on test set
print(fit_obj.score(X_test, y_test))

# Area under the curve
print(fit_obj.score(X_test, y_test, scoring="roc_auc"))

These results can be found in nnetsauce/demo/. MultitaskClassifier’s accuracy on this dataset is 99.1%, and other indicators such as precision are equal to 99% on this dataset too. Let’s visualize the missclassification results, as we did in a previous post, on the same dataset.

In this case with MultitaskClassifier, and no advanced hyperparameter tweaking, there is one patient out of 114 who is missclassified. A robust way to understand MultitaskClassifier’s accuracy on this dataset using these same parameters, could be to repeat the same procedure for multiple random reproducibility seeds (see code, the training and testing sets randomly change when we change the seed, and we change MultitaskClassifier’s seed too).

We obtain the results below for 100 reproducibility seeds. The accuracy is always at least 90%, mostly 95% and quite sometimes, higher than 98% (with no advanced hyperparameter tweaking).

French version

Une nouvelle version de nnetsauce, la version 0.4.0, est maintenant disponible sur Pypi et pour R. Comme d’habitude, vous pouvez l’installer sur Python en utilisant les commandes suivantes (ligne de commande):

pip install nnetsauce

Et si vous utilisez R, c’est toujours (console R):

library(devtools)
devtools::install_github("thierrymoudiki/nnetsauce/R-package")
library(nnetsauce)

La version R peut être légèrement en retard sur certaines fonctionnalités; n’hésitez pas à le signaler sur GitHub ou à me contacter directement en cas de problème. Cette nouvelle version s’accompagne de quelques goodies:

1) Nouvelles fonctionnalités, détaillées dans le fichier changelog.

2) Une page Web actualisée contenant toutes les informations sur l’installation, l’utilisation, la documentation de travail en cours, et la contribution au développement de l’outil.

3) Un flux RSS spécifique lié à nnetsauce sur ce blog (il existe toujours un flux général contenant tous les autres sujets, ici).

4) Un document de travail relatif à Bayesianrvfl2Regressor, Ridge2Regressor, Ridge2Classifier et Ridge2MultitaskClassifier: Quasi-randomized networks for regression and classification, with two shrinkage parameters. À propos de Ridge2Classifier en particulier, vous pouvez également consulter cet autre article.

Parmi les nouvelles fonctionnalités, il existe une nouvelle classe de modèle appelée MultitaskClassifier, brièvement décrite dans le premier article du point 4). Il s’agit d’un modèle de classification multitâche basé sur des modèles de régression, avec des covariables partagées. Qu’est-ce que cela signifie? Nous utilisons la figure ci-dessous pour commencer l’explication:

Imaginez que nous ayons 4 fruits à notre disposition, et nous aimerions les classer comme avocats (un avocat est-il un fruit?), pommes ou tomates, en regardant leur couleur et leur forme. Ce que nous appelions auparavant les covariables dans la description du modèle sont la couleur et la forme, également appelées variables explicatives ou prédicteurs. La colonne contenant les noms des fruits sur la figure – à gauche – est la réponse; la variable que MultitaskClassifier doit apprendre à classer (qui contient généralement beaucoup plus d’observations). Cette réponse brute est transformée en une réponse codée – à droite.

Au lieu d’une réponse, nous avons maintenant trois réponses différentes. Et au lieu d’un problème de classification sur une réponse, trois problèmes de classification à deux classes différents, sur trois réponses: ce fruit est-il ou non une pomme? Ce fruit est-il ou non une tomate? Ce fruit est-il ou non un avocat? Ces trois problèmes partagent les mêmes covariables: la couleur et la forme.

MultitaskClassifier peut utiliser n’importe quel modèle de régression (c’est-à-dire un modèle d’apprentissage statistique pour des réponses continues) pour résoudre ces trois problèmes simultanément; avec le même modèle de régression utilisé pour les trois - ce qui est a priori une hypothèse relativement forte. Les prédictions du modèle de régression sur chaque réponse sont alors interprétées comme des probabilités brutes que le fruit soit ou non, dans l’une ou l’autre des classes.

Nous utilisons maintenant MultitaskClassifier sur des données de cancer du sein, comme nous l’avions fait dans cet article pour AdaBoostClassifier. La version R de ce code serait presque identique, il s’agirait essentiellement de remplacer les «.» ’S par des « $ »’ s.

Import des packages:

import nnetsauce as ns
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics
from time import time

Ajustement du modèle sur l’ensemble d’entraînement:

breast_cancer = load_breast_cancer()
Z = breast_cancer.data
t = breast_cancer.target

# Training/testing datasets (using reproducibility seed)
np.random.seed(123)
X_train, X_test, y_train, y_test = train_test_split(Z, t, test_size=0.2)

# Linear Regression is used (can be anything, but must be a regression model)
regr = LinearRegression()
fit_obj = ns.MultitaskClassifier(regr, n_hidden_features=5, 
                             n_clusters=2, type_clust="gmm")

# Model fitting on training set 
start = time()
fit_obj.fit(X_train, y_train)
print(time() - start)

# Model accuracy on test set
print(fit_obj.score(X_test, y_test))

# Area under the curve
print(fit_obj.score(X_test, y_test, scoring="roc_auc"))

Les résultats de ce traitement se trouvent dans ce notebook. La précision du MultitaskClassifier sur cet ensemble de données est de 99,1%, et d’autres indicateurs sont également de l’ordre de 99% sur ces données. Visualisons maintenant comment les observations sont bien ou mal classées en fonction de leur classe réelle, comme nous l’avions fait dans un post précédent, sur le même ensemble de données.

Dans ce cas, avec MultitaskClassifier et aucun ajustement avancé des hyperparamètres, il n’y a qu’un patient sur 114 qui est mal classé. Un moyen robuste de comprendre la précision de MultitaskClassifier sur cet ensemble de données en utilisant ces mêmes paramètres, pourrait être de répéter la même procédure pour plusieurs graines de reproductibilité aléatoire (voir le code, les ensembles d’apprentissage et de test changent de manière aléatoire lorsque nous changeons la graine seed, et nous changeons aussi la graine de MultitaskClassifier ).

Nous obtenons les résultats ci-dessous pour 100 graines de reproductibilité. La précision est toujours d’au moins 90%, la plupart du temps égale à 95% et assez souvent, supérieure à 98% (sans ajustement avancé des hyperparamètres).

Feedback forms for contributing

2020-02-07T00:00:00+00:00

To those who are not comfortable with using Git/Version Control yet: I’ve created detailed feedback forms for the tools that I’m actively maintaining.

the nnetsauce, a general-purpose tool for Statistical/Machine Learning: feedback form.
the querier, an SQL-like query language: feedback form.
the teller, a model-agnostic tool for Machine Learning explainability: feedback form.
ESGtoolkit , a tool for Monte Carlo simulation: feedback form.
crossval, a tool offering generic validation functions for predictive models: feedback form.

A link to these forms can also be found in each package’s README, in section Contributing. Next week, I’ll publish a post about Git/Github and submitting Pull/Requests.

nnetsauce for R

2020-01-31T00:00:00+00:00

nnetsauce is now available to R users (currently, a development version). As a reminder, for those who are interested, the following page illustrates different use-cases for the nnetsauce, including deep learning application examples. This post from September 18 is about an Adaptive Boosting (boosting) algorithm variant available in the nnetsauce. This other post from September 25 presents a Bootstrap aggregating (bagging) algorithm variant also available in the nnetsauce, and is about recognizing tomatoes and apples.

Not all Python functions are available in R so far, but the majority of them are. R implementation is catching up fast though. The general rule for invoking methods on objects in R as we’ll see in the example below, is to mirror the Python way, but replacing .’s by $‘s. Contributions/remarks are welcome as usual, and you can submit a pull request on Github.

Installation

Here is how to install nnetsauce from Github using R console:

# use library devtools for installing packages from Github 
library(devtools)

# install nnetsauce from Github 
devtools::install_github("thierrymoudiki/nnetsauce/R-package")

# load nnetsauce
library(nnetsauce)

Having installed and loaded nnetsauce, we can now showcase a simple classification example based on iris dataset and Ridge2Classifier model (cf. this paper and this notebook). With the iris dataset, we’d like to classify flowers as setosa, versicolor or virginica species, based on the following characteristics: sepal lengths, sepal widths, petal lengths and petal widths.

Example

Data preprocessing:

library(datasets)

# Explanatory variables:  Sepal.Length, Sepal.Width, Petal.Length, Petal.Width
X <- as.matrix(iris[, 1:4])

# Classes of flowers: setosa, versicolor, virginica
y <- as.integer(iris[, 5]) - 1L

# Number of examples
n <- dim(X)[1]

# Split data into training/testing sets
set.seed(123)
index_train <- sample.int(n, size = floor(0.8*n))
X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train] 

Fit Ridge2Classifier model to iris dataset with default parameters:

# Create Ridge2Classifier model with 5 nodes in 
# the hidden layer, and clustering of inputs
# all R-nnetsauce models are created this way
obj <- Ridge2Classifier(activation_name='relu', 
n_hidden_features=5L, n_clusters=2L, type_clust='gmm')

# Model fitting on training set 
# Notice Python `.`'s replaced by `$`'s
print(obj$fit(X_train, y_train))

Obtain model accuracy on test set:

print(obj$score(X_test, y_test))

Obtain model probabilities on test set, for each class:

probs <- data.frame(obj$predict_proba(X_test))
colnames(probs) <- dimnames(iris)[[2]][1:3]
print(head(probs))

As a reminder, your examples of use of nnetsauce do have a home in this repository.

A new version of nnetsauce (v0.3.1)

2020-01-24T00:00:00+00:00

A new version (v0.3.1) of nnetsauceis now available. The stable version on PyPi, and a development version on Github. Notable changes for this new version are:

The inclusion of an upper bound on the error rate of Adaboost: crucial, because the error rate at each iteration has to be at least as good as random guess’.
New quasi-randomized networks models for regression and classification, with two shrinkage parameters (for model regularization).

The full list of changes can always be found here on Github and a notebook describing some of the new models (for classification) here for 4 datasets (with a snippet below on a wine classification dataset).

Contributions/remarks are welcome as usual, you can submit a pull request on Github.

2019 Recap, the nnetsauce, the teller and the querier

2019-12-20T00:00:00+00:00

This post is a recap of 2019. It’s especially about the querier, the nnetsauce and the teller. If you want a summary (or a reminder!) of how these tools can be beneficial to you, this is the perfect place.

The querier

The querier is a query language which helps you to retrieve data from Python pandas Data Frames. This language is inspired from Structured Query Language (SQL)’s logic for relational databases. There are currently 9 types of operations available in the querier – with no plan to expand the list much further, to maintain a relatively simple mental model.

In this post from October 25 we present the querier and different verbs constituting its grammar for wrangling data: concat, delete, drop, filtr, join, select, summarize, update, request.

In this post from November 22, we show how our querier verbs can be composed to form efficient data wrangling pipelines. Pipelines? For example: select columns, then filter on rows based on given criteria, and finally, obtain column averages.

In this post from November 29, we examine the querier’s performance (speed) on datasets with increasing sizes (up to 1 million rows and 100 columns). The post gives you an idea of what you can expect from the querier when using it on your data.

The nnetsauce

The nnetsauce is a Statistical/Machine learning tool, in which pattern recognition is achieved by combining layers of (randomized and) quasi-randomized networks. These building blocks – layers – constitute the basis of many custom models, including models with deeper learning architectures for regression, classification, and multivariate time series forecasting. The following page illustrates different use-cases for the nnetsauce, including deep learning application examples.

This post from September 18 is about an Adaptive Boosting (boosting) algorithm variant available in the nnetsauce. This other post from September 25 presents a Bootstrap aggregating (bagging) algorithm variant also available in the nnetsauce, and talks about recognizing tomatoes and apples. The main strength of the nnetsauce here on bagging and boosting variants, is to heavily take advantage of randomization to increase ensembles’ diversity and accuracy.

On October 18, we presented 3 ways for measuring the uncertainty around nnetsauce model predictions: using a bayesian linear model , using a uniform distribution for the network’s hidden layers, and using dropout regularization technique.

The teller

The teller is a model-agnostic tool for Statistical Machine Learning (ML) explainability. It uses numerical derivatives to obtain insights on the influence of explanatory variables on a response variable (variable to be explained).

This post from November 1 introduces the teller’s philosophy: a little increase in model’s explanatory variables + a little decrease, and we can obtain approximate sensitivities of model predictions to changes in the explanatory variables. Some ML models are accurate, but are considered to be hard to explain (black boxes, relatively to the intuitive linear models). We do not want to sacrifice this high model accuracy to explainability, hence: the teller.

In this post from November 8, we use Jackknife resampling to obtain confidence intervals around explanatory variables’ marginal effects. This resampling procedure allows us to derive confidence intervals and hypotheses tests for the significance of marginal effects (yes, I know that some of you do not like p-values). With these tests, we can identify important explanatory variables (not in the sense of causation, though).

This post from November 15 uses the teller to compare two ML models on the Boston Housing dataset; Extremely Randomized Trees and Random Forest Regressions. By using the teller, we can compare model residuals (model predicted values minus observed real values), their Multiple R-Squared and their respective marginal effects side and side.

This post from December 6 examines model interactions. By interactions, we mean: how does the response variable (variable to be explained) changes when both explanatory variable 1 increases of 1, and explanatory variable 2 increases of 1. On the Boston Housing dataset used in this post, it means for example: understanding how median value of owner-occupied homes (in $1000’s) changes when the index of accessibility to radial highways and the number of rooms per dwelling increase of 1 simultaneously.

Conclusion

There’s certainly at least one of these tools that excites your curiosity: comments/remarks/contributions are welcome as usual, either on Github or via email.

This is my last post of 2019, happy holidays!

Prediction intervals for nnetsauce models

2019-10-18T00:00:00+00:00

There is always some uncertainty attached to Statistical/Machine Learning (ML) model’s predictions. In this post, we examine a few ways to obtain prediction intervals for nnetsauce’s ML models. As a reminder, every model in the nnetsauce is based on a component g(XW+b) where:

X is a matrix containing some explanatory variables and optional clustering information (taking into account input data’s heterogeneity).
W creates additional explanatory variables from X and is drawn from various random and quasirandom sequences.
b is an optional bias parameter.
g is an activation function such as the hyperbolic tangent or the sigmoid function.

If we consider again our example with tomatoes and apples, the y below (figure) contains our model’s decisions: either “observation is a tomato” or “observation is an apple”). The X’s are each fruit’s characteristics (color, shape, etc.).

This figure presents the structure of a linear nnetsauce model but it could be anything including nonlinear models, as long as the model has methods fit and predict. There are many ways to obtain prediction intervals for nnetsauce models. Here, we present 3 ways for doing that:

Using a bayesian linear model as a basis for our nnetsauce model
Using a uniform distribution for W
Using the randomness of dropout regularization technique on g(XW+b)

Here are the Python packages needed for our demo:

import nnetsauce as ns
import numpy as np      
from sklearn import datasets
from sklearn import linear_model
from sklearn.model_selection import train_test_split

Loading dataset for model training:

# Boston Housing data
boston = datasets.load_boston()
X = boston.data 
y = np.log(boston.target)

# split data in training/test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,  
                                                    random_state=123)

Using a bayesian model as a basis for our nnetsauce model

Fitting a Bayesian Ridge regression model:

# base model for nnetsauce
regr = linear_model.BayesianRidge()

# create nnetsauce model based on the bayesian linear model
fit_obj = ns.CustomRegressor(obj=regr, n_hidden_features=10, 
                             activation_name='relu', n_clusters=2)

# model fitting
fit_obj.fit(X_train, y_train)

# mean prediction and
# standard deviation of the predictions
mean_preds1, std_preds1 = fit_obj.predict(X_test, return_std=True)

The model’s prediction interval (using the mean prediction and the standard deviation of our predictions) is depicted below:

Red line represents values of y_test; actual observed houses prices on unseen data (data not used for training the model). The black line represents model predictions on unseen data, and the shaded region is our prediction interval.

Using a uniform distribution for W

For doing that, we simulate the W in a nnetsauce CustomRegressor 100 times. With nodes_sim="uniform".

# number of model simulations
n = 100 

# create nnetsauce models
fit_obj2 = [ns.CustomRegressor(obj=regr, n_hidden_features=10, 
                               activation_name='relu', n_clusters=2,
                               nodes_sim="uniform", 
                               seed=(i*1000 + 2)) for i in range(n)]

# model fitting
fit_obj2_fits = [fit_obj2[i].fit(X_train, y_train) for i in range(n)]

# models predictions 
preds2 = np.asarray([fit_obj2_fits[i].predict(X_test) for i in range(n)]).T

# mean prediction 
mean_preds2 = preds2.mean(axis=1)

# standard deviation of the predictions
std_preds2 = preds2.std(axis=1)

We obtain the following prediction interval:

Using the dropout regularization technique on g(XW+b)

The dropout technique randomly drops some elements in the matrix g(XW+b). We can use its randomness to incorporate some uncertainty into our model:

# create nnetsauce models
fit_obj3 = [ns.CustomRegressor(obj=regr, n_hidden_features=10, 
                               activation_name='relu', n_clusters=2,
                               nodes_sim="sobol", dropout=0.1, 
                               seed=(i*1000 + 2)) for i in range(n)]

# models fitting
fit_obj3_fits = [fit_obj3[i].fit(X_train, y_train) for i in range(n)]

# model predictions 
preds3 = np.asarray([fit_obj3_fits[i].predict(X_test) for i in range(n)]).T

# mean prediction 
mean_preds3 = preds3.mean(axis=1)

# standard deviation of the predictions
std_preds3 = preds3.std(axis=1)

Using the dropout technique, we obtain:

Using a uniform distribution for W seems to be the best way to obtain prediction intervals on this dataset and for this model. However, it’s worth mentioning that the base model (a bayesian linear model) is not calibrated. Calibrating the base model could lead to different results than those displayed here.

Bagging in the nnetsauce

2019-09-25T00:00:00+00:00

In this post, I will show you how to use a bootstrap aggregating classification algorithm (do not leave yet, I will explain it with apples and tomatoes!). This algorithm is implemented in the new version of nnetsauce (v0.2.0) and is called randomBag. The complete list of changes included in this new version can be found here.

nnetsauce (v0.2.0) can be installed (using the command line) from Pypi as:

pip install nnetsauce

The development/cutting-edge version can still be installed from Github, by using:

pip install git+https://github.com/Techtonique/nnetsauce.git

Let’s start with an introduction on how randomBag works, with apple and tomatoes. Like we’ve seen for Adaboost, randomBag is an ensemble learning method. It combines multiple individual statistical/machine learning (ML) models into one, with the hope of achieving a greater performance. We consider the following dataset, containing 2 apples, 3 tomatoes, and a few characteristics describing them: shape, color, size of the leaves.

3 individual ML models are tasked to classify these fruits (is tomato… a fruit?). That is, to say: “given that the shape is x and the size of leaves is y, observation 1 is an apple”. For all the 5 fruits. In the randomBag, these individual ML models are quasi-randomized networks, and they typically classify fruits by choosing a part of each fruit’s characteristics, and a part of these 5 fruits with repetition allowed. For example:

-ML model #1 uses shape and size of the leaves, and fruits 1, 2, 3, 4

-ML model #2 uses shape and color (yes, here apples are always green!), and fruits 1, 3, 3, 4

-ML model #3 uses color and size of the leaves, and fruits 1, 2, 3, 5

Then, each individual trained model provides probabilities to say, for each fruit, “it’s an apple” (or not):

Fruit#	ML model #1	ML model #2	ML model #3	is an apple?
1	79.9%	48.5%	35.0%	yes
2	51.5%	11.1%	20.0%	no
3	26.1%	5.8%	90.0%	yes
4	85.5%	70.5%	51.0%	no
5	22.5%	61.3%	55.0%	no

How to read this table? Model #1 estimates that fruit #2 has 51% of chances of being an apple (thus, 49% of being a tomato). Similarly, model #3 estimates that, given its characteristics, fruit #5 has 55% of chances of being an apple. When a probability in the previous table is > 50%, the model decides “it’s an apple”. Therefore, here are ML models’ classification accuracies:

Accuracy	ML model #1	ML model #2	ML model #3
	40.0%	20.0%	40.0%

If we calculate a standard deviation of decision probabilities per model, we can obtain an ad hoc - reaaally ad hoc - measure of their uncertainty around their decisions. These uncertainties are respectively 29.3%, 29.4%, 26.2% for the 3 ML models we estimated. randomBag will now take each fruit, and calculate an average probability over the 3 ML models that, “it’s an apple”. The ensemble’s decision probabilities are:

Observation#	`randomBag` ensemble	is an apple?	ensemble decision
1	54.4%	yes	yes
2	27.5%	no	no
3	40.6%	yes	no
4	69.0%	no	yes
5	46.3%	no	no

Doing this, the accuracy of the ensemble increases to 60.0% (compared to 40.0%, 20.0%, 40.0% for individual ML models), and the ensemble’s ad hoc uncertainty about its decisions is now 15.5% (compared to 29.3%, 29.4%, 26.2% for individual ML models).

How does it work in the nnetsauce? As mentioned before, the individual models are quasi-randomized networks (deterministic, cf. here for details). At each bootstrapping repeat, a fraction of dataset’s observations and columns are randomly chosen (with replacement for observations), in order to increase diversity within the ensemble and reduce its variance.

We use the wine dataset from UCI repository for training nnetsauce’s randomBag. This dataset contains information about wines’ quality, depending on their characteristics. With ML applied to this dataset, we can deduce the quality of a wine - previously unseen - by using its characteristics.

# Load dataset

wine = load_wine()
Z = wine.data
t = wine.target
np.random.seed(123)
Z_train, Z_test, y_train, y_test = train_test_split(Z, t, test_size=0.2)


# Define the ensemble model, with 100 individual models (n_estimators)
# that are decision tree classifiers with depth=2

# One half of the observations and one half of the columns are considered 
# at each repeat (`col_sample`, `row_sample`)

clf = DecisionTreeClassifier(max_depth=2, random_state=123)
fit_obj = ns.RandomBagClassifier(clf, n_hidden_features=5,
                                direct_link=True,
                                n_estimators=100, 
                                col_sample=0.5, row_sample=0.5,
                                dropout=0.1, n_clusters=3, 
                                type_clust="gmm", verbose=1)


# Fitting the model to the training set

fit_obj.fit(Z_train, y_train)
print(fit_obj.score(Z_test, y_test))


# Obtain model predictions on test set's unseen data 

preds = fit_obj.predict(Z_test)


# Obtain classifiction report on test set

print(metrics.classification_report(preds, y_test))

The randomBag is ridiculously accurate on this dataset. So, you might have some fun trying these other examples, or any other real world example of your choice! If you happen to create a notebook, it will find its home here (naming convention: yourgithubname_ddmmyy_shortdescriptionofdemo.ipynb).

Note: I am currently looking for a side hustle. You can hire me on Malt or send me an email: thierry dot moudiki at pm dot me. I can do descriptive statistics, data preparation, feature engineering, model calibration, training and validation, and model outputs’ interpretation. I am fluent in Python, R, SQL, Microsoft Excel, Visual Basic (among others) and French. My résumé? Here!

Adaboost learning with nnetsauce

2019-09-18T00:00:00+00:00

Two variants of Adaboost (Adaptive boosting) algorithms are now included in the development version of nnetsauce, available on Github. My nnetsauce implementation of Adaboost has some specificities, as it will be shown in the sequel of this post. It is also worth noting that the current implementation is 100% Python (neither underlying C, nor C++).

The package can be imported from Github, by doing:

pip install git+https://github.com/Techtonique/nnetsauce.git

I’ll show you how to use these Adaboost classifiers on two popular datasets.

First, a few words about statistical/machine learning (ML hereafter). ML is about pattern recognition. A phenomenon that has a trend or a seasonality, such as the evolution of the weather, can be studied by ML. Other use cases include identifying fraudulent transactions (unless, of course, the smarts increase at a dramatically fast pace), determining if a tumor is benign or malignant, natural language processing, etc. On the other hand ML cannot say which of heads or tail will appear next when you flip a fair coin. By using statistical inference, you can derive quantities such as the probability of the number of trials until head or tails appear, but that’s it.

Another illustration is presented below. All that I can say about my simulated stock returns (on the left), is that their average is 0, and their standard deviation is 1. Trying to predict the next return will (extremely) likely give me: 0. On the right, I can see a trend in my simulated rents. So, I can predict more or less accurately the rent of an appartment; assuming that an increase of 1 squared meter in and appartment’s surface produces an increase of 3€ in rents.

Adaboost is an ML algorithm, i.e it achieves pattern recognition. More specifically, it’s an ensemble learning algorithm called boosting. For more details about boosting in general, the interested reader can consult this paper. And for Adaboost in particular, that one. The aim of ensemble learning is to combine multiple individual ML models into one. Ensembling thus aims at obtaining a model, that has an improved recognition error over the individual models’ recognition error. And most of the times, it works.

We start by importing the packages necessary for the job, along with nnetsauce (namely numpy and sklearn, nothing weird!):

import nnetsauce as ns
import numpy as np
from sklearn.datasets import load_breast_cancer, load_wine, load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics

Our first example is based on wisconsin breast cancer dataset from UCI (University of California at Irvine) repository, and available in sklearn. More details about the content of these datasets can be found here and here. wisconsin breast cancer dataset is splitted into a training set (for training the model to pattern recognition) and test set (for model validation):

# Import dataset from sklearn
breast_cancer = load_breast_cancer()
X = breast_cancer.data
y = breast_cancer.target

# training test and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=123)

The first version of Adaboost that we apply is SAMME.R, also known as Real Adaboost. The acronym SAMME stands for Stagewise Additive Modeling using a Multi-class Exponential loss function, and nnetsauce’s implementation of SAMME has some specificities:

-The base learners (individual models in the ensemble) are quasi-randomized (deterministic) networks.

-At each boosting iteration, a fraction of dataset’s observations can be randomly chosen, in order to increase diversity within the ensemble.

-For SAMME (not for SAMME.R, yet), an experimental feature allows to apply an elastic net-like constraint to individual observations weights. That is: the norm of these individual weights can be bounded during the learning procedure. I am curious to hear how well (or not) it works for you.

# SAMME.R

# base learner
clf = LogisticRegression(solver='liblinear', multi_class = 'ovr', 
                         random_state=123)

# nnetsauce's Adaboost
fit_obj = ns.AdaBoostClassifier(clf, 
                                n_hidden_features=11, 
                                direct_link=True,
                                n_estimators=250, learning_rate=0.01126343,
                                col_sample=0.72684326, row_sample=0.86429443,
                                dropout=0.63078613, n_clusters=2,
                                type_clust="gmm",
                                verbose=1, seed = 123, 
                                method="SAMME.R")  

The base learner, clf, is a logistic regression model but it could be anything including decision trees. fit_obj is a nnetsauce object that augments clf with a hidden layer of transformed predictors, and typically makes clf’s predictions nonlinear. n_hidden_features is the number of nodes in the hidden layer, and dropout randomly drops some of these nodes at each boosting iteration (which reduces overtraining). col_sample and row_sample specify the fraction of columns and rows chosen for fitting the base learner at each iteration. With n_clusters, the data can be clustered into homogeneous groups before model training.

nnetsauce’s Adaboost can now be fitted; 250 iterations are used:

# Fitting the model to training set 
fit_obj.fit(X_train, y_train)  

# Obtain model's accuracy on test set
print(fit_obj.score(X_test, y_test))

With the following graph, we can visualize how well our data have been classified by nnetsauce’s Adaboost.

import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
from sklearn.metrics import confusion_matrix
preds = fit_obj.predict(X_test)
mat = confusion_matrix(y_test, preds)
sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False)
plt.xlabel('true label')
plt.ylabel('predicted label');

1 denotes a malignant tumor, and 0, its absence. For the 3 (out of 114) patients remaining missclassified, it could be interesting to change the model sample_weights, and give them more weight in the learning procedure. Then, we could see how well the result evolves with this change; depending on which classifier’s decision we consider being the worst (or best). But note that:

1.The model will never be perfect (plus, the labels are based on human-eyed labelling ;) ). Still: though he said “all models are wrong”, he didn’t mean “are false”. He meant wrong in the sense that these are simply (even sometimes, great) representations of a reality. “False” would be: wrong to an extent that can’t be tolerated. And indeed in that regard, some models are false, for certain purposes. If I fit a model to this dataset and get an accuracy of 30%, no matter how sophisticated or expensive it is, the model is just plainly unacceptable - for that purpose.

2.Patients are not labelled. Label is just a generic term in classification, for all types of classification models and data. Here, those are 0 and 1.

Our second example is based on wine dataset from UCI repository. This dataset contains information about wines’ quality, depending on their characteristics. With ML applied to this dataset, we can deduce the quality of a wine, previously unseen, by using its characteristics. SAMME is now used instead of SAMME.R. This second algorithm seems to require more iterations to converge than SAMME.R (but you, tell me from your experience!):

# load dataset
wine = load_wine()
Z = wine.data
t = wine.target
np.random.seed(123)
Z_train, Z_test, y_train, y_test = train_test_split(Z, t, test_size=0.2)


# SAMME
clf = LogisticRegression(solver='liblinear', multi_class = 'ovr', 
                         random_state=123)
fit_obj = ns.AdaBoostClassifier(clf, 
                                n_hidden_features=np.int(8.21154785e+01), 
                                direct_link=True,
                                n_estimators=1000, learning_rate=2.96252441e-02,
                                col_sample=4.22766113e-01, row_sample=7.87268066e-01,
                                dropout=1.56909180e-01, n_clusters=3,
                                type_clust="gmm",
                                verbose=1, seed = 123, 
                                method="SAMME") 
 
 # Fitting the model to training set
fit_obj.fit(Z_train, y_train)  

After fitting the model, we can obtain some statistics about its quality (accuracy, precision, recall, f1-score; every nnetsauce model is 100% sklearn-compatible) in classifying unseen wines:

# model predictions on unseen wines 
preds = fit_obj.predict(Z_test)     

# descriptive statistics of model performance
print(metrics.classification_report(preds, y_test))    

A Jupyter notebook for this post can be found here. More examples of use of nnetsauce’s Adaboost here.

nnetsauce on Pypi

2019-06-05T00:00:00+00:00

Machine Learning and Deep Learning package nnetsauce (introduced here ) is now available on Pypi – and I’m so proud of it ;). Which means, you can install it by using the command line:

pip install nnetsauce

For more examples of use of this package, you can consult this post, or the package’s Github repo README.

More nnetsauce (examples of use)

2019-05-09T00:00:00+00:00

As mentioned in a previous post, nnetsauce is a Python package for Statistical/Machine learning and deep learning, based on combinations of neural networks layers. It could be used for solving regression, classification and multivariate time series forecasting problems. This post makes a more detailed introduction of nnetsauce, with a few examples based on classification and deep learning.

Installing the package

Currently, nnetsauce can be installed through Github (but it will be available on PyPi in a few weeks).

Here is how:

git clone https://github.com/Techtonique/nnetsauce.git
cd nnetsauce
python setup.py install

Examples of use of `nnetsauce`

Below, are two examples of use of nnetsauce. A classification example based on breast cancer data, and an illustrative deep learning example. In the classification example, we show how a logistic regression model can be enhanced, for a higher accuracy (accuracy is used here for simplicity), by using nnetsauce. The deep learning example shows how custom building blocks of nnetsauce objects can be combined together, to form a - perfectible - deeper learning architecture.

scikit-learn models are heavily used in these examples, but nnetsauce will work with any learning model possessing methods fit() and predict() (plus, predict_proba() for a classifier). That is, it could be used in conjunction with xgboost, LightGBM, or CatBoost for example. For the purpose of model validation, sklearn’s cross-validation functions such as GridSearchCV and cross_val_score can be employed (on nnetsauce models), as it will be shown in the classification example.

Classification example

For this first example, we start by fitting a logistic regression model to breast cancer data on a training set, and measure its accuracy on a validation set:

    # 0 - Packages ----- 

    # Importing the packages that will be used in the demo
    import nnetsauce as ns
    from sklearn import datasets, linear_model
    from sklearn.model_selection import train_test_split
    

    # 1 - Datasets -----

    # Loading breast cancer data
    breast_cancer = datasets.load_breast_cancer()
    Z = breast_cancer.data
    t = breast_cancer.target

    
    # 2 - Data splitting -----            

    # Separating the data into training/testing set, and 
    # a validation set
    Z_train, Z_test, t_train, t_test = train_test_split(
        Z, t, test_size=0.2, random_state=42)


    # 3 - Logistic regression -----

    # Fitting the Logistic regression model on 
    # training set
    regr = linear_model.LogisticRegression()                        
    regr.fit(Z_train, t_train)

    # predictive accuracy of the model on test set
    regr.score(Z_test, t_test)  

The accuracy of this model is equal to 0.9561. The logistic regression is now augmented of n_hidden_features additional features with nnetsauce. We use GridSearchCV to find a better combination of hyperparameters; additional hyperparameters such as row subsampling (row_sample) and dropout are included and reseached:

    # Defining nnetsauce model
    # based on the logistic regression model
    # defined previously
    fit_obj = ns.CustomClassifier(
    obj=regr,
    n_hidden_features=10,
    direct_link=True,
    bias=True,
    nodes_sim="sobol",
    activation_name="relu", 
    seed = 123)
    
    # Grid search ---
    from sklearn.model_selection import GridSearchCV
    # grid search for finding better hyperparameters
    np.random.seed(123)
    clf = GridSearchCV(cv = 3, estimator = fit_obj,
                       param_grid={'n_hidden_features': range(5, 25), 
                                   'row_sample': [0.7,0.8, 0.9], 
                                   'dropout': [0.7, 0.8, 0.9], 
                                   'n_clusters': [0, 2, 3, 4]}, 
                                   verbose=2)
    
    # fitting the model
    clf.fit(Z_train, t_train)

    # 'best' hyperparameters found 
    print(clf.best_params_)
    print(clf.best_score_)

    # predictive accuracy on test set
    clf.best_estimator_.score(Z_test, t_test)

After using nnetsauce, the accuracy is now equal to 0.9692.

Deep learning example

This second example, is an illustrative example of deep learning with nnetsauce. Many, more advanced things could be tried. In this example, predictive accuracy of the model increases as new layers are added to the stack.

The first layer is a Bayesian ridge regression. Model accuracy (Root Mean Squared Error, RMSE) is equal to 63.56. The second layer notably uses 3 additional features, an hyperbolic tangent activation function and the first layer; accuracy is 61.76. To finish, the third layer uses 5 additional features, a sigmoid activation function and the second layer. The final accuracy, after adding this third layer is equal to: 61.68.

    import nnetsauce as ns
    from sklearn import datasets, metrics

    diabetes = datasets.load_diabetes()
    X = diabetes.data 
    y = diabetes.target
    
    # layer 1 (base layer) ----
    layer1_regr = linear_model.BayesianRidge()
    layer1_regr.fit(X[0:100,:], y[0:100])
    # RMSE score
    np.sqrt(metrics.mean_squared_error(y[100:125], layer1_regr.predict(X[100:125,:])))


    # layer 2 using layer 1 ----
    layer2_regr = ns.CustomRegressor(obj = layer1_regr, n_hidden_features=3, 
                            direct_link=True, bias=True, 
                            nodes_sim='sobol', activation_name='tanh', 
                            n_clusters=2)
    layer2_regr.fit(X[0:100,:], y[0:100])


    # RMSE score
    np.sqrt(layer2_regr.score(X[100:125,:], y[100:125]))

    # layer 3 using layer 2 ----
    layer3_regr = ns.CustomRegressor(obj = layer2_regr, n_hidden_features=5, 
                direct_link=True, bias=True, 
                nodes_sim='hammersley', activation_name='sigmoid', 
                n_clusters=2)
    layer3_regr.fit(X[0:100,:], y[0:100])

    # RMSE score
    np.sqrt(layer3_regr.score(X[100:125,:], y[100:125]))

nnetsauce

2019-03-13T00:00:00+00:00

nnetsauce is a Python package for Statistical/Machine learning built on top of Numpy, Scipy, and scikit-learn under the BSD license. In nnetsauce, pattern recognition is achieved by combining single-layer networks (SLNN). These SLNN building blocks constitute the basis of many custom models that can be built by the user, including models with deeper learning architectures.

Examples of use, for illustration: nnetsauce can (for example) be trained to distinguish between malignant or benign tumors, depending on their characteristics (see examples on Github), with a certain degree of confidence . Or, by using your historical data of income and expenditures, it can be trained to forecast your savings in the next months.

The package is very new. For those who want to try it, give feedback, make it available in R, contribute to the development on Github, feel free to jump in.

T. Moudiki's Webpage - nnetsauce

Gradient-Boosting anything (alert: high performance): Part4, Time series forecasting

1 - Hold-out set cross-validation

2 - Individual examples

Gradient-Boosting anything (alert: high performance): Part3, Histogram-based boosting

My presentation at ISF 2024 conference (slides with nnetsauce probabilistic forecasting news)

10 uncertainty quantification methods in nnetsauce forecasting

Forecasting with XGBoost embedded in Quasi-Randomized Neural Networks

Forecasting Monthly Airline Passenger Numbers with Quasi-Randomized Neural Networks

0 - Install and load packages

1 - Load AirPassengers data

2 - Train a quasi-randomized neural network based on Ridge regression

3 - Forecasting 40 steps ahead

Automated hyperparameter tuning using any conformalized surrogate

0 - Install and load packages

1 - Cross-validation and hyperparameter tuning

2 - Out-of-sample scores

Recognizing handwritten digits with Ridge2Classifier

0 - Install and load packages

1 - Cross-validation and hyperparameter tuning

2 - Out-of-sample scores

A plethora of datasets at your fingertips Part3: how many times do couples cheat on each other?

0 - Install packages

1- Import data from R universe API

2 - Data preprocessing

3 - Automated Machine Learning with nnetsauce

4 - Explaining best model’s “decisions”

5 - Prediction interval on test set

nnetsauce's introduction as of 2024-02-11 (new version 0.17.0)

0 - What is nnetsauce?

1 - Installing nnetsauce for Python

2 - Automated Machine Learning (AutoML) with nnetsauce classifiers on Wisconsin breast cancer data set

dependencies

DeepMTS, a Deep Learning Model for Multivariate Time Series

DeepMTS in nnetsauce v0.16.3 for Multivariate time series (MTS)

1 - Install

2 - DeepMTS

A classifier that's very accurate (and deep) Pt.2: there are > 90 classifiers in nnetsauce

0 - Install packages

1 - Lazy classifiers

1 - 1 breast cancer data

1 - 2 - iris data

1 - 3 - wine data

1 - 4 - digits data

2 - Lazy Deep classifiers

2 - 1 breast cancer data

2 - 2 - iris data

2 - 3 - wine data

2 - 4 - digits data

Quasi-randomized nnetworks in Julia, Python and R

1 - Install Julia

2 - Example using a nnetsauce classifier in Julia language

A classifier that's very accurate (and deep)

Contents

0 - Install and import packages

1 - breast cancer data

2 - iris data

3 - wine data

4 - digits data

5 - R examples

AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Pt.2: multivariate time series forecasting

1 - Install

2 - MTS

2 - 1 - nnetsauce.MTS

2 - 2 - VAR

2 - 3 - VECM

AutoML in nnetsauce (randomized and quasi-randomized nnetworks)

2 - Installation

2 - Classification

2 - 1 Loading the Dataset

2 - 2 Building the classification model using LazyPredict

3 - Regression

Version v0.14.0 of nnetsauce for R and Python

An infinity of time series forecasting models in nnetsauce (Part 2 with uncertainty quantification)

0 - Install and import packages + get data

1 - Simulation-based forecasting using Kernel Density Estimation

1 - 1 With Ridge regression

1 - 2 With Random Forest

2 - Bayesian Forecasting

Appendix

1 - Load `AirPassengers` data

3 - Automated Machine Learning with `nnetsauce`

0 - What is `nnetsauce`?

1 - Installing `nnetsauce` for Python

2 - Automated Machine Learning (AutoML) with `nnetsauce` classifiers on Wisconsin breast cancer data set

`DeepMTS` in nnetsauce v0.16.3 for Multivariate time series (MTS)

2 - `DeepMTS`

2 - 1 - `nnetsauce.MTS`

Create an Explainer object in order to understand `LSBoost` decisions

Examples of use of `nnetsauce`