crossval is an R package which contains generic functions for cross-validation. Two weeks ago, I presented an example of time series cross-validation based on crossval. This week’s post is about cross-validation on a grid of hyperparameters. glmnet is used as statistical learning model for the demo, but it could be any other package of your choice.
Installing and loading the packages
Installing crossval from GitHub (in R console):
devtools::install_github("thierrymoudiki/crossval")
Loading packages:
library(glmnet)
library(crossval)
Load mtcars dataset
data("mtcars")
df <- mtcars[, c(1, 2, 3, 4, 6, 11)]
summary(df)
Create response and explanatory variables from mtcars dataset
X <- as.matrix(df[, -1]) # explanatory variables
y <- df$mpg # response
Grid of hyperparameters for glmnet
tuning_grid <- expand.grid(alpha = c(0, 0.5, 1),
lambda = c(0.01, 0.1, 1))
n_params <- nrow(tuning_grid)
print(tuning_grid)
## alpha lambda
## 1 0.0 0.01
## 2 0.5 0.01
## 3 1.0 0.01
## 4 0.0 0.10
## 5 0.5 0.10
## 6 1.0 0.10
## 7 0.0 1.00
## 8 0.5 1.00
## 9 1.0 1.00
Grid search cross-validation
- list of cross-validation results
- 5-fold cross-validation (
k) - repeated 3 times (
repeats) - cross-validation of 80% of the data (
p) - validation on the remaining 20%
cv_results <- lapply(1:n_params,
function(i)
crossval::crossval_ml(
x = X,
y = y,
k = 5,
repeats = 3,
p = 0.8,
fit_func = glmnet::glmnet,
predict_func = predict.glmnet,
packages = c("glmnet", "Matrix"),
fit_params = list(alpha = tuning_grid[i, "alpha"],
lambda = tuning_grid[i, "lambda"])
))
names(cv_results) <- paste0("params_set", 1:n_params)
Print grid search results
Remarks are welcome.
print(cv_results)
## $params_set1
## $params_set1$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.7116571 3.4204585 2.970296
## fold_validation_1 1.1310676 2.1443185 2.038922
## fold_training_2 1.7335414 1.0317404 3.740119
## fold_validation_2 1.6528925 1.5592805 0.905873
## fold_training_3 2.9526843 4.4059576 3.063401
## fold_validation_3 2.4348686 0.9470344 1.227135
## fold_training_4 4.3206047 3.7097429 3.252773
## fold_validation_4 0.8305158 1.7408722 1.793542
## fold_training_5 1.3484699 1.9396528 1.322698
## fold_validation_5 1.4838844 1.8029411 1.288075
##
## $params_set1$mean_training
## [1] 2.79492
##
## $params_set1$mean_validation
## [1] 1.532082
##
## $params_set1$sd_training
## [1] 1.089414
##
## $params_set1$sd_validation
## [1] 0.4773198
##
## $params_set1$median_training
## [1] 2.970296
##
## $params_set1$median_validation
## [1] 1.559281
##
##
## $params_set2
## $params_set2$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.6942232 3.4034435 2.9509207
## fold_validation_1 1.1283288 2.1212168 2.0922106
## fold_training_2 1.7071382 1.0236950 3.7337454
## fold_validation_2 1.6183054 1.5395739 0.9289049
## fold_training_3 2.9572493 4.3913568 3.0347475
## fold_validation_3 2.4458845 0.9670278 1.2149724
## fold_training_4 4.3683721 3.6924562 3.2383772
## fold_validation_4 0.8786582 1.7168194 1.7714379
## fold_training_5 1.3451998 1.9338217 1.3169037
## fold_validation_5 1.4832682 1.7971002 1.2850981
##
## $params_set2$mean_training
## [1] 2.78611
##
## $params_set2$mean_validation
## [1] 1.532587
##
## $params_set2$sd_training
## [1] 1.093607
##
## $params_set2$sd_validation
## [1] 0.470716
##
## $params_set2$median_training
## [1] 2.957249
##
## $params_set2$median_validation
## [1] 1.539574
##
##
## $params_set3
## $params_set3$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.6762742 3.385479 2.9318273
## fold_validation_1 1.1267161 2.094206 2.1505220
## fold_training_2 1.6851155 1.017365 3.7272127
## fold_validation_2 1.5972579 1.519639 0.9543918
## fold_training_3 2.9614653 4.376096 3.0052024
## fold_validation_3 2.4567021 0.989157 1.2033089
## fold_training_4 4.4107761 3.674386 3.2273064
## fold_validation_4 0.9223574 1.691938 1.7506447
## fold_training_5 1.3421543 1.928040 1.3124042
## fold_validation_5 1.4833113 1.791768 1.2834694
##
## $params_set3$mean_training
## [1] 2.777407
##
## $params_set3$mean_validation
## [1] 1.534359
##
## $params_set3$sd_training
## [1] 1.096777
##
## $params_set3$sd_validation
## [1] 0.4656268
##
## $params_set3$median_training
## [1] 2.961465
##
## $params_set3$median_validation
## [1] 1.519639
##
##
## $params_set4
## $params_set4$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.582406 3.2605565 2.864273
## fold_validation_1 1.168777 1.9268152 2.078255
## fold_training_2 1.650031 0.8984717 3.686839
## fold_validation_2 1.482450 1.4721014 1.017757
## fold_training_3 2.708588 4.2802020 2.939830
## fold_validation_3 2.235334 1.0667584 1.204211
## fold_training_4 4.466894 3.5879682 3.081803
## fold_validation_4 1.004771 1.5646289 1.570059
## fold_training_5 1.326640 1.9144285 1.380771
## fold_validation_5 1.509964 1.7885983 1.335888
##
## $params_set4$mean_training
## [1] 2.708647
##
## $params_set4$mean_validation
## [1] 1.495091
##
## $params_set4$sd_training
## [1] 1.086611
##
## $params_set4$sd_validation
## [1] 0.3817352
##
## $params_set4$median_training
## [1] 2.864273
##
## $params_set4$median_validation
## [1] 1.48245
##
##
## $params_set5
## $params_set5$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.5706795 3.1103749 2.803793
## fold_validation_1 1.2068333 1.7463001 2.075065
## fold_training_2 1.5011386 0.8148667 3.688756
## fold_validation_2 1.3987359 1.3301017 1.001877
## fold_training_3 2.7010045 4.2517543 2.747419
## fold_validation_3 2.2688901 1.1557910 1.178889
## fold_training_4 4.4448265 3.4750530 3.016761
## fold_validation_4 0.9854596 1.4860992 1.437257
## fold_training_5 1.3487225 1.8742370 1.295727
## fold_validation_5 1.5372721 1.7800714 1.311683
##
## $params_set5$mean_training
## [1] 2.643008
##
## $params_set5$mean_validation
## [1] 1.460022
##
## $params_set5$sd_training
## [1] 1.093856
##
## $params_set5$sd_validation
## [1] 0.3720159
##
## $params_set5$median_training
## [1] 2.747419
##
## $params_set5$median_validation
## [1] 1.398736
##
##
## $params_set6
## $params_set6$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.5990228 2.9781640 2.7493269
## fold_validation_1 1.2121089 1.5799814 2.0848477
## fold_training_2 1.4225076 0.7604152 3.6906583
## fold_validation_2 1.4216262 1.2543630 0.9884764
## fold_training_3 2.7409312 4.2492745 2.7175981
## fold_validation_3 2.3240529 1.1598608 1.1607340
## fold_training_4 4.4339739 3.4654770 3.0117350
## fold_validation_4 0.9800525 1.4991208 1.4168583
## fold_training_5 1.3765304 1.8415788 1.3257447
## fold_validation_5 1.5496021 1.8006454 1.3220442
##
## $params_set6$mean_training
## [1] 2.624196
##
## $params_set6$mean_validation
## [1] 1.450292
##
## $params_set6$sd_training
## [1] 1.097017
##
## $params_set6$sd_validation
## [1] 0.3811666
##
## $params_set6$median_training
## [1] 2.740931
##
## $params_set6$median_validation
## [1] 1.416858
##
##
## $params_set7
## $params_set7$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.698210 2.885301 2.455576
## fold_validation_1 1.551401 1.704756 1.716643
## fold_training_2 1.783057 1.028166 3.528652
## fold_validation_2 1.688929 1.457255 1.192856
## fold_training_3 2.635762 3.951937 2.764754
## fold_validation_3 2.325906 1.361088 1.478338
## fold_training_4 4.383367 3.622788 2.966129
## fold_validation_4 1.262743 1.758041 1.628874
## fold_training_5 1.520805 1.968637 1.384429
## fold_validation_5 1.747330 2.061490 1.586987
##
## $params_set7$mean_training
## [1] 2.638505
##
## $params_set7$mean_validation
## [1] 1.634842
##
## $params_set7$sd_training
## [1] 0.9764259
##
## $params_set7$sd_validation
## [1] 0.2898281
##
## $params_set7$median_training
## [1] 2.69821
##
## $params_set7$median_validation
## [1] 1.628874
##
##
## $params_set8
## $params_set8$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 2.966475 2.806465 1.737976
## fold_validation_1 1.692210 1.804410 1.498461
## fold_training_2 1.392634 1.104673 3.578175
## fold_validation_2 2.068470 1.499582 1.163872
## fold_training_3 2.684285 3.930335 2.611488
## fold_validation_3 2.543810 1.441189 1.498748
## fold_training_4 4.269152 3.760451 3.327202
## fold_validation_4 1.381628 2.067037 2.049550
## fold_training_5 1.771081 2.323059 1.777073
## fold_validation_5 1.946920 2.405880 1.787871
##
## $params_set8$mean_training
## [1] 2.669368
##
## $params_set8$mean_validation
## [1] 1.789976
##
## $params_set8$sd_training
## [1] 0.9775324
##
## $params_set8$sd_validation
## [1] 0.3908084
##
## $params_set8$median_training
## [1] 2.684285
##
## $params_set8$median_validation
## [1] 1.787871
##
##
## $params_set9
## $params_set9$folds
## repeat_1 repeat_2 repeat_3
## fold_training_1 3.254495 2.789325 1.094701
## fold_validation_1 1.878893 1.938494 1.581836
## fold_training_2 1.546198 1.179068 3.647095
## fold_validation_2 2.495703 1.623228 1.219294
## fold_training_3 2.478050 3.922693 2.414970
## fold_validation_3 2.594489 1.592061 1.575155
## fold_training_4 4.171757 3.904126 3.695321
## fold_validation_4 1.754051 2.395155 2.455114
## fold_training_5 2.053809 2.660005 2.127650
## fold_validation_5 2.177259 2.741885 2.012704
##
## $params_set9$mean_training
## [1] 2.729284
##
## $params_set9$mean_validation
## [1] 2.002355
##
## $params_set9$sd_training
## [1] 1.014183
##
## $params_set9$sd_validation
## [1] 0.454853
##
## $params_set9$median_training
## [1] 2.660005
##
## $params_set9$median_validation
## [1] 1.938494
Note: I am currently looking for a gig. You can hire me on Malt or send me an email: thierry dot moudiki at pm dot me. I can do descriptive statistics, data preparation, feature engineering, model calibration, training and validation, and model outputs’ interpretation. I am fluent in Python, R, SQL, Microsoft Excel, Visual Basic (among others) and French. My résumé? Here!