Boosting nonlinear penalized least squares

21 Nov 2020 | RSS | Back to list of posts | Toggle dark mode | Hire me on Malt | Hire me on Fiverr | Hire me on Upwork

Today, give a try to Techtonique web app, a tool designed to help you make informed, data-driven decisions using Mathematics, Statistics, Machine Learning, and Data Visualization

Follow me on LinkedIn

1

1

1

1

---

For some reasons I couldn’t foresee, there’s been no blog post here on november 13 and november 20. So, here is the post about LSBoost announced here a few weeks ago.

First things first, what is LSBoost? Gradient boosted nonlinear penalized least squares. More precisely in LSBoost, the ensembles’ base learners are penalized, randomized neural networks.

These previous posts, with several Python and R examples, constitute a good introduction to LSBoost:

• https://thierrymoudiki.github.io/blog/2020/07/24/python/r/lsboost/explainableml/mlsauce/xai-boosting

• https://thierrymoudiki.github.io/blog/2020/07/31/python/r/lsboost/explainableml/mlsauce/xai-boosting-2

More recently, I’ve also written a more formal, short introduction to LSBoost:

• LSBoost, gradient boosted penalized nonlinear least squares (pdf).

The paper’s code – and more insights on LSBoost – can be found in the following Jupyter notebook:

• https://github.com/Techtonique/mlsauce/blob/master/mlsauce/demo/thierrymoudiki_211120_lsboost_sensi_to_hyperparams.ipynb

Comments, suggestions are welcome as usual.

Comments powered by Talkyard.

Previous publications

1. survivalist: Probabilistic model-agnostic survival analysis using scikit-learn, glmnet, xgboost, lightgbm, pytorch, keras, nnetsauce and mlsauce Dec 15, 2024
2. Model-agnostic 'Bayesian' optimization (for hyperparameter tuning) using conformalized surrogates in GPopt Dec 9, 2024
3. You can beat Forecasting LLMs (Large Language Models a.k.a foundation models) with nnetsauce.MTS Pt.2: Generic Gradient Boosting Dec 1, 2024
4. You can beat Forecasting LLMs (Large Language Models a.k.a foundation models) with nnetsauce.MTS Nov 24, 2024
5. Unified interface and conformal prediction (calibrated prediction intervals) for R package forecast (and 'affiliates') Nov 23, 2024
6. GLMNet in Python: Generalized Linear Models Nov 18, 2024
7. Gradient-Boosting anything (alert: high performance): Part4, Time series forecasting Nov 10, 2024
8. Predictive scenarios simulation in R, Python and Excel using Techtonique API Nov 3, 2024
9. Chat with your tabular data in www.techtonique.net Oct 30, 2024
10. Gradient-Boosting anything (alert: high performance): Part3, Histogram-based boosting Oct 28, 2024
11. R editor and SQL console (in addition to Python editors) in www.techtonique.net Oct 21, 2024
12. R and Python consoles + JupyterLite in www.techtonique.net Oct 15, 2024
13. Gradient-Boosting anything (alert: high performance): Part2, R version Oct 14, 2024
14. Gradient-Boosting anything (alert: high performance) Oct 6, 2024
15. Benchmarking 30 statistical/Machine Learning models on the VN1 Forecasting -- Accuracy challenge Oct 4, 2024
16. Automated random variable distribution inference using Kullback-Leibler divergence and simulating best-fitting distribution Oct 2, 2024
17. Forecasting in Excel using Techtonique's Machine Learning APIs under the hood Sep 30, 2024
18. Techtonique web app for data-driven decisions using Mathematics, Statistics, Machine Learning, and Data Visualization Sep 25, 2024
19. Parallel for loops (Map or Reduce) + New versions of nnetsauce and ahead Sep 16, 2024
20. Adaptive (online/streaming) learning with uncertainty quantification using Polyak averaging in learningmachine Sep 10, 2024
21. New versions of nnetsauce and ahead Sep 9, 2024
22. Prediction sets and prediction intervals for conformalized Auto XGBoost, Auto LightGBM, Auto CatBoost, Auto GradientBoosting Sep 2, 2024
23. Quick/automated R package development workflow (assuming you're using macOS or Linux) Part2 Aug 30, 2024
24. R package development workflow (assuming you're using macOS or Linux) Aug 27, 2024
25. A new method for deriving a nonparametric confidence interval for the mean Aug 26, 2024
26. Conformalized adaptive (online/streaming) learning using learningmachine in Python and R Aug 19, 2024
27. Bayesian (nonlinear) adaptive learning Aug 12, 2024
28. Auto XGBoost, Auto LightGBM, Auto CatBoost, Auto GradientBoosting Aug 5, 2024
29. Copulas for uncertainty quantification in time series forecasting Jul 28, 2024
30. Forecasting uncertainty: sequential split conformal prediction + Block bootstrap (web app) Jul 22, 2024
31. learningmachine for Python (new version) Jul 15, 2024
32. learningmachine v2.0.0: Machine Learning with explanations and uncertainty quantification Jul 8, 2024
33. My presentation at ISF 2024 conference (slides with nnetsauce probabilistic forecasting news) Jul 3, 2024
34. 10 uncertainty quantification methods in nnetsauce forecasting Jul 1, 2024
35. Forecasting with XGBoost embedded in Quasi-Randomized Neural Networks Jun 24, 2024
36. Forecasting Monthly Airline Passenger Numbers with Quasi-Randomized Neural Networks Jun 17, 2024
37. Automated hyperparameter tuning using any conformalized surrogate Jun 9, 2024
38. Recognizing handwritten digits with Ridge2Classifier Jun 3, 2024
39. Forecasting the Economy May 27, 2024
40. A detailed introduction to Deep Quasi-Randomized 'neural' networks May 19, 2024
41. Probability of receiving a loan; using learningmachine May 12, 2024
42. mlsauce's `v0.18.2`: various examples and benchmarks with dimension reduction May 6, 2024
43. mlsauce's `v0.17.0`: boosting with Elastic Net, polynomials and heterogeneity in explanatory variables Apr 29, 2024
44. mlsauce's `v0.13.0`: taking into account inputs heterogeneity through clustering Apr 21, 2024
45. mlsauce's `v0.12.0`: prediction intervals for LSBoostRegressor Apr 15, 2024
46. Conformalized predictive simulations for univariate time series on more than 250 data sets Apr 7, 2024
47. learningmachine v1.1.2: for Python Apr 1, 2024
48. learningmachine v1.0.0: prediction intervals around the probability of the event 'a tumor being malignant' Mar 25, 2024
49. Bayesian inference and conformal prediction (prediction intervals) in nnetsauce v0.18.1 Mar 18, 2024
50. Multiple examples of Machine Learning forecasting with ahead Mar 11, 2024
51. rtopy (v0.1.1): calling R functions in Python Mar 4, 2024
52. ahead forecasting (v0.10.0): fast time series model calibration and Python plots Feb 26, 2024
53. A plethora of datasets at your fingertips Part3: how many times do couples cheat on each other? Feb 19, 2024
54. nnetsauce's introduction as of 2024-02-11 (new version 0.17.0) Feb 11, 2024
55. Tuning Machine Learning models with GPopt's new version Part 2 Feb 5, 2024
56. Tuning Machine Learning models with GPopt's new version Jan 29, 2024
57. Subsampling continuous and discrete response variables Jan 22, 2024
58. DeepMTS, a Deep Learning Model for Multivariate Time Series Jan 15, 2024
59. A classifier that's very accurate (and deep) Pt.2: there are > 90 classifiers in nnetsauce Jan 8, 2024
60. learningmachine: prediction intervals for conformalized Kernel ridge regression and Random Forest Jan 1, 2024
61. A plethora of datasets at your fingertips Part2: how many times do couples cheat on each other? Descriptive analytics, interpretability and prediction intervals using conformal prediction Dec 25, 2023
62. Diffusion models in Python with esgtoolkit (Part2) Dec 18, 2023
63. Diffusion models in Python with esgtoolkit Dec 11, 2023
64. Julia packaging at the command line Dec 4, 2023
65. Quasi-randomized nnetworks in Julia, Python and R Nov 27, 2023
66. A plethora of datasets at your fingertips Nov 20, 2023
67. A classifier that's very accurate (and deep) Nov 12, 2023
68. mlsauce version 0.8.10: Statistical/Machine Learning with Python and R Nov 5, 2023
69. AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Pt.2: multivariate time series forecasting Oct 29, 2023
70. AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Oct 22, 2023
71. Version v0.14.0 of nnetsauce for R and Python Oct 16, 2023
72. A diffusion model: G2++ Oct 9, 2023
73. Diffusion models in ESGtoolkit + announcements Oct 2, 2023
74. An infinity of time series forecasting models in nnetsauce (Part 2 with uncertainty quantification) Sep 25, 2023
75. (News from) forecasting in Python with ahead (progress bars and plots) Sep 18, 2023
76. Forecasting in Python with ahead Sep 11, 2023
77. Risk-neutralize simulations Sep 4, 2023
78. Comparing cross-validation results using crossval_ml and boxplots Aug 27, 2023
79. Reminder Apr 30, 2023
80. Did you ask ChatGPT about who you are? Apr 16, 2023
81. A new version of nnetsauce (randomized and quasi-randomized 'neural' networks) Apr 2, 2023
82. Simple interfaces to the forecasting API Nov 23, 2022
83. A web application for forecasting in Python, R, Ruby, C#, JavaScript, PHP, Go, Rust, Java, MATLAB, etc. Nov 2, 2022
84. Prediction intervals (not only) for Boosted Configuration Networks in Python Oct 5, 2022
85. Boosted Configuration (neural) Networks Pt. 2 Sep 3, 2022
86. Boosted Configuration (_neural_) Networks for classification Jul 21, 2022
87. A Machine Learning workflow using Techtonique Jun 6, 2022
88. Super Mario Bros © in the browser using PyScript May 8, 2022
89. News from ESGtoolkit, ycinterextra, and nnetsauce Apr 4, 2022
90. Explaining a Keras _neural_ network predictions with the-teller Mar 11, 2022
91. New version of nnetsauce -- various quasi-randomized networks Feb 12, 2022
92. A dashboard illustrating bivariate time series forecasting with `ahead` Jan 14, 2022
93. Hundreds of Statistical/Machine Learning models for univariate time series, using ahead, ranger, xgboost, and caret Dec 20, 2021
94. Forecasting with `ahead` (Python version) Dec 13, 2021
95. Tuning and interpreting LSBoost Nov 15, 2021
96. Time series cross-validation using `crossvalidation` (Part 2) Nov 7, 2021
97. Fast and scalable forecasting with ahead::ridge2f Oct 31, 2021
98. Automatic Forecasting with `ahead::dynrmf` and Ridge regression Oct 22, 2021
99. Forecasting with `ahead` Oct 15, 2021
100. Classification using linear regression Sep 26, 2021
101. `crossvalidation` and random search for calibrating support vector machines Aug 6, 2021
102. parallel grid search cross-validation using `crossvalidation` Jul 31, 2021
103. `crossvalidation` on R-universe, plus a classification example Jul 23, 2021
104. Documentation and source code for GPopt, a package for Bayesian optimization Jul 2, 2021
105. Hyperparameters tuning with GPopt Jun 11, 2021
106. A forecasting tool (API) with examples in curl, R, Python May 28, 2021
107. Bayesian Optimization with GPopt Part 2 (save and resume) Apr 30, 2021
108. Bayesian Optimization with GPopt Apr 16, 2021
109. Compatibility of nnetsauce and mlsauce with scikit-learn Mar 26, 2021
110. Explaining xgboost predictions with the teller Mar 12, 2021
111. An infinity of time series models in nnetsauce Mar 6, 2021
112. New activation functions in mlsauce's LSBoost Feb 12, 2021
113. 2020 recap, Gradient Boosting, Generalized Linear Models, AdaOpt with nnetsauce and mlsauce Dec 29, 2020
114. A deeper learning architecture in nnetsauce Dec 18, 2020
115. Classify penguins with nnetsauce's MultitaskClassifier Dec 11, 2020
116. Bayesian forecasting for uni/multivariate time series Dec 4, 2020
117. Generalized nonlinear models in nnetsauce Nov 28, 2020
118. Boosting nonlinear penalized least squares Nov 21, 2020
119. Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates Nov 6, 2020
120. NEWS Oct 30, 2020
121. A glimpse into my PhD journey Oct 23, 2020
122. Submitting R package to CRAN Oct 16, 2020
123. Simulation of dependent variables in ESGtoolkit Oct 9, 2020
124. Forecasting lung disease progression Oct 2, 2020
125. New nnetsauce Sep 25, 2020
126. Technical documentation Sep 18, 2020
127. A new version of nnetsauce, and a new Techtonique website Sep 11, 2020
128. Back next week, and a few announcements Sep 4, 2020
129. Explainable 'AI' using Gradient Boosted randomized networks Pt2 (the Lasso) Jul 31, 2020
130. LSBoost: Explainable 'AI' using Gradient Boosted randomized networks (with examples in R and Python) Jul 24, 2020
131. nnetsauce version 0.5.0, randomized neural networks on GPU Jul 17, 2020
132. Maximizing your tip as a waiter (Part 2) Jul 10, 2020
133. New version of mlsauce, with Gradient Boosted randomized networks and stump decision trees Jul 3, 2020
134. Announcements Jun 26, 2020
135. Parallel AdaOpt classification Jun 19, 2020
136. Comments section and other news Jun 12, 2020
137. Maximizing your tip as a waiter Jun 5, 2020
138. AdaOpt classification on MNIST handwritten digits (without preprocessing) May 29, 2020
139. AdaOpt (a probabilistic classifier based on a mix of multivariable optimization and nearest neighbors) for R May 22, 2020
140. AdaOpt May 15, 2020
141. Custom errors for cross-validation using crossval::crossval_ml May 8, 2020
142. Documentation+Pypi for the `teller`, a model-agnostic tool for Machine Learning explainability May 1, 2020
143. Encoding your categorical variables based on the response variable and correlations Apr 24, 2020
144. Linear model, xgboost and randomForest cross-validation using crossval::crossval_ml Apr 17, 2020
145. Grid search cross-validation using crossval Apr 10, 2020
146. Documentation for the querier, a query language for Data Frames Apr 3, 2020
147. Time series cross-validation using crossval Mar 27, 2020
148. On model specification, identification, degrees of freedom and regularization Mar 20, 2020
149. Import data into the querier (now on Pypi), a query language for Data Frames Mar 13, 2020
150. R notebooks for nnetsauce Mar 6, 2020
151. Version 0.4.0 of nnetsauce, with fruits and breast cancer classification Feb 28, 2020
152. Create a specific feed in your Jekyll blog Feb 21, 2020
153. Git/Github for contributing to package development Feb 14, 2020
154. Feedback forms for contributing Feb 7, 2020
155. nnetsauce for R Jan 31, 2020
156. A new version of nnetsauce (v0.3.1) Jan 24, 2020
157. ESGtoolkit, a tool for Monte Carlo simulation (v0.2.0) Jan 17, 2020
158. Search bar, new year 2020 Jan 10, 2020
159. 2019 Recap, the nnetsauce, the teller and the querier Dec 20, 2019
160. Understanding model interactions with the `teller` Dec 13, 2019
161. Using the `teller` on a classifier Dec 6, 2019
162. Benchmarking the querier's verbs Nov 29, 2019
163. Composing the querier's verbs for data wrangling Nov 22, 2019
164. Comparing and explaining model predictions with the teller Nov 15, 2019
165. Tests for the significance of marginal effects in the teller Nov 8, 2019
166. Introducing the teller Nov 1, 2019
167. Introducing the querier Oct 25, 2019
168. Prediction intervals for nnetsauce models Oct 18, 2019
169. Using R in Python for statistical learning/data science Oct 11, 2019
170. Model calibration with `crossval` Oct 4, 2019
171. Bagging in the nnetsauce Sep 25, 2019
172. Adaboost learning with nnetsauce Sep 18, 2019
173. Change in blog's presentation Sep 4, 2019
174. nnetsauce on Pypi Jun 5, 2019
175. More nnetsauce (examples of use) May 9, 2019
176. nnetsauce Mar 13, 2019
177. crossval Mar 13, 2019
178. test Mar 10, 2019