Introducing the querier

25 Oct 2019 | RSS | Back to list of posts | Toggle dark mode

Follow me on LinkedIn

1

1

1

1

---

Data Frames are a way to represent tabular data, that is widely used and useful for Statistical Learning. Basically, a Data Frame = Tabular data + Named columns, and there are different implementations of this data structure, notably in R, Python and Apache Spark. The querier exposes a query language to retrieve data from Python pandas Data Frames, inspired from SQL’s relational databases querying. Currently, the querier can be installed from Github as:

pip install git+https://github.com/Techtonique/querier.git

There are 9 types of operations available in the querier, with no plan to extend that list much further (to maintain a relatively simple mental model). These verbs will look familiar to dplyr users, but the implementation (numpy, pandas and SQLite3 are used) and functions’ signatures are different:

• concat: concatenates 2 Data Frames, either horizontally or vertically

• delete: deletes rows from a Data Frame based on given criteria

• drop: drops columns from a Data Frame

• filtr: filters rows of the Data Frame based on given criteria

• join: joins 2 Data Frames based on given criteria (available for completeness of the interface, this operation is already straightforward in pandas)

• select: selects columns from the Data Frame

• summarize: obtains summaries of data based on grouping columns

• update: updates a column/creates a new column, using an operation given by the user

• request: for operations more complex than the previous 8 ones, makes it possible to directly use a SQL query on the Data Frame

The following notebooks present multiple examples of use of the querier:

• concat example
• delete example
• drop example
• filtr example
• join example
• select example
• summarize example
• update example
• request example

Contributions/remarks are welcome as usual, you can submit a pull request on Github.

Note: I am currently looking for a gig. You can hire me on Malt or send me an email: thierry dot moudiki at pm dot me. I can do descriptive statistics, data preparation, feature engineering, model calibration, training and validation, and model outputs’ interpretation. I am fluent in Python, R, SQL, Microsoft Excel, Visual Basic (among others) and French. My résumé? Here!

Under License Creative Commons Attribution 4.0 International.

Previous publications

1. mlsauce's `v0.13.0`: taking into account inputs heterogeneity through clustering Apr 21, 2024
2. mlsauce's `v0.12.0`: prediction intervals for LSBoostRegressor Apr 15, 2024
3. Conformalized predictive simulations for univariate time series on more than 250 data sets Apr 7, 2024
4. learningmachine v1.1.2: for Python Apr 1, 2024
5. learningmachine v1.0.0: prediction intervals around the probability of the event 'a tumor being malignant' Mar 25, 2024
6. Bayesian inference and conformal prediction (prediction intervals) in nnetsauce v0.18.1 Mar 18, 2024
7. Multiple examples of Machine Learning forecasting with ahead Mar 11, 2024
8. rtopy (v0.1.1): calling R functions in Python Mar 4, 2024
9. ahead forecasting (v0.10.0): fast time series model calibration and Python plots Feb 26, 2024
10. A plethora of datasets at your fingertips Part3: how many times do couples cheat on each other? Feb 19, 2024
11. nnetsauce's introduction as of 2024-02-11 (new version 0.17.0) Feb 11, 2024
12. Tuning Machine Learning models with GPopt's new version Part 2 Feb 5, 2024
13. Tuning Machine Learning models with GPopt's new version Jan 29, 2024
14. Subsampling continuous and discrete response variables Jan 22, 2024
15. DeepMTS, a Deep Learning Model for Multivariate Time Series Jan 15, 2024
16. A classifier that's very accurate (and deep) Pt.2: there are > 90 classifiers in nnetsauce Jan 8, 2024
17. learningmachine: prediction intervals for conformalized Kernel ridge regression and Random Forest Jan 1, 2024
18. A plethora of datasets at your fingertips Part2: how many times do couples cheat on each other? Descriptive analytics, interpretability and prediction intervals using conformal prediction Dec 25, 2023
19. Diffusion models in Python with esgtoolkit (Part2) Dec 18, 2023
20. Diffusion models in Python with esgtoolkit Dec 11, 2023
21. Julia packaging at the command line Dec 4, 2023
22. Quasi-randomized nnetworks in Julia, Python and R Nov 27, 2023
23. A plethora of datasets at your fingertips Nov 20, 2023
24. A classifier that's very accurate (and deep) Nov 12, 2023
25. mlsauce version 0.8.10: Statistical/Machine Learning with Python and R Nov 5, 2023
26. AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Pt.2: multivariate time series forecasting Oct 29, 2023
27. AutoML in nnetsauce (randomized and quasi-randomized nnetworks) Oct 22, 2023
28. Version v0.14.0 of nnetsauce for R and Python Oct 16, 2023
29. A diffusion model: G2++ Oct 9, 2023
30. Diffusion models in ESGtoolkit + announcements Oct 2, 2023
31. An infinity of time series forecasting models in nnetsauce (Part 2 with uncertainty quantification) Sep 25, 2023
32. (News from) forecasting in Python with ahead (progress bars and plots) Sep 18, 2023
33. Forecasting in Python with ahead Sep 11, 2023
34. Risk-neutralize simulations Sep 4, 2023
35. Comparing cross-validation results using crossval_ml and boxplots Aug 27, 2023
36. Reminder Apr 30, 2023
37. Did you ask ChatGPT about who you are? Apr 16, 2023
38. A new version of nnetsauce (randomized and quasi-randomized 'neural' networks) Apr 2, 2023
39. Simple interfaces to the forecasting API Nov 23, 2022
40. A web application for forecasting in Python, R, Ruby, C#, JavaScript, PHP, Go, Rust, Java, MATLAB, etc. Nov 2, 2022
41. Prediction intervals (not only) for Boosted Configuration Networks in Python Oct 5, 2022
42. Boosted Configuration (neural) Networks Pt. 2 Sep 3, 2022
43. Boosted Configuration (_neural_) Networks for classification Jul 21, 2022
44. A Machine Learning workflow using Techtonique Jun 6, 2022
45. Super Mario Bros © in the browser using PyScript May 8, 2022
46. News from ESGtoolkit, ycinterextra, and nnetsauce Apr 4, 2022
47. Explaining a Keras _neural_ network predictions with the-teller Mar 11, 2022
48. New version of nnetsauce -- various quasi-randomized networks Feb 12, 2022
49. A dashboard illustrating bivariate time series forecasting with `ahead` Jan 14, 2022
50. Hundreds of Statistical/Machine Learning models for univariate time series, using ahead, ranger, xgboost, and caret Dec 20, 2021
51. Forecasting with `ahead` (Python version) Dec 13, 2021
52. Tuning and interpreting LSBoost Nov 15, 2021
53. Time series cross-validation using `crossvalidation` (Part 2) Nov 7, 2021
54. Fast and scalable forecasting with ahead::ridge2f Oct 31, 2021
55. Automatic Forecasting with `ahead::dynrmf` and Ridge regression Oct 22, 2021
56. Forecasting with `ahead` Oct 15, 2021
57. Classification using linear regression Sep 26, 2021
58. `crossvalidation` and random search for calibrating support vector machines Aug 6, 2021
59. parallel grid search cross-validation using `crossvalidation` Jul 31, 2021
60. `crossvalidation` on R-universe, plus a classification example Jul 23, 2021
61. Documentation and source code for GPopt, a package for Bayesian optimization Jul 2, 2021
62. Hyperparameters tuning with GPopt Jun 11, 2021
63. A forecasting tool (API) with examples in curl, R, Python May 28, 2021
64. Bayesian Optimization with GPopt Part 2 (save and resume) Apr 30, 2021
65. Bayesian Optimization with GPopt Apr 16, 2021
66. Compatibility of nnetsauce and mlsauce with scikit-learn Mar 26, 2021
67. Explaining xgboost predictions with the teller Mar 12, 2021
68. An infinity of time series models in nnetsauce Mar 6, 2021
69. New activation functions in mlsauce's LSBoost Feb 12, 2021
70. 2020 recap, Gradient Boosting, Generalized Linear Models, AdaOpt with nnetsauce and mlsauce Dec 29, 2020
71. A deeper learning architecture in nnetsauce Dec 18, 2020
72. Classify penguins with nnetsauce's MultitaskClassifier Dec 11, 2020
73. Bayesian forecasting for uni/multivariate time series Dec 4, 2020
74. Generalized nonlinear models in nnetsauce Nov 28, 2020
75. Boosting nonlinear penalized least squares Nov 21, 2020
76. Statistical/Machine Learning explainability using Kernel Ridge Regression surrogates Nov 6, 2020
77. NEWS Oct 30, 2020
78. A glimpse into my PhD journey Oct 23, 2020
79. Submitting R package to CRAN Oct 16, 2020
80. Simulation of dependent variables in ESGtoolkit Oct 9, 2020
81. Forecasting lung disease progression Oct 2, 2020
82. New nnetsauce Sep 25, 2020
83. Technical documentation Sep 18, 2020
84. A new version of nnetsauce, and a new Techtonique website Sep 11, 2020
85. Back next week, and a few announcements Sep 4, 2020
86. Explainable 'AI' using Gradient Boosted randomized networks Pt2 (the Lasso) Jul 31, 2020
87. LSBoost: Explainable 'AI' using Gradient Boosted randomized networks (with examples in R and Python) Jul 24, 2020
88. nnetsauce version 0.5.0, randomized neural networks on GPU Jul 17, 2020
89. Maximizing your tip as a waiter (Part 2) Jul 10, 2020
90. New version of mlsauce, with Gradient Boosted randomized networks and stump decision trees Jul 3, 2020
91. Announcements Jun 26, 2020
92. Parallel AdaOpt classification Jun 19, 2020
93. Comments section and other news Jun 12, 2020
94. Maximizing your tip as a waiter Jun 5, 2020
95. AdaOpt classification on MNIST handwritten digits (without preprocessing) May 29, 2020
96. AdaOpt (a probabilistic classifier based on a mix of multivariable optimization and nearest neighbors) for R May 22, 2020
97. AdaOpt May 15, 2020
98. Custom errors for cross-validation using crossval::crossval_ml May 8, 2020
99. Documentation+Pypi for the `teller`, a model-agnostic tool for Machine Learning explainability May 1, 2020
100. Encoding your categorical variables based on the response variable and correlations Apr 24, 2020
101. Linear model, xgboost and randomForest cross-validation using crossval::crossval_ml Apr 17, 2020
102. Grid search cross-validation using crossval Apr 10, 2020
103. Documentation for the querier, a query language for Data Frames Apr 3, 2020
104. Time series cross-validation using crossval Mar 27, 2020
105. On model specification, identification, degrees of freedom and regularization Mar 20, 2020
106. Import data into the querier (now on Pypi), a query language for Data Frames Mar 13, 2020
107. R notebooks for nnetsauce Mar 6, 2020
108. Version 0.4.0 of nnetsauce, with fruits and breast cancer classification Feb 28, 2020
109. Create a specific feed in your Jekyll blog Feb 21, 2020
110. Git/Github for contributing to package development Feb 14, 2020
111. Feedback forms for contributing Feb 7, 2020
112. nnetsauce for R Jan 31, 2020
113. A new version of nnetsauce (v0.3.1) Jan 24, 2020
114. ESGtoolkit, a tool for Monte Carlo simulation (v0.2.0) Jan 17, 2020
115. Search bar, new year 2020 Jan 10, 2020
116. 2019 Recap, the nnetsauce, the teller and the querier Dec 20, 2019
117. Understanding model interactions with the `teller` Dec 13, 2019
118. Using the `teller` on a classifier Dec 6, 2019
119. Benchmarking the querier's verbs Nov 29, 2019
120. Composing the querier's verbs for data wrangling Nov 22, 2019
121. Comparing and explaining model predictions with the teller Nov 15, 2019
122. Tests for the significance of marginal effects in the teller Nov 8, 2019
123. Introducing the teller Nov 1, 2019
124. Introducing the querier Oct 25, 2019
125. Prediction intervals for nnetsauce models Oct 18, 2019
126. Using R in Python for statistical learning/data science Oct 11, 2019
127. Model calibration with `crossval` Oct 4, 2019
128. Bagging in the nnetsauce Sep 25, 2019
129. Adaboost learning with nnetsauce Sep 18, 2019
130. Change in blog's presentation Sep 4, 2019
131. nnetsauce on Pypi Jun 5, 2019
132. More nnetsauce (examples of use) May 9, 2019
133. nnetsauce Mar 13, 2019
134. crossval Mar 13, 2019
135. test Mar 10, 2019