A Survey of Open Source Automation Tools for Data Science Predictions

Feature interactions are obtained from simple products of existing features. Here, we use the term more broadly and define basic interactions between features to be sums, differences, products, and ratios between numerical features. The PyCaret library (see next section) offers automated generation of products and ratios of existing features as well as of multivariate polynomials generated by base features. The autofeat and featuretools packages also offer a simple interface for quickly generating basic interaction features.

One of the most common feature generation techniques is one-hot encoding. This refers to transforming a categorical feature vector into a collection of bit vectors, each row of which containing a $1$ if and only if the given observation falls into the associated category. One-hot encoding is part of the automated feature engineering stages of several packages including EvalML, PyCaret, and auto-sklearn.

The PyCaret package automates the binning of continuous variables, converting them into categories on the basis of applying one dimensional $K$-means clustering, with number of clusters determined using Sturges’ rule for optimal choice of bins. The package also allows for automatically combining infrequent categories in features with large numbers of categorical values, as reducing the number of categories reduces the resultant number of features obtained when applying one-hot encoding.

Clustering refers to unsupervised partitioning of the dataset examples into a fixed number of clusters. The derived feature from the clustering can be the categorical cluster label, the distance to the cluster representative (cluster centroid, say), or another derived quantity (see [121] for a thorough overview). A host of clustering techniques exist, with $k$-means (having an scikit-learn implementation) and the density-based HDBSCAN ([63]) algorithms being two popular choices for data scientists.

Genetic programming is a common type of genetic algorithm (see, e.g. [99] for a fuller description), a class of algorithm inspired by principles from evolution in the natural world which incorporates fitness selection, genetic recombination (sexual reproduction, although asexual techniques exist as well), and random mutation on a population of solutions to a given problem. These solution populations undergo iterated evolution guiding them towards a superior solution.

Genetic programming is a genetic algorithm whose populations are symbolic expression trees, with intermediate degree $k$ nodes given by primitive $k$-ary operators and with leaves given by symbols representing constants. In the context of feature engineering, each expression tree represents a formula that can be applied to existing features to produce new derived features. The recombination and mutation act on the feature space to produce a robust set of expressions with a fitness function selecting for an optimal property along the way (for instance, small linear correlations and non-redundant expressions,). A popular open source implementation of genetic algorithms, including genetic programming is the DEAP package [37]., though we remark that DEAP may not be straightforward to implement into an automated pipeline as it is not particularly low-code.

An algorithm called deep feature synthesis was presented in [75] which formed the basis for the featuretools Python library. Deep feature synthesis was designed to work with data held in relational databases though reduced forms of it can work on single-table data. The operators used to construct derived features are aggregations and transformations. The former produces the deep part of deep feature synthesis by synthesizing data held in different tables via prescribed relationships among their unique keys and recursively traversing a graph made up of available primitive operations on existing features. The latter consist of more tabular operations within a single table.

The featuretools package implementing this algorithm offers very good graphical and even natural language representations of how the derived features are obtained, making it a particularly excellent choice for data scientists needing to speed up reporting. The PyCaret package offers a somewhat related, though stripped down, version of synthesis by performing descriptive statistics on passed in groups of features as part of its feature engineering pipeline stage.

In [70] the authors use random forests (see [24]) to create synthetic features to feed into a downstream forest. The authors rely on interpreting the terminal node size hyperparameter of a random forest as a form of smoothness control and generate synthetic features by using the predicted values from a collection of random forests obtained with different node sizes. For classification problems, these features are $\#classes$-dimensional vectors of estimated probability for each class, from each random forest. The synthetic features are combined with the original base features to feed as input to a downstream random forest. This general paradigm of creating synthetic features, extended to include other classifier models, is used by the AutoML system TPOT and has a straightforward implementation using scikit-learn’s ensemble module as well.

There are often cases where datasets in question have a natural ordering to them, whether time-based or positional. In the case where there in a natural one-dimensional ordering to the observational data, a host of common features present themselves as reasonable and informative. We consider the case where each training example corresponds to various non-sequential features in addition to one or more time series. An input example, the $i$’th say, may therefore take the form

where in the above, the $f_j^{\left(i\right)}\in R,j=1,...,n$ and $\left\{X_k^{\left(i\right)}\right(t){\}}_{t\in \left[t_0\right(X_k^{\left(i\right)}),t_{final}(X_k^{\left(i\right)}\left)\right]}$ for $k=1,...,m$ each represent time series, at possibly different cadence, length, and even type (categorical or numeric) associated to the observation $x^{\left(i\right)}$. In this case, the feature generation stage consists of “flattening” the dataset by reducing each time series associated to an observation to a collection of single value, unordered, derived features.

The tsfresh (Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests, see [94]) Python package is a common library for generating a host of traditional features for ordered datasets. It computes 794 different features from 63 characterization methods. While deep feature synthesis can handle time series aggregations, the specificity of the methods used in tsfresh, including spectral features, make it extremely well suited for extracting informative derived features quickly.

We close this subsection by referring the interested reader to the article [95] which covers some of the additional automated feature generation tools available as part of automated forecasting libraries. It presents a discussion of some of the general classes of features these tools can obtain.

While traditional dimension reduction techniques were often not aimed at increasing the discriminatory power of downstream models, they have become an increasingly important part of the feature engineering stages of data science pipelines. These methods combine both feature generation and selection; the input feature vectors are not simply reduced in quantity, they are transformed as well. The transformation and selection are fundamentally entangled and not independently modularizable in a straightforward way.

The mutual information, $I(X;Y)$, between two random variables $X$ and $Y$, with distribution functions $p_X$, $p_Y$, respectively, and joint distribution $p_{XY}$, is given by

which measures the dependency of variables $X$, $Y$. In the above, the expectation is taken with respect to the joint distribution of $(X,Y)$. Among other properties, it is straightforward to show that $I(X;Y)$ is zero if and only if the variables $X$ and $Y$ are statistically independent. Information-based filtering are techniques that, at their core, aim to reduce features based on higher mutual information. Mutual information can be estimated using the scikit-learn package which is based on algorithms discussed in [131, 78] on entropy estimations. Mutual information has a well-known bias in favour of larger numbers of values, see, e.g. [126, 61].

Principal feature analysis [90] is a technique to select a subset of the derived features with low mutual information. The method is based on the observation that the absolute value vectors produced by taking the rows of the (truncated) eigenvector matrix obtained from the eigendecomposition of the covariance matrix are in close Euclidean proximity for features with high mutual information. Based on this observation, $k$-means clustering in the lower-dimensional space produced from the row span of these absolute value vectors will find clusters with similar informational content, and the original features corresponding to the best representatives from each cluster will be the ultimately selected features. While the author could not find an open source Python package implementing principal feature analysis, this algorithm is a straightforward chaining of PCA and clustering found within the scikit-learn package. Of course, a clear disadvantage of a straightforward implementation is the dependence on the nondeterministic clustering.

While mutual information can directly measure statistical independence, correlations can, at times, be a reasonable proxy (as zero correlation is a necessary condition of independence and a sufficient condition for linear independence) and thus falls into this class of filtering. This is highly intuitive: the smallest set of feature vectors spanning a $d$-dimensional subspace of the feature space will necessarily be linearly independent and, therefore, are uncorrelated. PCA, as discussed in the previous section, falls into this category of selection as the principal components are orthonormal eigenvectors of the empirical covariance matrix and so are uncorrelated. PCA is automated in the PyCaret package (see next section) as part of its feature engineering stage.

In [136] a generalization of PCA, kernel PCA, is introduced in which base features are embedded into a (higher-dimensional) new feature space in which eigenproblems for the empirical covariance matrix in the embedded space can be solved to get projections onto dominant eigenvectors, as in usual PCA (see also [130]). The high dimensional embedding allows for the process to efficiently handle potential nonlinearities in the dataset as explicit embeddings are not actually required to compute the projections, only inner products among embedded vectors are required for the procedure. As with linear PCA, projections can be truncated to projections on the most dominant (in terms of sizes of eigenvalues) eigenvectors. Both PCA and kernel PCA rely on covariance matrices and all techniques reliant on covariance estimation can be sensitive to outliers (see [103] and references therein). Kernel PCA is automated as part of preprocessing in some AutoML packages, such as Auto-PyTorch (see Section 3).

PCA and kernel PCA are often viewed not just as feature engineering steps but also as dimension reduction techniques and derive their power essentially from scale imbalances in the distribution of points along suitable coordinates. These differences in scale imply a reasonable approximation of the data should exist in a lower-dimensional space. A different approach, based on the idea that the true dimension of the dataset is lower than the number of features, is given by locally linear embedding (LLE) [132, 135]. In LLE, data from an $n$-dimensional space is assumed to be drawn from a distribution of points on a $d<n$ dimensional manifold. Because smooth manifolds are locally flat (see [142]), the low-dimensional manifold hypothesis of LLE implies that a suitable linear representation exists of data points in terms of their near neighbours. Once this linear map is obtained, projections to lower dimensions follow by minimizing an objective that penalizes variation from the linear relationship in the high-dimensional space holding in the lower-dimensional space, enforcing as much of the local geometry to hold in the lower dimensional space as possible (see also [27] for another take on using LLE in feature selection). Both kernel PCA and LLE have straightforward implementation in sklearn.

Finally, Independent Component Analysis (ICA) is a technique for uncovering latent, non-Gaussian, independent, features from a dataset. Namely, observations are assumed to be generated by independent, non-Gaussian components, viz. $x^{\left(i\right)}=A{s}^{\left(i\right)}$ where $A$ is a mixing matrix and the entries of $s^{\left(i\right)}$ represent statistically independent latent components. The independent components can be approximately determined by a projection pursuit algorithm, FactICA (introduced in [1]), that finds directions to project $x^{\left(i\right)}$ onto that result in maximizing the kurtosis of the result⁸⁸8Recall that for a random variable $X$, the kurtosis can be defined by $E{X}^4-3(E{X}^2{)}^2$ and is zero for normal distributions (see [1]).. In [139], the authors rank-order the induced latent features obtained from ICA by their kurtosis to obtain a feature subset. As with other approaches in this section, this approach combines both generation and selection. ICA is built into the automated preprocessing portion of the auto-sklearn package.

In the below, we outline a few methods for filtering that use information explicitly, or use correlation as a proxy.

A somewhat related method, not reliant upon statistical testing, is to rank features by their correlation (or covariance) with the target variable. In the case of a univariate linear least squares regression model this gives the square root of the coefficient of determination, namely the fraction of the variance in the response explained by the variance in the given feature. Extensions of this approach, and its relation to traditional statistical tests, is further discussed in [69].

Another simple example of this kind of filtering is the removal of all features whose correlation with other features exceeds a threshold. Linear correlations can easily be estimated from the values of the features and do not require direct distribution estimation. This property makes them a desirable proxy for information between random variables in some instances. A related issue is multicollinearity, wherein one feature may have linear dependence on possibly several others. The package PyCaret, for example, automates the handling of multicollinear features as part of the overall AutoML pipeline.

The article [59] introduces a hybrid approach whose filter stage uses a unique multi-objective optimization to prune features. For a given feature subset $\tilde{F}\subset F$, three objectives are calculated: the (negative of) average mutual information between features in $\tilde{F}$ and target $Y$, the average mutual information between features in $\tilde{F}$, and $\#\tilde{F}$. The Pareto frontier⁹⁹9Due to the classic analogy of optimizing for location and price of coastal hotels, this frontier is commonly called a skyline (see, e.g., the seminal article [23]). for a multi-objective optimization problem is a set of Pareto efficient (i.e. non-dominated¹⁰¹⁰10In this context, these are solutions for which no subset of features performs at least as well on 2 objectives and better on the remaining one. ) solutions. The authors use a forward selection strategy based on finding an approximation of the Pareto frontier by growing the size of examined subsets while finding subsets that are non-dominated according to the first two (information-based) objectives. To the authors knowledge, Pareto optimization is not performed in any open source automated tooling, however the DEAP package includes implementations of popular multi-objective optimizations such as NSGA-II (see [38]).

In the doctoral thesis [61], Mark Hall outlines a method for feature selection, correlation-based feature selection (CFS), for classification problems, based on the following idea:

“A good feature subset is one that contains features highly correlated with (predictive of) the class, yet uncorrelated with (not predictive of) each other”

Stated this way, CFS aims to find a trade-off between relevancy (dependence of the target on the response, as above) and redundancy (correlations among the derived features)¹¹¹¹11The article [69] provides several helpful cautionary tales about naive application of ranking schemes. Among other things, they show that identically distributed variates may not be immediately redundant, irrelevant features may combine to give relevance as a group, and high correlation does not imply uselessness in separating classes.¹²¹²12As discussed in [76], some machine learning algorithms have highly robust performance in the face of irrelevant variables but degrade when given highly correlated (redundant) relevant features.. Features are retained based on their high relevance, low redundancy, and ability to predict classes in areas of the features space that are not predicted well by existing features. For a classification prediction on targets $C$, a proposed subset, $\tilde{F}\subseteq F$ is scored according to

where ${\rho }_{fg}$ is the correlation coefficient between features $f,$ and $g$, and ${\rho }_{cf}$ is the correlation coefficient¹³¹³13These coefficients are not the usual linear Pearson coefficients, but represent a more general sense of the word “correlation”. In Hall’s presentation the coefficients can be symmetrical versions of relief, minimum description length, or symmetric uncertainty. See [61] for full details. between feature $f$ and target values $c$. The numerator in (1) indicates how predictive the features are, while the denominator penalizes non-parsimony and redundancy. The pyautoweka package, which implements the Java Auto-WEKA application [124, 148] (described in more detail below) performs CFS in addition to informational and entropy-based filtering as part of its general automated feature and model selection algorithm.

A related technique to the informational approaches above are statistical filtering techniques. The popular scikit-learn package (see [117, 137]) offers a host of basic statistical feature selection techniques through its feature_selection module [117]. These include methods such as SelectKBest which uses scores based on $p$-values of passed hypothesis tests of the features, VarianceThreshold, which makes sure that variance in a feature exceeds a minimal limit and removes those features below, as well as others. Some of these methods are automated in the OptimalFlow, EvalML and TPOT (see next section) feature selection pipeline stage., while some implement their own takes on these patterns (such as MLBox’s automated implementation of variance thresholding).

A very good example of automated statistical filtering is outlined in [30], as implemented in tsfresh, wherein for each derived feature, a $p$-value is computed to determine whether the derived feature is relevant for the prediction of the target. Here, irrelevance is equivalent to zero mutual information between the given feature and the target. Each univariate feature has its $p$-value chosen (using potentially different hypothesis tests) resulting in a vector of $p$-values. From here, a Benjamini-Yekutieli procedure is applied to avoid false positives (meaningless features making it through this round of filtering) among these $p$-values. This procedure is based on a method for multiple comparison testing that controls the so called false discovery rate, $FDR$, defined by

Feature reduction then takes place based on the $p$-values while keeping bounds on the $FDR$. Of course, other techniques, like the simple Bonferroni correction or other multiple comparison solutions may be adopted in place of $FDR$-controlled methods. In addition to the above, there can be overlap with the information pruning techniques mentioned previously when including the target variable in the information pruning basket.

We close this discussion of filtering by noting that feature relevance is not at all an easy determination as there are many different, incompatible, definitions of relevance in the literature. For examples of these definitions, see Kohavi and John’s article [76] from which we quote:

‘‘...it is important to realize that relevance according to these definitions does not imply membership in the optimal feature subset, and that irrelevance does not imply that a feature cannot be in the optimal feature subset.”¹⁴¹⁴14Optimality is given a precise meaning in their paper.

The term feature importance can refer to a handful of different techniques. A simple case is the case of linear models, where model coefficients can be viewed as feature importances when the features are appropriately normalized. In the classic article [24], permutation feature importances are explored in the context of random forests. This amounts to looking at the resultant decrease in performance (misclassification rate, say) when each feature undergoes a random permutation of its values, keep all other features fixed. EvalML (see next section) and scikit-learn (which can be used with non-native variants such as XGBoost) include this form of importance as part of their basic suites.

Sticking with forests, the determination of splits, the Gini index¹⁵¹⁵15If ${\pi }_i\left(t\right)$ is the estimated probability of class $i$ in node $t$, ${\sum }_{i\le \#classes}{\pi }_i\left(t\right)(1-{\pi }_i(t\left)\right)$ denotes the Gini index for $t$, see [73]. (or impurity) is often used for classification problems and sum of squares for regression (see, e.g. [144] for more details). XGBoost offers multiple additional options for measuring feature importances, such as the number of times a given feature was used to provide a splitting of the data across all trees in the forest as well as the gain from each feature used in model construction/pruning, among others (see [40] for full details). Fully automated random forest impurity based feature importance is implemented by MLBox, EvalML, and other autoML packages.

Building on work in [145] for the case of filters, in [98] a wrapper algorithm and R package, Boruta, are presented which are based around random forest feature importances and meant to find all relevant features for a given classification problem. Each feature in the dataset is used to create a “shadow” feature whose values are permutations of the base feature. Then, the enlarged dataset with the original and shadow features is used to perform a prediction and feature importances are computed. The maximum $Z$ score is calculated among all shadow attributes (MZSA)¹⁶¹⁶16The $Z$ score of a variable $X$ is $\frac{X-EX}{\sqrt{var\left(X\right)}}$. In this case, the importances are random variables. and those features with $Z$ scores testing significantly less than the MZSA are removed, while those testing significantly above are counted as important. This process repeats until a stopping criteria is met. Boruta has an open source Python implementation (see [22]) as well as a Python extension BoostARoota [21] which uses a variant of the Boruta algorithm with XGBoost for increased speed relative to the traditional random forest based Boruta. Boruta is part of PyCaret’s automated feature selection module.

Shapley additive explanations (SHAP, see [92]) can provide an alternative feature importance paradigm. Traditional Shapley values can be defined by

where $f_i$ is the $i$’th feature, $c\left(S\right)=\frac{\#S!(\#F-\#S-1)!}{\#F!}$, $m_B$ denotes the predictive model obtained by training on the subset $B\subset F$ features and $x_B$ the corresponding inputs from the feature subset, $B$. Morally speaking, the SHAP values are a variation of (2) wherein $m_B\left(x_B\right)$ is approximated by a conditional expectation $E\left[m\right(z)\mid z_B]$ where $z_B$ is a binary vector indicating which features are selected, and $m$ is the full predictive model. As the sum in (2) extends over subsets of a feature set, sampling can be performed to give a tractable approximate solution, among other approaches (see [92] and [8] for more details). SHAP values are implemented in, e.g., EvalML as part of model explainability.

A genetic algorithm based wrapper technique is to use a performance based fitness function during the evolutionary cycle. A clear example of this general idea in the context of regression problems with categorical features, using the DEAP package, is found in [108]. Similar ideas were explored in [60].

In [84], an additional feature subset selector pipeline was added to the basic TPOT package (see Section 3.1.2 for more details on the TPOT package) that performs removal of input features as a pipeline module. The TPOT evolved model then can be used as a wrapper to select the best performing feature subset. Expanding on the TPOT framework, in the article [114] a method inspired from genetics is added, paring down the full feature set to a restricted number of parameterized feature pairs. Each pair is then assessed using its accuracy on a custom tree built just on those two features. This method presumes the possible strong interaction between two features as can happen between two genes.

The least absolute shrinkage and selection operator (LASSO) [88, 149] is an ${\ell }^1$-regularized variant on the linear least squares regression problem, that tries to solve

The first term in the above is a fidelity term penalizing poorness of fit to a linear mode, while the second term helps promote sparsity of the resultant $\theta$; the gradient of (3) will produce terms like $\frac{\partial }{\partial {\theta }_i}|{\theta }_i|=\frac{|{\theta }_i|}{{\theta }_i}=sgn\left({\theta }_i\right)$ when ${\theta }_i\ne 0$ so that gradient descent will result in components of $\theta$ not improving fidelity tending to zero. LASSO on its own, as a predictive model, is an embedded technique rather than a wrapper, but it can be repurposed as part of a larger wrapper method.

The autofeat Python package [65] offers a multi-step feature reduction technique, consisting of removal of highly correlated features, followed by division of data into chunks on which ${\ell }^1$-regularized models are trained to guide feature selection. This process repeats through several rounds of subsampling. In addition, the package implements a form of symbolic pruning, utilizing the SymPy library to simplify and reduce redundant symbolic expressions. This eliminates redundant but complex features without evaluating their associated expressions. The popular autoML package MLBox also offers automated LASSO feature selection as part of its preprocessing pipeline optimization.

A distinct kind of wrapper technique is one based on stability, which can be interpreted in a number of different ways¹⁷¹⁷17In the literature, stability generally refers to what we call here dataset stability. We use the term stability in a looser, more general, way.. In [59], the authors use an approach based on classification stability: for each subset $\tilde{F}\subset F$ of the feature set, two scores is computed based on classification rates for a handful of classifiers. The first is $A\left(\tilde{F}\right)={max}_{f\in M}R(f,\tilde{F})-{min}_{f\in M}R(f,\tilde{F})$, where $M$ is a set of classifiers and $R(f,\tilde{F}))$ represents the rate of correct classification with model $f$ using feature subset $\tilde{F}$, and measures the overall worst variation in performance with a given subset. The other objective is simply the average $R(f,\tilde{F})$ over the classifier set. Subsets can be retained based on minimization of the former and maximization of the latter objectives. This multi-objective optimization can help ensure both reliability and performance of the final feature subset on a wide range of models.

In [91], a modification to forward selection is proposed, which we will refer to as dataset stability. The algorithm rests on the fact that error rates are random variables, as they depend on the data-generating distribution and therefore are influenced by sampling. Since wrappers generally assess feature subsets on the basis of these random variables (error rates), the sequence of feature sets obtained during a single run of forward selection, is also a random variable, and ultimately dependent upon the training set used in the selection process. The proposed algorithm works by performing $K$ runs of forward selection, each producing a sequence of obtained feature sets,

Evaluation of runs involves a new training/testing split of the dataset, and therefore the sequences $S_1,...,S_K$ generally will have feature variety. The set of feature sequences, $S=\{S_1,....,S_K\}$ can then be given an index,

where, in the above, $S_i\left(k\right)$ denotes the first $k$ features selected by the $i$’th step of forward selection and $I_C\left(S_i\right(k),S_j(k\left)\right)$ is a measure of consistency between two feature subsets¹⁸¹⁸18Specifically, we have for equal cardinality feature subsets ${\tilde{F}}_1,{\tilde{F}}_2$ . With a stability index for the set of feature sequences, a final reduced feature subset can then be obtained by selecting an optimal subsequence and truncating its size based on the lowest cost (see [120] for full details). The main advantage of this approach is that the returned feature subset should be fairly robust with respect to data samples. A fuller discussion of stability, with differing notions of measuring it, is found in [120], to which we refer the interested reader. We remark that, to the author’s knowledge, no current open source automated feature engineering tools implement the stability-based approaches discussed here. We include them as they may make a promising addition to future systems.

In [148], the authors introduce a combined algorithm selection and hyperparameter optimization (CASH) scheme to find

The authors’ implement a Sequential Model-Based Optimization (SMBO, see [138] for open source solutions) algorithm called Sequential Model-Based Algorithm Configuration (SMAC, see Appendix A.4 for more details) to solve (4). SMBO is a general Bayesian technique for solving algorithm configuration problems (see [46]), attempting to maximize the expected loss improvement at a given hyperparameter choice with respect to a model-specific conditional probability. Their implementation of this resulted in the auto-WEKA Java application which has Python implementations (see, e.g. [124]). Since the original release of auto-WEKA, auto-WEKA 2.0 (see [82]) introduced several improvements on the original package, including support for regression models and a higher level of parallelism. In Table 2 we refer to the various iterations of auto-WEKA, as the auto-WEKA family .

The auto-sklearn package [50, 49] builds on the CASH formulation of auto-WEKA but incorporates two enhancements: meta-learning and ensembling. The meta-learning entailed first establishing meta-features and model performance on 140 datasets in the repository OpenML [151], and using this information to assign a new dataset $D$ a similarity score to the known datasets, and using the associated model metadata for top performing models on the nearest 25 datasets to seed the Bayesian optimization portion of the CASH solver. Evaluated meta-features are numerous and include things like numbers of instances/classes, ratio of categorical to numeric columns, some simple descriptive statistics, prediction performance with simple models, and others (see [11]).

The ensembling portion of the auto-sklearn algorithm keeps track of models during the Bayesian CASH optimization and then applies ensemble selection as described above. The auto-sklearn package is capable of performing end-to-end modelling, including various preprocessing (imputation, balancing, etc) routines.

In the article [49] the authors extend the original auto-sklearn package in a few ways. On the prosaic end, they expand on datasets used for the meta-learning portion. On the more challenging side, they consider a time-constrained version of the cash optimization (4). The main novelty used to satisfy the given time-constraint is to no longer use the meta-datasets to make a $K$-nearest dataset (KND) score on each new dataset given at runtime, but to use meta-datasets to construct a pipeline portfolio of pipeline configurations (hierarchical hyperparameters) that perform well in general, in terms of generalization error on validation subsets of the meta-dataset. The portfolio obtained serves as a robust warmstart set for the Bayesian optimization used for solving CASH on a new dataset. The authors introduced a successive halving technique for reducing the portfolio optimization time: train pipelines for a low given budget, select the best performing pipelines, increase their time budget in proportion to the number selected, and repeat. This technique offers significant speedup on the time-constrained portfolio optimization. The described variant of auto-sklearn is called PoSH Auto-sklearn. We refer to auto-sklearn and its various variants as the auto-sklearn family in Table 2.

In [154] the authors introduce the FLAML (Fast and Lightweight AutoML) library (see [52, 51]) which implements a few novelties in the algorithm selection and hyperparameter optimization space. In FLAML a learning configuration is specified by a tuple $(m_j^{\lambda },\lambda ,r,s)$ where, as earlier, $m_j^{\lambda }$ is a machine learning model as usual, $\lambda \in {\Lambda }^{\left(j\right)}$, the associated hyperparameters for that model, $r$ is a binary variable indicating whether to use cross-validation or a holdout set (effectively determining how much of the training data is used in assessing performance) known as resampling, and $s$ is the sample size to use during training. The main novelty here is the decoupling of the sample size, resampling, hyperparameters and choice of model from each other allowing the potential exploitation of several heuristics¹⁹¹⁹19Some of these heuristics are the relationships between the resampling strategy and computation time, the relationships between smaller samples and need for regularization, etc. See [154] for the full discussion of observations and properties..

The method works according to the following. First a resampling strategy is decided based upon the size of the training data, with cross-validation preferred for smaller sized datasets. After this, a machine learning model $m_j^{\lambda }$ is selected. This selection is probabilistic with probability density determined based on the expected cost of improvement, a measure of the expected computation time to find a configuration of the given model with a lower validation error than the current lowest validation error among all models used thus far. Models with a lower expected cost of improvement are expected to spend less time improving the overall performance and therefore are given a higher probability of being proposed. After the model is proposed, hyperparameters are selected using an algorithm described in [158] which uses a randomized stepping updates to $\lambda$ based on observed validation error. For a given model $m_j^{\lambda }$, starting with an initially small sample size, decisions are made about whether to increase the size with given choice of hyperparameters, or move to new hyperparameters with the current sample size.

Although FLAML does not offer built-in ensembling as many of its competitors do, it manages to achieve solid performance. Additionally it is highly extensible, offering users the ability to easily add in models, requiring very simple compatibility requirements.

In [18] the authors introduce the hyperopt package [66] for optimization in a variety of algorithm configuration problems. Building on this framework, the hyperopt-sklearn package [67] uses hyperopt to performing pipeline selection from preprocessing and model modules implemented in scikit-learn. The hyperopt-sklearn package offers users a high level of control over the method used for hyperparameter optimization, supporting a number of popular optimization algorithms such as random search (see, e.g., [20]) and annealing. The package is unique in allowing users a bit more control over the probabilistic aspects of the final estimators (probability of certain models being used, as well as distributions of hyperparameters).

The MLBox package [101, 100] provides a fully automated, low-code wrapper for hyperopt and scikit-learn that includes multiple forms of feature selection (feature importances, LASSO, and others) as well as performing basic ensemble stacking.

The PyCaret package (see [125]) provides a very low-code, automated wrapper for scikit-learn, hyperopt, and others. Heavily targeted at business case focussed data scientists, it offers some unique additions over competing packages, such as dedicated anomaly detection, classification, and natural language processing modules, robust preprocessing, as well as very good plotting and visualization tools with very minimal human input.

In [85] the OptimalFlow automated pipeline ensembling package is introduced (see also [115]). Unlike some of the other packages discussed so far, OptimalFlow used random search ([20]) or grid search rather than Bayesian optimization to solve the CASH problem. OptimalFlow constructs a collection of pipelines to optimize, each with modularizable components, and then searches through the collection to obtain an optimal baseline in a technique the authors call Pipeline Cluster Traversal Experiments.