An Empirical Study on the Usage of Automated Machine Learning Tools

Although AutoML tools help ML experts and non-expert practitioners save effort and time by automating the repetitive tasks (truong2019:towards), choosing the right AutoML tool is still a challenging task (xin2021:whither; passi2018trust; wang2019:human; drozdal2020:trust). The goal of RQ1 is to highlight the most used AutoML tools and provide detailed information about the projects using these tools. This information can help ML practitioners select appropriate AutoML tools for their projects. Besides, the insights gained from the projects using the tools can help AutoML tool’s developers improve their tools in future releases.

To find the most used AutoML tools, we collected the AutoML function calls in the GitHub projects. Also, we analyzed the characteristics of the GitHub projects that are using AutoML tools.

Analyzing the characteristics of GitHub projects using AutoML tools: To understand the characteristics of the projects that use AutoML tools, we analyzed the details of the projects that use AutoML tools, such as the number of contributors, number of forks, size, age, star, and commits.

Figure 3 shows the total number of function calls for each AutoML tool. As seen in Figure 3, the top 10 AutoML tools with the most number of function calls are specified in green color. The percentage of function calls that are associated with the top 10 most used tools (i.e., 18% of the 57 selected AutoML tools) is 68%; these top tools include Optuna, HyperOpt, Skopt, Featuretools, Tpot, Bayes_opt, Autokeras, Auto-sklearn, AX, and Snorkel. The details of each of these top 10 most used AutoML tools are provided in Table 1. As seen in Table  1, the most used AutoML tools are popular based on the number of forks and number of stars. Also, they are 67.1 months old in average, which means that they are not young AutoML tools and it took them years to become popular.

The 18% (10 out of 57) most used AutoML tools (e.g., Optuna) account for 68% of all function calls.

Furthermore, the profile of the projects that are using AutoML tools is shown in Figure 4. For clear visualization, we removed the outliers within the last 5 percent of data in each metric and showed the distribution of 95% percent of the data in Figure 4. As seen in Figure  4, 75% of the repositories that are using AutoML tools are less than five months old and have fewer than 65 commits, two contributors, two forks, 35,753 lines of code, two stars. On the other hand, the projects that use AutoML tools also include some projects that are very mature (up to 10 years in age), popular (with up to 102,983 stars), with many contributors (with up to 433 contributors), and large in size (with up to 17,453,577 lines of code).

The most used AutoML tools are popular tools according to their number of stars and forks. Furthermore, most of the projects that use AutoML tools are young and not popular (based on the number of stars).

ML practitioners may use AutoML tools for different purposes across their ML pipeline. We analyzed the stages of the ML pipeline in which AutoML tools are used and the purposes of using AutoML tools to help AutoML tools’ developers gain more insights on the usage of their tool and enable them to improve their tools in a more efficient way.

Understanding the stages of the ML pipeline in which the AutoML tools are most popular can help developers of the tools find the shortcomings of their tools and help improve their tools efficiently. Besides, our results can provide insights for ML practitioners to better leverage AutoML tools in different stages of their ML pipeline and for different purposes.

To understand the purposes of using AutoML tools and the stages of the ML pipeline where they are being used, we used the extracted function calls in Section  4.1.3 and labeled them. To reduce the required time and effort for manual labeling, we only focused on the function calls of the top 10 most used AutoML tools (identified in RQ1). Besides, we focused on the function calls that are most commonly used in different projects. Thus, from the set of all unique function calls of the selected AutoML tools, we compute their total frequency and Shannon entropy value, then choose the functions that have more than 80% of 1) the Shannon entropy value and 2) the total frequency for manual labeling. In the following, we explain 1) the process of selecting functions based on Shannon entropy and usage frequency and 2) the process of manual labeling.

In Equation (1), F is a function that we want to measure its Shannon’s entropy value, n is the total number of unique projects, i is the unique number of projects, $p_i$ is the probability of the use of function $F(p_i\ge 0$, and ${\sum }_{i=1}^{i=n}{p}_i=1)$ in a given project $i$. $p_i$ is calculated by dividing the total number of occurrence of function F in the project i by the total number of occurrence of function F in all the projects. If all the unique projects have the same probability of using function F (i.e., $p_i=\frac{1}{n},\forall i\in 1,2,..,n$), the Shannon entropy of F is maximal (i.e., $H_n\left(F\right)=1$). Thus, the function F with higher Shannon entropy is more likely to be used more by ML practitioners compared to the functions with less Shannon entropy.

The functions with $H_n\left(F\right)>0.8$ and with their respective counts of function calls above 80% percentage (sorted in descending order) were chosen for manual labeling. A total of 619 functions from the top 10 AutoML tools were selected in this step.

During the first iteration of labeling, the first author labeled the functions of Optuna, Skopt, Featuretools, and Ax, and the second author labeled the functions of the remaining six tools. In the Second iteration, all the initial labels were combined, and the first two authors underwent through each label together and subsequently further discussed the labels that were not agreed upon until a consensus was achieved. A random sample of 238 functions was selected with a confidence level of 95% and a confidence interval of 5% for further validation by the other two authors (not involved in the initial labeling). They agreed on 98% of the stages of the functions and 96% of the purposes of the functions.

Table 2 details the stage of the ML pipeline (i.e., the table columns) in which ML practitioners use AutoML tools. Each cell value in Table 2 represents the percentage of the total number of calls of the functions from the respective AutoML tool corresponding to the stages of the ML pipeline. As shown in Table 2, the AutoML tools are mostly used for hyperparameter optimization (accounting for 40.9% of the function calls). One of the possible reasons is that hyperparameter optimization involves multiple repetitive tasks and requires a lot of effort and time (truong2019:towards); similarly, there might be numerous different sub-tasks involved compared to other stages of the ML pipeline.

AutoML tools are mainly used ($40.9\%$) during the Hyperparameter Optimization stage of the ML pipeline, followed by Model training ($11.5\%$), Model evaluation ($11.2\%$), and Feature Engineering ($10.1\%$). AutoML tools mostly focus on providing functions for training ML models effectively, selecting a model, and evaluating the performance of the resulting model.

Figure 5 presents high-level (in the grey background) and low-level (in white background) categories of the purposes for which ML practitioners use AutoML tools. As shown in Figure 5, the high-level categories of purposes are in the order: Hyperparameter Optimization (30.6%), Model management (26.5%), Data management (15%), Visualization (7.4%), Logging (3.4%), Utility functions (3.2%), Feature Engineering (3.1%), Storage management (2.1%), Pipeline management (2%) and User Interface (0.5%). The following section describes some of the categories and sub-categories of purposes.

We used the label Others for the function calls which do not fall into any of the categories.

We derived 10 high-level categories of the purposes of using AutoML tools, including Hyperparameter optimization (30.6%), Model management (26.5%), Data management (15%), Visualization (7.4%), and Logging (3.4%). ML practitioners can save time and effort by using AutoML tools to automate their ML pipeline tasks (truong2019:towards) for similar purposes.

In Figure 6 we provide the breakdown of the major categories of purposes (x-axis) of using AutoML provided in the top 10 AutoML tools (y-axis). The circle size represents the composition of using the respective AutoML tools for that purpose, computed as the percentage of total function calls defining that purpose to the total number of function calls from the individual AutoML tool.

We can clearly see in Figure 6 that the functions for Hyperparameter optimization, Model management, and Data management are used the most for most of the top 10 AutoML tools. Specifically, Hyperparameter optimization and Model management dominate at least 40% of the AutoML tools. In contrast, some of the purposes of using AutoML tools are specific to certain tools. For example, we can observe that the function calls for Feature engineering ($45\%$) and Pipeline management ($22.7\%$) tasks are dominantly used from Featuretools and Tpot, respectively, while a minimal percentage ($1.3\%$) of function calls related to User Interface comes from Snorkel and Optuna.

From Table 2 we observe that at least two of the AutoML tools can be used to automate the functionality in each stage of the ML pipeline. This question aims to understand whether the different AutoML tools are used together in the same source code file. For instance, ML practitioners may use AutoSklearn for model training and tune the hyperparameters using the functionality provided by Optuna within the same code. Answering this question could help ML practitioners choose the AutoML tools used together for different purposes. Similarly, the information can help the developer of the AutoML tools improve the integration procedure of the AutoML tools used together, such as providing an API to facilitate the usage of the other AutoML tools while getting input or sending output.

We compared the composition of the source code files (of the projects using AutoML tool) where different AutoML tools are used together as follows. For every source code files $\{T{s}_1,T{s}_2,T{s}_3,...T{s}_n\}$ importing and using the functions from a given AutoML tool, we computed the percentage composition of the shared source files as:

where $T{s}_{i\cdot j}$ is the total number of source files using the functions of two AutoML tools $T_i$ and $T_j$ in the same file, and $T{s}_{iN}$ is the total number of files using at least one function from the AutoML tool $T_i$. We define the results of the equation as the correlation between two AutoML tools.

In addition, we also calculated the distribution of the projects that use the different number of AutoML tools to understand how these AutoML tools are used together in the same projects.

Table 3 shows the distribution of the projects and source codes that use a different number of AutoML tools. We focus on the top 10 AutoML tools and the projects that use these tools. We observe that the vast majority of the projects use only one AutoML tool. There are only fewer than 8% of the projects that use two or more different AutoML tools in the same project.

Figure 7 shows the correlation matrix of the percentage composition of shared source files using different top AutoML tools together, computed using Equation (2). We only reported the correlation of the top 10 most used AutoML tools for clear visualization.

From Figure 7 we observe that, across the top 10 AutoML tools, only a few of the source files ($<2\%$) uses the functions from more than one AutoML tool. This result implies that most of these tools are used independently. Among the most used AutoML tools, the few AutoML tools used together are Tpot with Hyperopt ($0.69\%$), Optuna with Hyperopt ($0.29\%$), Skopt with Hyperopt ($0.25\%$) and Bayes-Opt with Skopt ($0.20\%$). In Table 4, we provide examples of the functions used together in the same source code file. For instance, both Hyperopt and Optuna provide rich functions for optimizing hyperparameters according to Figure 6; however, optuna also offers other features that are useful during hyperparameter optimization, such as Visualization and Logging. This implies that ML practitioners using HyperOpt with Optuna together have more options for HPO while also utilizing the visualization and logging functions from Optuna. Similarly, Tpot provides few functions for optimizing hyperparameters but offers more functions such as pipeline management and model management; the ML practitioners can simultaneously automate these tasks when using Tpot with Hyperopt.

Less than 8% of projects use two or more different AutoML tools. Different AutoML tools are rarely used in the same source code files. Among the few AutoML tools used together, Hyperopt is used with other tools, such as Tpot, Skopt, and Optuna. ML practitioners can consider using the combination of functions from different AutoML tools together to help automate multiple tasks across the ML pipeline. Similarly, the AutoML developers can propose an efficient API to integrate the AutoML tools that are used together.