CF-DAML: Distributed automated machine learning based on collaborative filtering

Abstract

The search for a good machine learning (ML) model takes a long time and requires the considerations of many alternatives, including data preprocessing, algorithm selection, and hyperparameter tuning methods. Thus, tedious searches face a combinatorial explosion problem. In this work, we build a new automated machine learning (AutoML) system called CF-DAML, a distributed automated system based on collaborative filtering (CF), to address these challenges by recommending and training suitable models for supervised learning tasks. CF-DAML first computes some informative meta-features for a new dataset, then uses a weighted $l_1$-norm (W1-norm) to accurately calculate the k nearest neighbors (kNN) of the new dataset, and finally recommends the top N models with good performances on each of its neighbors to the new dataset. We also design a distributed system (DSTM) for training the models to reduce the time complexity substantially. In addition, we develop a multilayer selective stacked ensemble system (MSSE), whose base models are selected from among suitable candidate models based on their runtimes, classification accuracies, and diversities, to enhance the stability of CF-DAML. To our knowledge, this is the first work to combine memory-based CF and the selective stacked ensemble to solve the AutoML problem. Extensive experiments are conducted on many UCI datasets and the comparative results demonstrate that our approach outperforms the current state-of-the-art methods.

Appendices

Appendix A: The models and their hyperparameters in the DataBase

Each number in brackets in the first column indicates times that the associated model appears in the DataBase. The second column contains hyperparameters of the models, where ${\{a, b,...\}}$ indicates discrete values; [a, b] represents values of a, a + 1, a + 2, ..., b; U(a, b) represents a uniform distribution in the region (a, b); $e^{\lambda }, (a, b)$ represent a value distribution obeying an exponential distribution with the parameter $\lambda$ and the value ranging from a to b. Preprocessors and their hyperparameter configurations are shown in Table 8, classifiers and their hyperparameter configurations are shown in Table 9.

Table 8 The preprocessors and their hyperparameter configurations

Table 9 The classifiers and their hyperparameter configurations

Appendix B: The distance correlation coefficients between selected meta-features

Here, we present the Dcc values for DatasetRatio (DR), InverseDatasetRatio (IDR), LogDatasetRatio (LDR), and LogInverseDatasetRatio (LIDR), in Table 10; for LogNumberOfFeatures (LOF) and NumberOfFeatures (NOF), in Table 11; for LogNumberOfInstances (LONI) and NumberOfInstances (NOI), in Table 12; for RatioNominalToNumerical (RNoTNu) and RatioNumericalToNominal (RNuTNo), in Table 13; for NumberOfClasses (NOC) and ClassEntropy (CE), in Table 14; for NumberOfCategoricalFeatures (NOCF) and SymbolsSum (SS), in Table 15; for NumberOfNumericFeatures (NuONuF) and LogNumberOfFeatures (LNuOF), in Table 16; and for PCAKurtosisFirstPC (PCAKFPC) and KurtosisMin (KM), in Table 17.

Table 10 Dcc values for {DR, IDR, LDR, LIDR}

Table 11 Dcc values for {LOF, NOF}

Table 12 Dcc values for {LNOI, NOI}

Table 13 Dcc values for {RNoTNu, RNuTNo}

Table 14 Dcc values for {NOC, CE}

Table 15 Dcc values for {NOCF, SS}

Table 16 Dcc values for {NuONuF, LNuOF}

Table 17 Dcc values for {PCAKFPC, KM}

The physical meanings of DR, LDR, IDR, and LIDR involve comparisons between the numbers of rows and columns in the datasets. The correlation coefficients between LDR and the other three meta-features are very large; therefore, LDR is chosen, and the other three meta-features are discarded (Tables 18 and 19).

Appendix C: Detailed attributes of the experimental datasets

Table 18 The detailed attributes of the small datasets

Table 19 The detailed attributes of the medium and large datasets