Treffer: Multi‐aspect permutation tests for model selection.

In typical machine learning frameworks, model selection is of fundamental importance: commonly, multiple models have to be trained and compared in order to identify the one with the best predictive performances. The aim of this study is to provide a new tool to improve the model selection process, allowing the user to identify the algorithm which significantly outperforms the other candidates. It proposes a robust model selection procedure based on a multi‐aspect permutation test which makes it possible to detect differences in both location and variability between two paired samples of prediction errors. A new extension of the nonparametric combination (NPC) methodology is therefore introduced and is integrated with an appropriate ranking procedure in order to deal with the comparison of C≥2 candidate models. A simulation study is conducted to evaluate the performances of this testing procedure in 2‐sample and C‐sample problems, by generating data from various well‐known distributions and simulating several possible null and alternative scenarios. The adoption of the proposed technique in machine learning model selection problems is then discussed by means of multiple real data applications. These applications confirm what emerges from the simulation study: the introduced NPC‐based approach performs well under several different scenarios and represents a valuable tool for robust machine learning model selection. [ABSTRACT FROM AUTHOR]

Copyright of Expert Systems is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Volltext ist im Gastzugang nicht verfügbar. Login für vollen Zugriff.