The validation and assessment of machine learning: a game of prediction from high-dimensional data

Research output: Contribution to journal › Journal article › Research › peer-review

Standard

The validation and assessment of machine learning: a game of prediction from high-dimensional data. / Pers, Tune H; Albrechtsen, Anders; Holst, Claus; Sørensen, Thorkild I A; Gerds, Thomas A.

In: PLoS ONE, Vol. 4, No. 8, 2009, p. e6287.

Research output: Contribution to journal › Journal article › Research › peer-review

Harvard

Pers, TH, Albrechtsen, A, Holst, C, Sørensen, TIA & Gerds, TA 2009, 'The validation and assessment of machine learning: a game of prediction from high-dimensional data', PLoS ONE, vol. 4, no. 8, pp. e6287. https://doi.org/10.1371/journal.pone.0006287

APA

Pers, T. H., Albrechtsen, A., Holst, C., Sørensen, T. I. A., & Gerds, T. A. (2009). The validation and assessment of machine learning: a game of prediction from high-dimensional data. PLoS ONE, 4(8), e6287. https://doi.org/10.1371/journal.pone.0006287

Vancouver

Pers TH, Albrechtsen A, Holst C, Sørensen TIA, Gerds TA. The validation and assessment of machine learning: a game of prediction from high-dimensional data. PLoS ONE. 2009;4(8):e6287. https://doi.org/10.1371/journal.pone.0006287

Author

Pers, Tune H ; Albrechtsen, Anders ; Holst, Claus ; Sørensen, Thorkild I A ; Gerds, Thomas A. / The validation and assessment of machine learning: a game of prediction from high-dimensional data. In: PLoS ONE. 2009 ; Vol. 4, No. 8. pp. e6287.

Bibtex

@article{936358407e0211df928f000ea68e967b,
title = "The validation and assessment of machine learning: a game of prediction from high-dimensional data",
abstract = "In applied statistics, tools from machine learning are popular for analyzing complex and high-dimensional data. However, few theoretical results are available that could guide to the appropriate machine learning tool in a new application. Initial development of an overall strategy thus often implies that multiple methods are tested and compared on the same set of data. This is particularly difficult in situations that are prone to over-fitting where the number of subjects is low compared to the number of potential predictors. The article presents a game which provides some grounds for conducting a fair model comparison. Each player selects a modeling strategy for predicting individual response from potential predictors. A strictly proper scoring rule, bootstrap cross-validation, and a set of rules are used to make the results obtained with different strategies comparable. To illustrate the ideas, the game is applied to data from the Nugenob Study where the aim is to predict the fat oxidation capacity based on conventional factors and high-dimensional metabolomics data. Three players have chosen to use support vector machines, LASSO, and random forests, respectively.",
author = "Pers, {Tune H} and Anders Albrechtsen and Claus Holst and S{\o}rensen, {Thorkild I A} and Gerds, {Thomas A}",
note = "Keywords: Computers; Humans; Learning; Models, Theoretical",
year = "2009",
doi = "10.1371/journal.pone.0006287",
language = "English",
volume = "4",
pages = "e6287",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "8",

}

RIS

TY - JOUR

T1 - The validation and assessment of machine learning: a game of prediction from high-dimensional data

AU - Pers, Tune H

AU - Albrechtsen, Anders

AU - Holst, Claus

AU - Sørensen, Thorkild I A

AU - Gerds, Thomas A

N1 - Keywords: Computers; Humans; Learning; Models, Theoretical

PY - 2009

Y1 - 2009

N2 - In applied statistics, tools from machine learning are popular for analyzing complex and high-dimensional data. However, few theoretical results are available that could guide to the appropriate machine learning tool in a new application. Initial development of an overall strategy thus often implies that multiple methods are tested and compared on the same set of data. This is particularly difficult in situations that are prone to over-fitting where the number of subjects is low compared to the number of potential predictors. The article presents a game which provides some grounds for conducting a fair model comparison. Each player selects a modeling strategy for predicting individual response from potential predictors. A strictly proper scoring rule, bootstrap cross-validation, and a set of rules are used to make the results obtained with different strategies comparable. To illustrate the ideas, the game is applied to data from the Nugenob Study where the aim is to predict the fat oxidation capacity based on conventional factors and high-dimensional metabolomics data. Three players have chosen to use support vector machines, LASSO, and random forests, respectively.

AB - In applied statistics, tools from machine learning are popular for analyzing complex and high-dimensional data. However, few theoretical results are available that could guide to the appropriate machine learning tool in a new application. Initial development of an overall strategy thus often implies that multiple methods are tested and compared on the same set of data. This is particularly difficult in situations that are prone to over-fitting where the number of subjects is low compared to the number of potential predictors. The article presents a game which provides some grounds for conducting a fair model comparison. Each player selects a modeling strategy for predicting individual response from potential predictors. A strictly proper scoring rule, bootstrap cross-validation, and a set of rules are used to make the results obtained with different strategies comparable. To illustrate the ideas, the game is applied to data from the Nugenob Study where the aim is to predict the fat oxidation capacity based on conventional factors and high-dimensional metabolomics data. Three players have chosen to use support vector machines, LASSO, and random forests, respectively.

U2 - 10.1371/journal.pone.0006287

DO - 10.1371/journal.pone.0006287

M3 - Journal article

C2 - 19652722

VL - 4

SP - e6287

JO - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 8

ER -

ID: 20421184