Alternative performance measures for prediction models
Journal
PLoS ONE
Journal Volume
9
Journal Issue
3
Date Issued
2014
Author(s)
Wu Y.-C.
Abstract
As a performance measure for a prediction model, the area under the receiver operating characteristic curve (AUC) is insensitive to the addition of strong markers. A number of measures sensitive to performance change have recently been proposed; however, these relative-performance measures may lead to self-contradictory conclusions. This paper examines alternative performance measures for prediction models: the Lorenz curve-based Gini and Pietra indices, and a standardized version of the Brier score, the scaled Brier. Computer simulations are performed in order to study the sensitivity of these measures to performance change when a new marker is added to a baseline model. When the discrimination power of the added marker is concentrated in the gray zone of the baseline model, the AUC and the Gini show minimal performance improvements. The Pietra and the scaled Brier show more significant improvements in the same situation, comparatively. The Pietra and the scaled Brier indices are therefore recommended for prediction model performance measurement, in light of their ease of interpretation, clinical relevance and sensitivity to gray-zone resolving markers. ? 2014 Wu, Lee.
SDGs
Other Subjects
article; Brier score; computer simulation; controlled study; Lorenz curve based Gini and Pietra indices; mathematical model; named inventories, questionnaires and rating scales; performance measurement system; prediction; prediction model; receiver operating characteristic; scaled Brier; sensitivity and specificity; area under the curve; diseases; human; metabolism; risk; risk assessment; statistical model; biological marker; Area Under Curve; Biological Markers; Disease; Humans; Models, Statistical; Odds Ratio; Risk Assessment
Publisher
Public Library of Science
Type
journal article