schliessen

Filtern

 

Bibliotheken

Measuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)

The biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from st... Full description

Journal Title: 2014 Vol.9(10), p.e110840
Main Author: Ma, Shuyi
Other Authors: Sung, Jaeyun , Magis, Andrew T , Wang, Yuliang , Geman, Donald , Price, Nathan D
Format: Electronic Article Electronic Article
Language: English
Subjects:
ID: E-ISSN: 1932-6203 ; DOI: 10.1371/journal.pone.0110840
Zum Text:
SendSend as email Add to Book BagAdd to Book Bag
Staff View
recordid: plos10.1371/journal.pone.0110840
title: Measuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)
format: Article
creator:
  • Ma, Shuyi
  • Sung, Jaeyun
  • Magis, Andrew T
  • Wang, Yuliang
  • Geman, Donald
  • Price, Nathan D
subjects:
  • Research Article
  • Biology And Life Sciences
  • Computer And Information Sciences
  • Medicine And Health Sciences
  • Physical Sciences
  • Research And Analysis Methods
ispartof: 2014, Vol.9(10), p.e110840
description: The biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes (e.g., laboratory settings resulting in “batch-effects”) and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies. ; Here we quantify the impact of these combined “study-effects” on a disease signature’s predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance. ; As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification. ; We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when “sufficient” diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings.
language: eng
source:
identifier: E-ISSN: 1932-6203 ; DOI: 10.1371/journal.pone.0110840
fulltext: fulltext
issn:
  • 1932-6203
  • 19326203
url: Link


@attributes
ID425580524
RANK0.07
NO1
SEARCH_ENGINEprimo_central_multiple_fe
SEARCH_ENGINE_TYPEPrimo Central Search Engine
LOCALfalse
PrimoNMBib
record
control
sourcerecordid10.1371/journal.pone.0110840
sourceidplos
recordidTN_plos10.1371/journal.pone.0110840
sourcesystemPC
pqid1613423658
galeid418424518
display
typearticle
titleMeasuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)
creatorMa, Shuyi ; Sung, Jaeyun ; Magis, Andrew T ; Wang, Yuliang ; Geman, Donald ; Price, Nathan D
contributorHsiao, Chuhsing Kate (Editor)
ispartof2014, Vol.9(10), p.e110840
identifierE-ISSN: 1932-6203 ; DOI: 10.1371/journal.pone.0110840
subjectResearch Article ; Biology And Life Sciences ; Computer And Information Sciences ; Medicine And Health Sciences ; Physical Sciences ; Research And Analysis Methods
descriptionThe biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes (e.g., laboratory settings resulting in “batch-effects”) and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies. ; Here we quantify the impact of these combined “study-effects” on a disease signature’s predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance. ; As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification. ; We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when “sufficient” diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings.
languageeng
source
version8
lds50peer_reviewed
links
openurl$$Topenurl_article
openurlfulltext$$Topenurlfull_article
search
creatorcontrib
0Ma, Shuyi
1Sung, Jaeyun
2Magis, Andrew T
3Wang, Yuliang
4Geman, Donald
5Price, Nathan D
6Hsiao, Chuhsing Kate (Editor)
titleMeasuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)
descriptionThe biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes (e.g., laboratory settings resulting in “batch-effects”) and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies. ; Here we quantify the impact of these combined “study-effects” on a disease signature’s predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance. ; As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification. ; We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when “sufficient” diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings.
subject
0Research Article
1Biology And Life Sciences
2Computer And Information Sciences
3Medicine And Health Sciences
4Physical Sciences
5Research And Analysis Methods
general
010.1371/journal.pone.0110840
1English
sourceidplos
recordidplos10.1371/journal.pone.0110840
issn
01932-6203
119326203
rsrctypearticle
creationdate2014
recordtypearticle
addtitleEffect of Inter-Study Variability on Classification
searchscopeplos
scopeplos
lsr30VSR-Enriched:[galeid, pqid]
sort
titleMeasuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)
authorMa, Shuyi ; Sung, Jaeyun ; Magis, Andrew T ; Wang, Yuliang ; Geman, Donald ; Price, Nathan D
creationdate20141017
facets
frbrgroupid4945651712345826726
frbrtype5
languageeng
creationdate2014
topic
0Research Article
1Biology And Life Sciences
2Computer And Information Sciences
3Medicine And Health Sciences
4Physical Sciences
5Research And Analysis Methods
collectionPLoS
prefilterarticles
rsrctypearticles
creatorcontrib
0Ma, Shuyi
1Sung, Jaeyun
2Magis, Andrew T
3Wang, Yuliang
4Geman, Donald
5Price, Nathan D
6Hsiao, Chuhsing Kate
toplevelpeer_reviewed
delivery
delcategoryRemote Search Resource
fulltextfulltext
addata
aulast
0Ma
1Sung
2Magis
3Wang
4Geman
5Price
6Hsiao
aufirst
0Shuyi
1Jaeyun
2Andrew T.
3Yuliang
4Donald
5Nathan D.
6Chuhsing Kate
au
0Ma, Shuyi
1Sung, Jaeyun
2Magis, Andrew T
3Wang, Yuliang
4Geman, Donald
5Price, Nathan D
addauHsiao, Chuhsing Kate
atitleMeasuring the Effect of Inter-Study Variability on Estimating Prediction Error (Effect of Inter-Study Variability on Classification)
risdate20141017
volume9
issue10
spagee110840
pagese110840
eissn1932-6203
genrearticle
ristypeJOUR
abstractThe biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes (e.g., laboratory settings resulting in “batch-effects”) and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies. ; Here we quantify the impact of these combined “study-effects” on a disease signature’s predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance. ; As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification. ; We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when “sufficient” diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings.
copSan Francisco, USA
pubPublic Library of Science
doi10.1371/journal.pone.0110840
oafree_for_read
date2014-10-17