schliessen

Filtern

 

Bibliotheken

Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE... Full description

Journal Title: arXiv.org Jun 3, 2019
Main Author: Dhingra, Bhuwan
Other Authors: Faruqui, Manaal , Parikh, Ankur , Ming-Wei, Chang , Das, Dipanjan , Cohen, William
Format: Electronic Article Electronic Article
Language: English
Subjects:
Quelle: © ProQuest LLC All rights reserved
Zum Text:
SendSend as email Add to Book BagAdd to Book Bag
Staff View
recordid: proquest2235435305
title: Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
format: Article
creator:
  • Dhingra, Bhuwan
  • Faruqui, Manaal
  • Parikh, Ankur
  • Ming-Wei, Chang
  • Das, Dipanjan
  • Cohen, William
subjects:
  • Judgments
  • Texts
  • Information Retrieval
  • Computation and Language
ispartof: arXiv.org, Jun 3, 2019
description: Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using...
language: eng
source: © ProQuest LLC All rights reserved
identifier:
fulltext: fulltext_linktorsrc
url: Link


@attributes
ID1636873749
RANK0.07
NO1
SEARCH_ENGINEprimo_central_multiple_fe
SEARCH_ENGINE_TYPEPrimo Central Search Engine
LOCALfalse
PrimoNMBib
record
control
sourcerecordid2235435305
sourceidproquest
recordidTN_proquest2235435305
sourcesystemPC
pqid2235435305
display
typearticle
titleHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
creatorDhingra, Bhuwan ; Faruqui, Manaal ; Parikh, Ankur ; Ming-Wei, Chang ; Das, Dipanjan ; Cohen, William
contributorCohen, William (pacrepositoryorg)
ispartofarXiv.org, Jun 3, 2019
subjectJudgments ; Texts ; Information Retrieval ; Computation and Language
descriptionAutomatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using...
languageeng
source
0© ProQuest LLC All rights reserved
1Engineering Database
2Publicly Available Content Database
3ProQuest Engineering Collection
4ProQuest Technology Collection
5ProQuest SciTech Collection
6Materials Science & Engineering Database
7ProQuest Central (new)
8ProQuest Central Korea
9SciTech Premium Collection
10Technology Collection
11ProQuest Central Essentials
12ProQuest One Academic
13Engineering Collection (ProQuest)
oafree_for_read
links
openurl$$Topenurl_article
openurlfulltext$$Topenurlfull_article
linktorsrc$$Uhttp://search.proquest.com/docview/2235435305/?pq-origsite=primo$$EView_record_in_ProQuest_(subscribers_only)
search
creatorcontrib
0Dhingra, Bhuwan
1Faruqui, Manaal
2Parikh, Ankur
3Ming-Wei, Chang
4Das, Dipanjan
5Cohen, William
titleHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
descriptionAutomatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using...
subject
0Judgments
1Texts
2Information Retrieval
3Computation and Language
general
0English
1Cornell University Library, arXiv.org
2Engineering Database
3Publicly Available Content Database
4ProQuest Engineering Collection
5ProQuest Technology Collection
6ProQuest SciTech Collection
7Materials Science & Engineering Database
8ProQuest Central (new)
9ProQuest Central Korea
10SciTech Premium Collection
11Technology Collection
12ProQuest Central Essentials
13ProQuest One Academic
14Engineering Collection (ProQuest)
sourceidproquest
recordidproquest2235435305
rsrctypearticle
creationdate2019
addtitlearXiv.org
searchscope
01007853
11008875
21008886
31009127
41009240
510000041
610000053
710000120
810000250
910000255
1010000256
1110000258
1210000260
1310000265
1410000268
1510000281
1610000348
1710000356
1810000360
19proquest
scope
01007853
11008875
21008886
31009127
41009240
510000041
610000053
710000120
810000250
910000255
1010000256
1110000258
1210000260
1310000265
1410000268
1510000281
1610000348
1710000356
1810000360
19proquest
lsr43
01007853true
11008875true
21008886true
31009127true
41009240true
510000041true
610000053true
710000120true
810000250true
910000255true
1010000256true
1110000258true
1210000260true
1310000265true
1410000268true
1510000281true
1610000348true
1710000356true
1810000360true
startdate20190603
enddate20190603
sort
titleHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
authorDhingra, Bhuwan ; Faruqui, Manaal ; Parikh, Ankur ; Ming-Wei, Chang ; Das, Dipanjan ; Cohen, William
creationdate20190603
lso0120190603
facets
frbrgroupid7125003812640448185
frbrtype6
newrecords20190611
languageeng
creationdate2019
topic
0Judgments
1Texts
2Information Retrieval
3Computation and Language
collection
0Engineering Database
1Publicly Available Content Database
2ProQuest Engineering Collection
3ProQuest Technology Collection
4ProQuest SciTech Collection
5Materials Science & Engineering Database
6ProQuest Central (new)
7ProQuest Central Korea
8SciTech Premium Collection
9Technology Collection
10ProQuest Central Essentials
11ProQuest One Academic
12Engineering Collection (ProQuest)
prefilterarticles
rsrctypearticles
creatorcontrib
0Dhingra, Bhuwan
1Faruqui, Manaal
2Parikh, Ankur
3Ming-Wei, Chang
4Das, Dipanjan
5Cohen, William
jtitlearXiv.org
delivery
delcategoryRemote Search Resource
fulltextfulltext_linktorsrc
addata
aulast
0Dhingra
1Faruqui
2Parikh
3Ming-Wei
4Das
5Cohen
aufirst
0Bhuwan
1Manaal
2Ankur
3Chang
4Dipanjan
5William
au
0Dhingra, Bhuwan
1Faruqui, Manaal
2Parikh, Ankur
3Ming-Wei, Chang
4Das, Dipanjan
5Cohen, William
atitleHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
jtitlearXiv.org
risdate20190603
formatjournal
genrearticle
ristypeJOUR
abstractAutomatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using...
copIthaca
pubCornell University Library, arXiv.org
urlhttp://search.proquest.com/docview/2235435305/
oafree_for_read
date2019-06-03