Assessing the trade-off between system building cost and output quality in data-to-text generation


As of July 2018 University of Brighton Repository is no longer updated. Please see our new repository at

Belz, Anja and Kow, Eric (2010) Assessing the trade-off between system building cost and output quality in data-to-text generation In: Krahmer, E. and Theune, M., eds. Empirical Methods in Natural Language Generation. Lecture Notes in Computer Science, 5790 . Springer-Verlag, Berlin, Heidelberg, pp. 180-200. ISBN 9783642155727

Full text not available from this repository.


Data-to-text generation systems tend to be knowledge-based and manually built, which limits their reusability and makes them time and cost-intensive to create and maintain. Methods for automating (part of) the system building process exist, but do such methods risk a loss in output quality? In this paper, we investigate the cost/quality trade-off in generation system building. We compare six data-to-text systems which were created by predominantly automatic techniques against six systems for the same domain which were created by predominantly manual techniques. We evaluate the systems using intrinsic automatic metrics and human quality ratings. We find that there is some correlation between degree of automation in the system-building process and output quality (more automation tending to mean lower evaluation scores). We also find that there are discrepancies between the results of the automatic evaluation metrics and the human-assessed evaluation experiments. We discuss caveats in assessing system-building cost and implications of the discrepancies in automatic and human evaluation.

Item Type: Chapter in book
Subjects: G000 Computing and Mathematical Sciences > G400 Computing
Q000 Languages and Literature - Linguistics and related subjects > Q100 Linguistics
DOI (a stable link to the resource): 10.1007/978-3-642-15573-4_10
Faculties: Faculty of Science and Engineering > School of Computing, Engineering and Mathematics > Natural Language Technology
Depositing User: Converis
Date Deposited: 16 Feb 2012 10:34
Last Modified: 25 Mar 2015 12:05

Actions (login required)

View Item View Item


Downloads per month over past year