Margaret King


2006

pdf bib
A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation
Andrei Popescu-Belis | Paula Estrella | Margaret King | Nancy Underwood
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

In this paper, we propose a formal framework that takes into account the influence of the intended context of use of an NLP system on the procedure and the metrics used to evaluate the system. We introduce in particular the notion of a context-dependent quality model and explain how it can be adapted to a given context of use. More specifically, we define vector-space representations of contexts of use and of quality models, which are connected by a generic contextual quality model (GCQM). For each domain, experts in evaluation are needed to build a GCQM based on analytic knowledge and on previous evaluations, using the mechanism proposed here. The main inspiration source for this work is the FEMTI framework for the evaluation of machine translation, which implements partly the present model, and which is described briefly along with insights from other domains.

pdf bib
Evaluating Symbiotic Systems: the challenge
Margaret King | Nancy Underwood
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

This paper looks at a class of systems which pose severe problems in evaluation design for current conventional approaches to evaluation. After describing the two conventional evaluation paradigms: the “functionality paradigm” as typified by evaluation campaigns and the ISO inspired “user-centred” paradigm typified by the work of the EAGLES and ISLE projects, it goes on to outline the problems posed by the evaluation of systems which are designed to work in critical interaction with a human expert user and to work over vast amounts of data. These systems pose problems for both paradigms although for different reasons. The primary aim of this paper is to provoke discussion and the search for solutions. We have no proven solutions at present. However, we describe a programme of exploratory research on which we have already embarked, which involves ground clearing work which we expect to result in a deep understanding of the systems and users, a pre-requisite for developing a general framework for evaluation in this field.

2002

pdf bib
Computer-Aided Specification of Quality Models for Machine Translation Evaluation
Eduard Hovy | Margaret King | Andrei Popescu-Belis
Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02)

2000

pdf bib
Methods and Metrics for the Evaluation of Dictation Systems: a Case Study
Maria Canelli | Daniele Grasso | Margaret King
Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00)

1994

bib
Evaluating translation
Margaret King
Machine Translation and Translation Theory

1990

pdf bib
Using Test Suites in Evaluation of Machine Translation Systems
Margaret King | Kirsten Falkedal
COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics

1986

pdf bib
Machine Translation already does Work
Margaret King
24th Annual Meeting of the Association for Computational Linguistics

1984

pdf bib
When Is the Next ALPAC Report Due?
Margaret King
10th International Conference on Computational Linguistics and 22nd Annual Meeting of the Association for Computational Linguistics