Albert Gatt


2020

pdf bib
Gradations of Error Severity in Automatic Image Descriptions
Emiel van Miltenburg | Wei-Ting Lu | Emiel Krahmer | Albert Gatt | Guanyi Chen | Lin Li | Kees van Deemter
Proceedings of the 13th International Conference on Natural Language Generation

Earlier research has shown that evaluation metrics based on textual similarity (e.g., BLEU, CIDEr, Meteor) do not correlate well with human evaluation scores for automatically generated text. We carried out an experiment with Chinese speakers, where we systematically manipulated image descriptions to contain different kinds of errors. Because our manipulated descriptions form minimal pairs with the reference descriptions, we are able to assess the impact of different kinds of errors on the perceived quality of the descriptions. Our results show that different kinds of errors elicit significantly different evaluation scores, even though all erroneous descriptions differ in only one character from the reference descriptions. Evaluation metrics based solely on textual similarity are unable to capture these differences, which (at least partially) explains their poor correlation with human judgments. Our work provides the foundations for future work, where we aim to understand why different errors are seen as more or less severe.

pdf bib
Annotating for Hate Speech: The MaNeCo Corpus and Some Input from Critical Discourse Analysis
Stavros Assimakopoulos | Rebecca Vella Muskat | Lonneke van der Plas | Albert Gatt
Proceedings of the 12th Language Resources and Evaluation Conference

This paper presents a novel scheme for the annotation of hate speech in corpora of Web 2.0 commentary. The proposed scheme is motivated by the critical analysis of posts made in reaction to news reports on the Mediterranean migration crisis and LGBTIQ+ matters in Malta, which was conducted under the auspices of the EU-funded C.O.N.T.A.C.T. project. Based on the realisation that hate speech is not a clear-cut category to begin with, appears to belong to a continuum of discriminatory discourse and is often realised through the use of indirect linguistic means, it is argued that annotation schemes for its detection should refrain from directly including the label ‘hate speech,’ as different annotators might have different thresholds as to what constitutes hate speech and what not. In view of this, we propose a multi-layer annotation scheme, which is pilot-tested against a binary ±hate speech classification and appears to yield higher inter-annotator agreement. Motivating the postulation of our scheme, we then present the MaNeCo corpus on which it will eventually be used; a substantial corpus of on-line newspaper comments spanning 10 years.

pdf bib
MASRI-HEADSET: A Maltese Corpus for Speech Recognition
Carlos Daniel Hernandez Mena | Albert Gatt | Andrea DeMarco | Claudia Borg | Lonneke van der Plas | Amanda Muscat | Ian Padovani
Proceedings of the 12th Language Resources and Evaluation Conference

Maltese, the national language of Malta, is spoken by approximately 500,000 people. Speech processing for Maltese is still in its early stages of development. In this paper, we present the first spoken Maltese corpus designed purposely for Automatic Speech Recognition (ASR). The MASRI-HEADSET corpus was developed by the MASRI project at the University of Malta. It consists of 8 hours of speech paired with text, recorded by using short text snippets in a laboratory environment. The speakers were recruited from different geographical locations all over the Maltese islands, and were roughly evenly distributed by gender. This paper also presents some initial results achieved in baseline experiments for Maltese ASR using Sphinx and Kaldi. The MASRI HEADSET Corpus is publicly available for research/academic purposes.

pdf bib
Proceedings of LREC2020 Workshop "People in language, vision and the mind" (ONION2020)
Patrizia Paggio | Albert Gatt | Roman Klinger
Proceedings of LREC2020 Workshop "People in language, vision and the mind" (ONION2020)

2019

pdf bib
You Write like You Eat: Stylistic Variation as a Predictor of Social Stratification
Angelo Basile | Albert Gatt | Malvina Nissim
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Inspired by Labov’s seminal work on stylisticvariation as a function of social stratification,we develop and compare neural models thatpredict a person’s presumed socio-economicstatus, obtained through distant supervision,from their writing style on social media. Thefocus of our work is on identifying the mostimportant stylistic parameters to predict socio-economic group. In particular, we show theeffectiveness of morpho-syntactic features aspredictors of style, in contrast to lexical fea-tures, which are good predictors of topic

pdf bib
Visually grounded generation of entailments from premises
Somayeh Jafaritazehjani | Albert Gatt | Marc Tanti
Proceedings of the 12th International Conference on Natural Language Generation

Natural Language Inference (NLI) is the task of determining the semantic relationship between a premise and a hypothesis. In this paper, we focus on the generation of hypotheses from premises in a multimodal setting, to generate a sentence (hypothesis) given an image and/or its description (premise) as the input. The main goals of this paper are (a) to investigate whether it is reasonable to frame NLI as a generation task; and (b) to consider the degree to which grounding textual premises in visual information is beneficial to generation. We compare different neural architectures, showing through automatic and human evaluation that entailments can indeed be generated successfully. We also show that multimodal models outperform unimodal models in this task, albeit marginally

pdf bib
Best practices for the human evaluation of automatically generated text
Chris van der Lee | Albert Gatt | Emiel van Miltenburg | Sander Wubben | Emiel Krahmer
Proceedings of the 12th International Conference on Natural Language Generation

Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated. While there is some agreement regarding automatic metrics, there is a high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.

2018

pdf bib
Face2Text: Collecting an Annotated Image Description Corpus for the Generation of Rich Face Descriptions
Albert Gatt | Marc Tanti | Adrian Muscat | Patrizia Paggio | Reuben A Farrugia | Claudia Borg | Kenneth P Camilleri | Michael Rosner | Lonneke van der Plas
Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

pdf bib
Grounded Textual Entailment
Hoa Trong Vu | Claudio Greco | Aliia Erofeeva | Somayeh Jafaritazehjan | Guido Linders | Marc Tanti | Alberto Testoni | Raffaella Bernardi | Albert Gatt
Proceedings of the 27th International Conference on Computational Linguistics

Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics. Logic-based models analyse entailment in terms of possible worlds (interpretations, or situations) where a premise P entails a hypothesis H iff in all worlds where P is true, H is also true. Statistical models view this relationship probabilistically, addressing it in terms of whether a human would likely infer H from P. In this paper, we wish to bridge these two perspectives, by arguing for a visually-grounded version of the Textual Entailment task. Specifically, we ask whether models can perform better if, in addition to P and H, there is also an image (corresponding to the relevant “world” or “situation”). We use a multimodal version of the SNLI dataset (Bowman et al., 2015) and we compare “blind” and visually-augmented models of textual entailment. We show that visual information is beneficial, but we also conduct an in-depth error analysis that reveals that current multimodal models are not performing “grounding” in an optimal fashion.

pdf bib
Proceedings of the 11th International Conference on Natural Language Generation
Emiel Krahmer | Albert Gatt | Martijn Goudbeek
Proceedings of the 11th International Conference on Natural Language Generation

pdf bib
Meteorologists and Students: A resource for language grounding of geographical descriptors
Alejandro Ramos-Soto | Ehud Reiter | Kees van Deemter | Jose Alonso | Albert Gatt
Proceedings of the 11th International Conference on Natural Language Generation

We present a data resource which can be useful for research purposes on language grounding tasks in the context of geographical referring expression generation. The resource is composed of two data sets that encompass 25 different geographical descriptors and a set of associated graphical representations, drawn as polygons on a map by two groups of human subjects: teenage students and expert meteorologists.

pdf bib
Specificity measures and reference
Albert Gatt | Nicolás Marín | Gustavo Rivas-Gervilla | Daniel Sánchez
Proceedings of the 11th International Conference on Natural Language Generation

In this paper we study empirically the validity of measures of referential success for referring expressions involving gradual properties. More specifically, we study the ability of several measures of referential success to predict the success of a user in choosing the right object, given a referring expression. Experimental results indicate that certain fuzzy measures of success are able to predict human accuracy in reference resolution. Such measures are therefore suitable for the estimation of the success or otherwise of a referring expression produced by a generation algorithm, especially in case the properties in a domain cannot be assumed to have crisp denotations.

2017

pdf bib
Morphological Analysis for the Maltese Language: The challenges of a hybrid system
Claudia Borg | Albert Gatt
Proceedings of the Third Arabic Natural Language Processing Workshop

Maltese is a morphologically rich language with a hybrid morphological system which features both concatenative and non-concatenative processes. This paper analyses the impact of this hybridity on the performance of machine learning techniques for morphological labelling and clustering. In particular, we analyse a dataset of morphologically related word clusters to evaluate the difference in results for concatenative and non-concatenative clusters. We also describe research carried out in morphological labelling, with a particular focus on the verb category. Two evaluations were carried out, one using an unseen dataset, and another one using a gold standard dataset which was manually labelled. The gold standard dataset was split into concatenative and non-concatenative to analyse the difference in results between the two morphological systems.

pdf bib
What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?
Marc Tanti | Albert Gatt | Kenneth Camilleri
Proceedings of the 10th International Conference on Natural Language Generation

Image captioning has evolved into a core task for Natural Language Generation and has also proved to be an important testbed for deep learning approaches to handling multimodal representations. Most contemporary approaches rely on a combination of a convolutional network to handle image features, and a recurrent network to encode linguistic information. The latter is typically viewed as the primary “generation” component. Beyond this high-level characterisation, a CNN+RNN model supports a variety of architectural designs. The dominant model in the literature is one in which visual features encoded by a CNN are “injected” as part of the linguistic encoding process, driving the RNN’s linguistic choices. By contrast, it is possible to envisage an architecture in which visual and linguistic features are encoded separately, and merged at a subsequent stage. In this paper, we address two related questions: (1) Is direct injection the best way of combining multimodal information, or is a late merging alternative better for the image captioning task? (2) To what extent should a recurrent network be viewed as actually generating, rather than simply encoding, linguistic information?

pdf bib
LCT-MALTA’s Submission to RepEval 2017 Shared Task
Hoa Trong Vu | Thuong-Hai Pham | Xiaoyu Bai | Marc Tanti | Lonneke van der Plas | Albert Gatt
Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP

System using BiLSTM and max pooling. Embedding is enhanced by POS, character and dependency info.

2015

pdf bib
Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)
Anya Belz | Albert Gatt | François Portet | Matthew Purver
Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)

2014

pdf bib
Learning when to point: A data-driven approach
Albert Gatt | Patrizia Paggio
Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers

pdf bib
Crowd-sourcing evaluation of automatically acquired, morphologically related word groupings
Claudia Borg | Albert Gatt
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

The automatic discovery and clustering of morphologically related words is an important problem with several practical applications. This paper describes the evaluation of word clusters carried out through crowd-sourcing techniques for the Maltese language. The hybrid (Semitic-Romance) nature of Maltese morphology, together with the fact that no large-scale lexical resources are available for Maltese, make this an interesting and challenging problem.

2013

pdf bib
Proceedings of the 14th European Workshop on Natural Language Generation
Albert Gatt | Horacio Saggion
Proceedings of the 14th European Workshop on Natural Language Generation

pdf bib
What and Where: An Empirical Investigation of Pointing Gestures and Descriptions in Multimodal Referring Actions
Albert Gatt | Patrizia Paggio
Proceedings of the 14th European Workshop on Natural Language Generation

2012

pdf bib
A Repository of Data and Evaluation Resources for Natural Language Generation
Anja Belz | Albert Gatt
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

Starting in 2007, the field of natural language generation (NLG) has organised shared-task evaluation events every year, under the Generation Challenges umbrella. In the course of these shared tasks, a wealth of data has been created, along with associated task definitions and evaluation regimes. In other contexts too, sharable NLG data is now being created. In this paper, we describe the online repository that we have created as a one-stop resource for obtaining NLG task materials, both from Generation Challenges tasks and from other sources, where the set of materials provided for each task consists of (i) task definition, (ii) input and output data, (iii) evaluation software, (iv) documentation, and (v) publications reporting previous results.

pdf bib
Incorporating an Error Corpus into a Spellchecker for Maltese
Michael Rosner | Albert Gatt | Andrew Attard | Jan Joachimsen
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

This paper discusses the ongoing development of a new Maltese spell checker, highlighting the methodologies which would best suit such a language. We thus discuss several previous attempts, highlighting what we believe to be their weakest point: a lack of attention to context. Two developments are of particular interest, both of which concern the availability of language resources relevant to spellchecking: (i) the Maltese Language Resource Server (MLRS) which now includes a representative corpus of c. 100M words extracted from diverse documents including the Maltese Legislation, press releases and extracts from Maltese web-pages and (ii) an extensive and detailed corpus of spelling errors that was collected whilst part of the MLRS texts were being prepared. We describe the structure of these resources as well as the experimental approaches focused on context that we are now in a position to adopt. We describe the framework within which a variety of different approaches to spellchecking and evaluation will be carried out, and briefly discuss the first baseline system we have implemented. We conclude the paper with a roadmap for future improvements.

2011

pdf bib
Proceedings of the UCNLG+Eval: Language Generation and Evaluation Workshop
Anja Belz | Roger Evans | Albert Gatt | Kristina Striegnitz
Proceedings of the UCNLG+Eval: Language Generation and Evaluation Workshop

pdf bib
What is in a text and what does it do: Qualitative Evaluations of an NLG system – the BT-Nurse – using content analysis and discourse analysis
Rahul Sambaraju | Ehud Reiter | Robert Logie | Andy Mckinlay | Chris McVittie | Albert Gatt | Cindy Sykes
Proceedings of the 13th European Workshop on Natural Language Generation

pdf bib
If it may have happened before, it happened, but not necessarily before
Albert Gatt | François Portet
Proceedings of the 13th European Workshop on Natural Language Generation

pdf bib
Generation Challenges 2011 Preface
Anja Belz | Albert Gatt | Alexander Koller | Kristina Striegnitz
Proceedings of the 13th European Workshop on Natural Language Generation

2010

pdf bib
Textual Properties and Task-based Evaluation: Investigating the Role of Surface Properties, Structure and Content
Albert Gatt | François Portet
Proceedings of the 6th International Natural Language Generation Conference

pdf bib
Generation Challenges 2010 Preface
Anja Belz | Albert Gatt | Alexander Koller
Proceedings of the 6th International Natural Language Generation Conference

2009

pdf bib
Text Content and Task Performance in the Evaluation of a Natural Language Generation System
Albert Gatt | François Portet
Proceedings of the International Conference RANLP-2009

pdf bib
SimpleNLG: A Realisation Engine for Practical Applications
Albert Gatt | Ehud Reiter
Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009)

pdf bib
A Hearer-Oriented Evaluation of Referring Expression Generation
Imtiaz Hussain Khan | Kees van Deemter | Graeme Ritchie | Albert Gatt | Alexandra A. Cleland
Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009)

pdf bib
Generation Challenges 2009: Preface
Anja Belz | Albert Gatt
Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009)

pdf bib
The TUNA-REG Challenge 2009: Overview and Evaluation Results
Albert Gatt | Anja Belz | Eric Kow
Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009)

pdf bib
The GREC Main Subject Reference Generation Challenge 2009: Overview and Evaluation Results
Anja Belz | Eric Kow | Jette Viethen | Albert Gatt
Proceedings of the 2009 Workshop on Language Generation and Summarisation (UCNLG+Sum 2009)

2008

pdf bib
Intrinsic vs. Extrinsic Evaluation Measures for Referring Expression Generation
Anja Belz | Albert Gatt
Proceedings of ACL-08: HLT, Short Papers

pdf bib
Attribute Selection for Referring Expression Generation: New Algorithms and Evaluation Methods
Albert Gatt | Anja Belz
Proceedings of the Fifth International Natural Language Generation Conference

pdf bib
The Importance of Narrative and Other Lessons from an Evaluation of an NLG System that Summarises Clinical Data
Ehud Reiter | Albert Gatt | François Portet | Marian van der Meulen
Proceedings of the Fifth International Natural Language Generation Conference

pdf bib
REG Challenge Preface
Anja Belz | Albert Gatt
Proceedings of the Fifth International Natural Language Generation Conference

pdf bib
The GREC Challenge 2008: Overview and Evaluation Results
Anja Belz | Eric Kow | Jette Viethen | Albert Gatt
Proceedings of the Fifth International Natural Language Generation Conference

pdf bib
The TUNA Challenge 2008: Overview and Evaluation Results
Albert Gatt | Anja Belz | Eric Kow
Proceedings of the Fifth International Natural Language Generation Conference

2007

pdf bib
Evaluating algorithms for the Generation of Referring Expressions using a balanced corpus
Albert Gatt | Ielka van der Sluis | Kees van Deemter
Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07)

pdf bib
Incremental Generation of Plural Descriptions: Similarity and Partitioning
Albert Gatt | Kees van Deemter
Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)

2006

pdf bib
Conceptual Coherence in the Generation of Referring Expressions
Albert Gatt | Kees van Deemter
Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions

pdf bib
Building a Semantically Transparent Corpus for the Generation of Referring Expressions.
Kees van Deemter | Ielka van der Sluis | Albert Gatt
Proceedings of the Fourth International Natural Language Generation Conference

pdf bib
Structuring Knowledge for Reference Generation: A Clustering Algorithm
Albert Gatt
11th Conference of the European Chapter of the Association for Computational Linguistics