Brigitte Grau


2020

pdf bib
Building a Multimodal Entity Linking Dataset From Tweets
Omar Adjali | Romaric Besançon | Olivier Ferret | Hervé Le Borgne | Brigitte Grau
Proceedings of the 12th Language Resources and Evaluation Conference

The task of Entity linking, which aims at associating an entity mention with a unique entity in a knowledge base (KB), is useful for advanced Information Extraction tasks such as relation extraction or event detection. Most of the studies that address this problem rely only on textual documents while an increasing number of sources are multimedia, in particular in the context of social media where messages are often illustrated with images. In this article, we address the Multimodal Entity Linking (MEL) task, and more particularly the problem of its evaluation. To this end, we propose a novel method to quasi-automatically build annotated datasets to evaluate methods on the MEL task. The method collects text and images to jointly build a corpus of tweets with ambiguous mentions along with a Twitter KB defining the entities. We release a new annotated dataset of Twitter posts associated with images. We study the key characteristics of the proposed dataset and evaluate the performance of several MEL approaches on it.

2018

pdf bib
An Adaption of BIOASQ Question Answering dataset for Machine Reading systems by Manual Annotations of Answer Spans.
Sanjay Kamath | Brigitte Grau | Yue Ma
Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering

BIOASQ Task B Phase B challenge focuses on extracting answers from snippets for a given question. The dataset provided by the organizers contains answers, but not all their variants. Henceforth a manual annotation was performed to extract all forms of correct answers. This article shows the impact of using all occurrences of correct answers for training on the evaluation scores which are improved significantly.

2017

pdf bib
Apprendre des représentations jointes de mots et d’entités pour la désambiguïsation d’entités (Combining Word and Entity Embeddings for Entity Linking)
José Moreno | Romaric Besançon | Romain Beaumont | Eva D’Hondt | Anne-Laure Ligozat | Sophie Rosset | Xavier Tannier | Brigitte Grau
Actes des 24ème Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 - Articles longs

La désambiguïsation d’entités (ou liaison d’entités), qui consiste à relier des mentions d’entités d’un texte à des entités d’une base de connaissance, est un problème qui se pose, entre autre, pour le peuplement automatique de bases de connaissances à partir de textes. Une difficulté de cette tâche est la résolution d’ambiguïtés car les systèmes ont à choisir parmi un nombre important de candidats. Cet article propose une nouvelle approche fondée sur l’apprentissage joint de représentations distribuées des mots et des entités dans le même espace, ce qui permet d’établir un modèle robuste pour la comparaison entre le contexte local de la mention d’entité et les entités candidates.

pdf bib
Generating a Training Corpus for OCR Post-Correction Using Encoder-Decoder Model
Eva D’hondt | Cyril Grouin | Brigitte Grau
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

In this paper we present a novel approach to the automatic correction of OCR-induced orthographic errors in a given text. While current systems depend heavily on large training corpora or external information, such as domain-specific lexicons or confidence scores from the OCR process, our system only requires a small amount of (relatively) clean training data from a representative corpus to learn a character-based statistical language model using Bidirectional Long Short-Term Memory Networks (biLSTMs). We demonstrate the versatility and adaptability of our system on different text corpora with varying degrees of textual noise, including a real-life OCR corpus in the medical domain.

2016

pdf bib
Low-resource OCR error detection and correction in French Clinical Texts
Eva D’hondt | Cyril Grouin | Brigitte Grau
Proceedings of the Seventh International Workshop on Health Text Mining and Information Analysis

pdf bib
AppFM, une plate-forme de gestion de modules de TAL (AppFM, a tool for managing NLP modules)
Paul Bui-Quang | Brigitte Grau | Patrick Paroubek
Actes de la conférence conjointe JEP-TALN-RECITAL 2016. volume 5 : Démonstrations

AppFM 1 est un outil à mi-chemin entre un environnement de création de chaînes modulaires de TAL et un gestionnaire de services systèmes. Il permet l’intégration d’applications ayant des dépendances complexes en des chaînes de traitements réutilisables facilement par le biais de multiples interfaces.

2015

pdf bib
Estimation de l’homogénéité sémantique pour les Questionnaires à Choix Multiples
Van-Minh Pho | Anne-Laure Ligozat | Brigitte Grau
Actes de la 22e conférence sur le Traitement Automatique des Langues Naturelles. Articles longs

L’homogénéité sémantique stipule que des termes sont sémantiquement proches mais non similaires. Cette notion est au cœur de travaux relatifs à la génération automatique de questionnaires à choix multiples, et particulièrement à la sélection automatique de distracteurs. Dans cet article, nous présentons une méthode d’estimation de l’homogénéité sémantique dans un cadre de validation automatique de distracteurs. Cette méthode est fondée sur une combinaison de plusieurs critères de voisinage et de similarité sémantique entre termes, par apprentissage automatique. Nous montrerons que notre méthode permet d’obtenir une meilleure estimation de l’homogénéité sémantique que les méthodes proposées dans l’état de l’art.

pdf bib
Noyaux de réécriture de phrases munis de types lexico-sémantiques
Martin Gleize | Brigitte Grau
Actes de la 22e conférence sur le Traitement Automatique des Langues Naturelles. Articles longs

De nombreux problèmes en traitement automatique des langues requièrent de déterminer si deux phrases sont des réécritures l’une de l’autre. Une solution efficace consiste à apprendre les réécritures en se fondant sur des méthodes à noyau qui mesurent la similarité entre deux réécritures de paires de phrases. Toutefois, ces méthodes ne permettent généralement pas de prendre en compte des variations sémantiques entre mots, qui permettraient de capturer un plus grand nombre de règles de réécriture. Dans cet article, nous proposons la définition et l’implémentation d’une nouvelle classe de fonction noyau, fondée sur la réécriture de phrases enrichie par un typage pour combler ce manque. Nous l’évaluons sur deux tâches, la reconnaissance de paraphrases et d’implications textuelles.

pdf bib
A Unified Kernel Approach for Learning Typed Sentence Rewritings
Martin Gleize | Brigitte Grau
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

2014

pdf bib
Event Role Extraction using Domain-Relevant Word Representations
Emanuela Boroş | Romaric Besançon | Olivier Ferret | Brigitte Grau
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

pdf bib
Event Role Labelling using a Neural Network Model (Étiquetage en rôles événementiels fondé sur l’utilisation d’un modèle neuronal) [in French]
Emanuela Boroş | Romaric Besançon | Olivier Ferret | Brigitte Grau
Proceedings of TALN 2014 (Volume 1: Long Papers)

pdf bib
A hierarchical taxonomy for classifying hardness of inference tasks
Martin Gleize | Brigitte Grau
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

Exhibiting inferential capabilities is one of the major goals of many modern Natural Language Processing systems. However, if attempts have been made to define what textual inferences are, few seek to classify inference phenomena by difficulty. In this paper we propose a hierarchical taxonomy for inferences, relatively to their hardness, and with corpus annotation and system design and evaluation in mind. Indeed, a fine-grained assessment of the difficulty of a task allows us to design more appropriate systems and to evaluate them only on what they are designed to handle. Each of seven classes is described and provided with examples from different tasks like question answering, textual entailment and coreference resolution. We then test the classes of our hierarchy on the specific task of question answering. Our annotation process of the testing data at the QA4MRE 2013 evaluation campaign reveals that it is possible to quantify the contrasts in types of difficulty on datasets of the same task.

pdf bib
Multiple Choice Question Corpus Analysis for Distractor Characterization
Van-Minh Pho | Thibault André | Anne-Laure Ligozat | Brigitte Grau | Gabriel Illouz | Thomas François
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

In this paper, we present a study of MCQ aiming to define criteria in order to automatically select distractors. We are aiming to show that distractor editing follows rules like syntactic and semantic homogeneity according to associated answer, and the possibility to automatically identify this homogeneity. Manual analysis shows that homogeneity rule is respected to edit distractors and automatic analysis shows the possibility to reproduce these criteria. These ones can be used in future works to automatically select distractors, with the combination of other criteria.

2013

pdf bib
LIMSIILES: Basic English Substitution for Student Answer Assessment at SemEval 2013
Martin Gleize | Brigitte Grau
Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)

pdf bib
Semantic relation clustering for unsupervised information extraction (Regroupement sémantique de relations pour l’extraction d’information non supervisée) [in French]
Wei Wang | Romaric Besançon | Olivier Ferret | Brigitte Grau
Proceedings of TALN 2013 (Volume 1: Long Papers)

2012

pdf bib
Simplification de phrases pour l’extraction de relations (Sentence Simplification for Relation Extraction) [in French]
Anne-Lyse Minard | Anne-Laure Ligozat | Brigitte Grau
Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 2: TALN

pdf bib
Evaluation of Unsupervised Information Extraction
Wei Wang | Romaric Besançon | Olivier Ferret | Brigitte Grau
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

Unsupervised methods gain more and more attention nowadays in information extraction area, which allows to design more open extraction systems. In the domain of unsupervised information extraction, clustering methods are of particular importance. However, evaluating the results of clustering remains difficult at a large scale, especially in the absence of reliable reference. On the basis of our experiments on unsupervised relation extraction, we first discuss in this article how to evaluate clustering quality without a reference by relying on internal measures. Then we propose a method, supported by a dedicated annotation tool, for building a set of reference clusters of relations from a corpus. Moreover, we apply it to our experimental framework and illustrate in this way how to build a significant reference for unsupervised relation extraction, more precisely made of 80 clusters gathering more than 4,000 relation instances, in a short time. Finally, we present how such reference is exploited for the evaluation of clustering with external measures and analyze the results of the application of these measures to the clusters of relations produced by our unsupervised relation extraction system.

pdf bib
Methods Combination and ML-based Re-ranking of Multiple Hypothesis for Question-Answering Systems
Arnaud Grappy | Brigitte Grau | Sophie Rosset
Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data

2011

pdf bib
Multi-class SVM for Relation Extraction from Clinical Reports
Anne-Lyse Minard | Anne-Laure Ligozat | Brigitte Grau
Proceedings of the International Conference Recent Advances in Natural Language Processing 2011

2010

pdf bib
Question Answering on Web Data: The QA Evaluation in Quæro
Ludovic Quintard | Olivier Galibert | Gilles Adda | Brigitte Grau | Dominique Laurent | Véronique Moriceau | Sophie Rosset | Xavier Tannier | Anne Vilnat
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

In the QA and information retrieval domains progress has been assessed via evaluation campaigns(Clef, Ntcir, Equer, Trec).In these evaluations, the systems handle independent questions and should provide one answer to each question, extracted from textual data, for both open domain and restricted domain. Quæro is a program promoting research and industrial innovation on technologies for automatic analysis and classification of multimedia and multilingual documents. Among the many research areas concerned by Quæro. The Quaero project organized a series of evaluations of Question Answering on Web Data systems in 2008 and 2009. For each language, English and French the full corpus has a size of around 20Gb for 2.5M documents. We describe the task and corpora, and especially the methodologies used in 2008 to construct the test of question and a new one in the 2009 campaign. Six types of questions were addressed, factual, Non-factual(How, Why, What), List, Boolean. A description of the participating systems and the obtained results is provided. We show the difficulty for a question-answering system to work with complex data and questions.

pdf bib
Fine-grained Linguistic Evaluation of Question Answering Systems
Sarra El Ayari | Brigitte Grau | Anne-Laure Ligozat
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

Question answering systems are complex systems using natural language processing. Some evaluation campaigns are organized to evaluate such systems in order to propose a classification of systems based on final results (number of correct answers). Nevertheless, teams need to evaluate more precisely the results obtained by their systems if they want to do a diagnostic evaluation. There are no tools or methods to do these evaluations systematically. We present REVISE, a tool for glass box evaluation based on diagnostic of question answering system results.

pdf bib
A Corpus for Studying Full Answer Justification
Arnaud Grappy | Brigitte Grau | Olivier Ferret | Cyril Grouin | Véronique Moriceau | Isabelle Robba | Xavier Tannier | Anne Vilnat | Vincent Barbier
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

Question answering (QA) systems aim at retrieving precise information from a large collection of documents. To be considered as reliable by users, a QA system must provide elements to evaluate the answer. This notion of answer justification can also be useful when developping a QA system in order to give criteria for selecting correct answers. An answer justification can be found in a sentence, a passage made of several consecutive sentences or several passages of a document or several documents. Thus, we are interesting in pinpointing the set of information that allows to verify the correctness of the answer in a candidate passage and the question elements that are missing in this passage. Moreover, the relevant information is often given in texts in a different form from the question form: anaphora, paraphrases, synonyms. In order to have a better idea of the importance of all the phenomena we underlined, and to provide enough examples at the QA developer's disposal to study them, we decided to build an annotated corpus.

2009

pdf bib
Corpus Study of Kidney-related Experimental Data in Scientific Papers
Brigitte Grau | Anne-Laure Ligozat | Anne-Lyse Minard
Proceedings of the Workshop on Biomedical Information Extraction

2006

pdf bib
EQueR: the French Evaluation campaign of Question-Answering Systems
Christelle Ayache | Brigitte Grau | Anne Vilnat
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

This paper describes the EQueR-EVALDA Evaluation Campaign, the French evaluation campaign of Question-Answering (QA) systems. The EQueR Evaluation Campaign included two tasks of automatic answer retrieval: the first one was a QA task over a heterogeneous collection of texts - mainly newspaper articles, and the second one a specialised one in the Medical field over a corpus of medical texts. In total, seven groups participated in the General task and five groups participated in the Medical task. For the General task, the best system obtained 81.46% of correct answers during the evalaution of the passages, while it obtained 67.24% during the evaluation of the short answers. We describe herein the specifications, the corpora, the evaluation, the phase of judgment of results, the scoring phase and the results for the two different types of evaluation.

pdf bib
FRASQUES: A Question Answering system in the EQueR evaluation campaign
Brigitte Grau | Anne-Laure Ligozat | Isabelle Robba | Anne Vilnat | Laura Monceaux
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

Question-answering (QA) systems aim at providing either a small passage or just the answer to a question in natural language. We have developed several QA systems that work on both English and French. This way, we are able to provide answers to questions given in both languages by searching documents in both languages also. In this article, we present our French monolingual system FRASQUES which participated in the EQueR evaluation campaign of QA systems for French in 2004. First, the QA architecture common to our systems is shown. Then, for every step of the QA process, we consider which steps are language-independent, and for those that are language-dependent, the tools or processes that need to be adapted to switch for one language to another. Finally, our results at EQueR are given and commented; an error analysis is conducted, and the kind of knowledge needed to answer a question is studied.

pdf bib
Evaluation and Improvement of Cross-Lingual Question AnsweringStrategies
Anne-Laure Ligozat | Brigitte Grau | Isabelle Robba | Anne Vilnat
Proceedings of the Workshop on Multilingual Question Answering - MLQA ‘06

2001

pdf bib
Two Levels of valuation in a Complex NL System
Jean-Baptiste Berthelin | Brigitte Grau | Martine Hurault-Plantet
Proceedings of the ACL 2001 Workshop on Evaluation Methodologies for Language and Dialogue Systems

pdf bib
A Cross-Comparison of Two Clustering Methods
Michele Jardino | Brigitte Grau | Olivier Ferret
Proceedings of the ACL 2001 Workshop on Evaluation Methodologies for Language and Dialogue Systems

pdf bib
Terminological Variants for Document Selection and Question/Answer Matching
Olivier Ferret | Brigitte Grau | Martine Hurault-Plantet | Gabriel Illouz | Christian Jacquemin
Proceedings of the ACL 2001 Workshop on Open-Domain Question Answering

1998

pdf bib
Thematic segmentation of texts: two methods for two kinds of texts
Olivier Ferret | Brigitte Grau | Nicolas Masson
COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics

pdf bib
Thematic Segmentation of Texts: Two Methods for Two Kind of Texts
Olivier Ferret | Brigitte Grau | Nicolas Masson
36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1