Gerald Penn


2020

pdf bib
Temporal Histories of Epidemic Events (THEE): A Case Study in Temporal Annotation for Public Health
Jingcheng Niu | Victoria Ng | Gerald Penn | Erin E. Rees
Proceedings of the 12th Language Resources and Evaluation Conference

We present a new temporal annotation standard, THEE-TimeML, and a corpus TheeBank enabling precise temporal information extraction (TIE) for event-based surveillance (EBS) systems in the public health domain. Current EBS must estimate the occurrence time of each event based on coarse document metadata such as document publication time. Because of the complicated language and narration style of news articles, estimated case outbreak times are often inaccurate or even erroneous. Thus, it is necessary to create annotation standards and corpora to facilitate the development of TIE systems in the public health domain to address this problem.We will discuss the adaptations that have proved necessary for this domain as we present THEE-TimeML and TheeBank. Finally, we document the corpus annotation process, and demonstrate the immediate benefit to public health applications brought by the annotations.

pdf bib
FAB: The French Absolute Beginner Corpus for Pronunciation Training
Sean Robertson | Cosmin Munteanu | Gerald Penn
Proceedings of the 12th Language Resources and Evaluation Conference

We introduce the French Absolute Beginner (FAB) speech corpus. The corpus is intended for the development and study of Computer-Assisted Pronunciation Training (CAPT) tools for absolute beginner learners. Data were recorded during two experiments focusing on using a CAPT system in paired role-play tasks. The setting grants FAB three distinguishing features from other non-native corpora: the experimental setting is ecologically valid, closing the gap between training and deployment; it features a label set based on teacher feedback, allowing for context-sensitive CAPT; and data have been primarily collected from absolute beginners, a group often ignored. Participants did not read prompts, but instead recalled and modified dialogues that were modelled in videos. Unable to distinguish modelled words solely from viewing videos, speakers often uttered unintelligible or out-of-L2 words. The corpus is split into three partitions: one from an experiment with minimal feedback; another with explicit, word-level feedback; and a third with supplementary read-and-record data. A subset of words in the first partition has been labelled as more or less native, with inter-annotator agreement reported. In the explicit feedback partition, labels are derived from the experiment’s online feedback. The FAB corpus is scheduled to be made freely available by the end of 2020.

pdf bib
Grammaticality and Language Modelling
Jingcheng Niu | Gerald Penn
Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

Ever since Pereira (2000) provided evidence against Chomsky’s (1957) conjecture that statistical language modelling is incommensurable with the aims of grammaticality prediction as a research enterprise, a new area of research has emerged that regards statistical language models as “psycholinguistic subjects” and probes their ability to acquire syntactic knowledge. The advent of The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) has earned a spot on the leaderboard for acceptability judgements, and the polemic between Lau et al. (2017) and Sprouse et al. (2018) has raised fundamental questions about the nature of grammaticality and how acceptability judgements should be elicited. All the while, we are told that neural language models continue to improve. That is not an easy claim to test at present, however, because there is almost no agreement on how to measure their improvement when it comes to grammaticality and acceptability judgements. The GLUE leaderboard bundles CoLA together with a Matthews correlation coefficient (MCC), although probably because CoLA’s seminal publication was using it to compute inter-rater reliabilities. Researchers working in this area have used other accuracy and correlation scores, often driven by a need to reconcile and compare various discrete and continuous variables with each other. The score that we will advocate for in this paper, the point biserial correlation, in fact compares a discrete variable (for us, acceptability judgements) to a continuous variable (for us, neural language model probabilities). The only previous work in this area to choose the PBC that we are aware of is Sprouse et al. (2018a), and that paper actually applied it backwards (with some justification) so that the language model probability was treated as the discrete binary variable by setting a threshold. With the PBC in mind, we will first reappraise some recent work in syntactically targeted linguistic evaluations (Hu et al., 2020), arguing that while their experimental design sets a new high watermark for this topic, their results may not prove what they have claimed. We then turn to the task-independent assessment of language models as grammaticality classifiers. Prior to the introduction of the GLUE leaderboard, the vast majority of this assessment was essentially anecdotal, and we find the use of the MCC in this regard to be problematic. We conduct several studies with PBCs to compare several popular language models. We also study the effects of several variables such as normalization and data homogeneity on PBC.

pdf bib
Supertagging with CCG primitives
Aditya Bhargava | Gerald Penn
Proceedings of the 5th Workshop on Representation Learning for NLP

In CCG and other highly lexicalized grammars, supertagging a sentence’s words with their lexical categories is a critical step for efficient parsing. Because of the high degree of lexicalization in these grammars, the lexical categories can be very complex. Existing approaches to supervised CCG supertagging treat the categories as atomic units, even when the categories are not simple; when they encounter words with categories unseen during training, their guesses are accordingly unsophisticated. In this paper, we make use of the primitives and operators that constitute the lexical categories of categorial grammars. Instead of opaque labels, we treat lexical categories themselves as linear sequences. We present an LSTM-based model that replaces standard word-level classification with prediction of a sequence of primitives, similarly to LSTM decoders. Our model obtains state-of-the-art word accuracy for single-task English CCG supertagging, increases parser coverage and F1, and is able to produce novel categories. Analysis shows a synergistic effect between this decomposed view and incorporation of prediction history.

2019

pdf bib
Rationally Reappraising ATIS-based Dialogue Systems
Jingcheng Niu | Gerald Penn
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

The Air Travel Information Service (ATIS) corpus has been the most common benchmark for evaluating Spoken Language Understanding (SLU) tasks for more than three decades since it was released. Recent state-of-the-art neural models have obtained F1-scores near 98% on the task of slot filling. We developed a rule-based grammar for the ATIS domain that achieves a 95.82% F1-score on our evaluation set. In the process, we furthermore discovered numerous shortcomings in the ATIS corpus annotation, which we have fixed. This paper presents a detailed account of these shortcomings, our proposed repairs, our rule-based grammar and the neural slot-filling architectures associated with ATIS. We also rationally reappraise the motivations for choosing a neural architecture in view of this account. Fixing the annotation errors results in a relative error reduction of between 19.4 and 52% across all architectures. We nevertheless argue that neural models must play a different role in ATIS dialogues because of the latter’s lack of variety.

pdf bib
Proceedings of the 16th Meeting on the Mathematics of Language
Philippe de Groote | Frank Drewes | Gerald Penn
Proceedings of the 16th Meeting on the Mathematics of Language

2017

pdf bib
Vowel and Consonant Classification through Spectral Decomposition
Patricia Thaine | Gerald Penn
Proceedings of the First Workshop on Subword and Character Level Models in NLP

We consider two related problems in this paper. Given an undeciphered alphabetic writing system or mono-alphabetic cipher, determine: (1) which of its letters are vowels and which are consonants; and (2) whether the writing system is a vocalic alphabet or an abjad. We are able to show that a very simple spectral decomposition based on character co-occurrences provides nearly perfect performance with respect to answering both question types.

2016

pdf bib
Evaluating Sentiment Analysis in the Context of Securities Trading
Siavash Kazemian | Shunan Zhao | Gerald Penn
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2014

pdf bib
Evaluating Sentiment Analysis Evaluation: A Case Study in Securities Trading
Siavash Kazemian | Shunan Zhao | Gerald Penn
Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis

pdf bib
Unsupervised Sentence Enhancement for Automatic Summarization
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

2013

pdf bib
Why Letter Substitution Puzzles are Not Hard to Solve: A Case Study in Entropy and Probabilistic Search-Complexity
Eric Corlett | Gerald Penn
Proceedings of the 13th Meeting on the Mathematics of Language (MoL 13)

pdf bib
Probabilistic Domain Modelling With Contextualized Distributional Semantic Vectors
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf bib
Towards Robust Abstractive Multi-Document Summarization: A Caseframe Analysis of Centrality and Domain
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf bib
The mathematics of language learning
András Kornai | Gerald Penn | James Rogers | Anssi Yli-Jyrä
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Tutorials)

2012

pdf bib
Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH
Miriam Butt | Sheelagh Carpendale | Gerald Penn | Jelena Prokić | Michael Cysouw
Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH

pdf bib
Ecological Validity and the Evaluation of Speech Summarization Quality
Anthony McCallum | Cosmin Munteanu | Gerald Penn | Xiaodan Zhu
Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization

pdf bib
Evaluating Distributional Models of Semantics for Syntactically Invariant Inference
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

pdf bib
Unsupervised Detection of Downward-Entailing Operators By Maximizing Classification Certainty
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

pdf bib
Flexible Structural Analysis of Near-Meet-Semilattices for Typed Unification-Based Grammar Design
Rouzbeh Farahmand | Gerald Penn
Proceedings of COLING 2012

pdf bib
On Panini and the Generative Capacity of Contextualized Replacement Systems
Gerald Penn | Paul Kiparsky
Proceedings of COLING 2012: Posters

2011

pdf bib
Indexing Spoken Documents with Hierarchical Semantic Structures: Semantic Tree-to-string Alignment Models
Xiaodan Zhu | Colin Cherry | Gerald Penn
Proceedings of 5th International Joint Conference on Natural Language Processing

2010

pdf bib
Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Ron Kaplan | Jill Burstein | Mary Harper | Gerald Penn
Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics

pdf bib
Imposing Hierarchical Browsing Structures onto Spoken Documents
Xiaodan Zhu | Colin Cherry | Gerald Penn
Coling 2010: Posters

pdf bib
Entity-Based Local Coherence Modelling Using Topological Fields
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

pdf bib
Accurate Context-Free Parsing with Combinatory Categorial Grammar
Timothy A. D. Fowler | Gerald Penn
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

pdf bib
An Exact A* Method for Deciphering Letter-Substitution Ciphers
Eric Corlett | Gerald Penn
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

pdf bib
A Generalized-Zero-Preserving Method for Compact Encoding of Concept Lattices
Matthew Skala | Victoria Krakovna | János Kramár | Gerald Penn
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

pdf bib
Utilizing Extra-Sentential Context for Parsing
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing

2009

pdf bib
Topological Field Parsing of German
Jackie Chi Kit Cheung | Gerald Penn
Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP

pdf bib
Summarizing multiple spoken documents: finding evidence from untranscribed audio
Xiaodan Zhu | Gerald Penn | Frank Rudzicz
Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP

pdf bib
Improving Automatic Speech Recognition for Lectures through Transformation-based Rules Learned from Minimal Data
Cosmin Munteanu | Gerald Penn | Xiaodan Zhu
Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP

2008

pdf bib
A Critical Reassessment of Evaluation Baselines for Speech Summarization
Gerald Penn | Xiaodan Zhu
Proceedings of ACL-08: HLT

pdf bib
Interactive Visualization for Computational Linguistics
Christopher Collins | Gerald Penn | Sheelagh Carpendale
Tutorial Abstracts of ACL-08: HLT

pdf bib
Proceedings of the Workshop on Parsing German
Sandra Kübler | Gerald Penn
Proceedings of the Workshop on Parsing German

2006

pdf bib
Control Strategies for Parsing with Freer Word-Order Languages
Gerald Penn | Stefan Banjevic | Michael Demko
Proceedings of the Third Workshop on Constraints and Language Processing

pdf bib
Quantitative Methods for Classifying Writing Systems
Gerald Penn | Travis Choma
Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers

pdf bib
Comparing the roles of textual, acoustic and spoken-language features on spontaneous-conversation summarization
Xiaodan Zhu | Gerald Penn
Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers

2004

pdf bib
Optimizing Typed Feature Structure Grammar Parsing through Non-Statistical Indexing
Cosmin Munteanu | Gerald Penn
Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)

pdf bib
Head-Driven Parsing for Word Lattices
Christopher Collins | Bob Carpenter | Gerald Penn
Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)

pdf bib
Balancing Clarity and Efficiency in Typed Feature Logic Through Delaying
Gerald Penn
Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)

2003

pdf bib
AVM Description Compilation using Types as Modes
Gerald Penn
10th Conference of the European Chapter of the Association for Computational Linguistics

pdf bib
Topological Parsing
Gerald Penn | Mohammad Haji-Abdolhosseini
10th Conference of the European Chapter of the Association for Computational Linguistics

pdf bib
A Tabulation-Based Parsing Method that Reduces Copying
Gerald Penn | Cosmin Munteanu
Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics

pdf bib
Book Reviews: Linguistic Evolution through Language Acquisition: Formal and Computational Models edited by Ted Briscoe; Implementing Typed Feature Structure Grammars by Ann Copestake
Michael A. Arbib | Gerald Penn
Computational Linguistics, Volume 29, Number 3, September 2003: Special Issue on the Web as Corpus

2002

pdf bib
A Web-based Instructional Platform for Contraint-Based Grammar Formalisms and Parsing
W. Detmar Meurers | Gerald Penn | Frank Richter
Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics

pdf bib
Generalized Encoding of Description Spaces and its Application to Typed Feature Structures
Gerald Penn
Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics

2001

pdf bib
Tractability and Structural Closures in Attribute Logic Type Signatures
Gerald Penn
Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics

2000

pdf bib
Book Reviews: The Mathematics of Syntactic Structure: Trees and their Logics
Gerald Penn
Computational Linguistics, Volume 26, Number 2, June 2000

1998

pdf bib
Parametric Types for Typed Attribute-Value Logic
Gerald Penn
COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics

pdf bib
Parametric Types for Typed Attribute-Value Logic
Gerald Penn
36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2

1997

pdf bib
Head-Driven Generation and Indexing in ALE
Gerald Penn
Computational Environments for Grammar Development and Linguistic Engineering

1994

pdf bib
Default Finite State Machines and Finite State Phonology
Gerald Penn | Richmond Thomason
Computational Phonology