Elijah Mayfield


2020

pdf bib
Machine-Aided Annotation for Fine-Grained Proposition Types in Argumentation
Yohan Jo | Elijah Mayfield | Chris Reed | Eduard Hovy
Proceedings of the 12th Language Resources and Evaluation Conference

We introduce a corpus of the 2016 U.S. presidential debates and commentary, containing 4,648 argumentative propositions annotated with fine-grained proposition types. Modern machine learning pipelines for analyzing argument have difficulty distinguishing between types of propositions based on their factuality, rhetorical positioning, and speaker commitment. Inability to properly account for these facets leaves such systems inaccurate in understanding of fine-grained proposition types. In this paper, we demonstrate an approach to annotating for four complex proposition types, namely normative claims, desires, future possibility, and reported speech. We develop a hybrid machine learning and human workflow for annotation that allows for efficient and reliable annotation of complex linguistic phenomena, and demonstrate with preliminary analysis of rhetorical strategies and structure in presidential debates. This new dataset and method can support technical researchers seeking more nuanced representations of argument, as well as argumentation theorists developing new quantitative analyses.

pdf bib
Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models
Christopher Grimsley | Elijah Mayfield | Julia R.S. Bursten
Proceedings of the 12th Language Resources and Evaluation Conference

As the demand for explainable deep learning grows in the evaluation of language technologies, the value of a principled grounding for those explanations grows as well. Here we study the state-of-the-art in explanation for neural models for NLP tasks from the viewpoint of philosophy of science. We focus on recent evaluation work that finds brittleness in explanations obtained through attention mechanisms. We harness philosophical accounts of explanation to suggest broader conclusions from these studies. From this analysis, we assert the impossibility of causal explanations from attention layers over text data. We then introduce NLP researchers to contemporary philosophy of science theories that allow robust yet non-causal reasoning in explanation, giving computer scientists a vocabulary for future research.

pdf bib
Should You Fine-Tune BERT for Automated Essay Scoring?
Elijah Mayfield | Alan W Black
Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

Most natural language processing research now recommends large Transformer-based models with fine-tuning for supervised classification tasks; older strategies like bag-of-words features and linear models have fallen out of favor. Here we investigate whether, in automated essay scoring (AES) research, deep neural models are an appropriate technological choice. We find that fine-tuning BERT produces similar performance to classical models at significant additional cost. We argue that while state-of-the-art strategies do match existing best results, they come with opportunity costs in computational resources. We conclude with a review of promising areas for research on student essays where the unique characteristics of Transformers may provide benefits over classical methods to justify the costs.

2019

pdf bib
Stance Classification, Outcome Prediction, and Impact Assessment: NLP Tasks for Studying Group Decision-Making
Elijah Mayfield | Alan Black
Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science

In group decision-making, the nuanced process of conflict and resolution that leads to consensus formation is closely tied to the quality of decisions made. Behavioral scientists rarely have rich access to process variables, though, as unstructured discussion transcripts are difficult to analyze. Here, we define ways for NLP researchers to contribute to the study of groups and teams. We introduce three tasks alongside a large new corpus of over 400,000 group debates on Wikipedia. We describe the tasks and their importance, then provide baselines showing that BERT contextualized word embeddings consistently outperform other language representations.

bib
Principled Frameworks for Evaluating Ethics in NLP Systems
Shrimai Prabhumoye | Elijah Mayfield | Alan W Black
Proceedings of the 2019 Workshop on Widening NLP

We critique recent work on ethics in natural language processing. Those discussions have focused on data collection, experimental design, and interventions in modeling. But we argue that we ought to first understand the frameworks of ethics that are being used to evaluate the fairness and justice of algorithmic systems. Here, we begin that discussion by outlining deontological and consequentialist ethics, and make predictions on the research agenda prioritized by each.

pdf bib
Equity Beyond Bias in Language Technologies for Education
Elijah Mayfield | Michael Madaio | Shrimai Prabhumoye | David Gerritsen | Brittany McLaughlin | Ezekiel Dixon-Román | Alan W Black
Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

There is a long record of research on equity in schools. As machine learning researchers begin to study fairness and bias in earnest, language technologies in education have an unusually strong theoretical and applied foundation to build on. Here, we introduce concepts from culturally relevant pedagogy and other frameworks for teaching and learning, identifying future work on equity in NLP. We present case studies in a range of topics like intelligent tutoring systems, computer-assisted language learning, automated essay scoring, and sentiment analysis in classrooms, and provide an actionable agenda for research.

2013

pdf bib
Recognizing Rare Social Phenomena in Conversation: Empowerment Detection in Support Group Chatrooms
Elijah Mayfield | David Adamson | Carolyn Penstein Rosé
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2012

pdf bib
Historical Analysis of Legal Opinions with a Sparse Mixed-Effects Latent Variable Model
William Yang Wang | Elijah Mayfield | Suresh Naidu | Jeremiah Dittmar
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf bib
Hierarchical Conversation Structure Prediction in Multi-Party Chat
Elijah Mayfield | David Adamson | Carolyn Penstein Rosé
Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue

2011

pdf bib
Recognizing Authority in Dialogue with an Integer Linear Programming Constrained Model
Elijah Mayfield | Carolyn Penstein Rosé
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

2010

pdf bib
An Interactive Tool for Supporting Error Analysis for Text Mining
Elijah Mayfield | Carolyn Penstein-Rosé
Proceedings of the NAACL HLT 2010 Demonstration Session

pdf bib
Sentiment Classification using Automatically Extracted Subgraph Features
Shilpa Arora | Elijah Mayfield | Carolyn Penstein-Rosé | Eric Nyberg
Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text

2009

pdf bib
Sentence diagram generation using dependency parsing
Elijah Mayfield
Proceedings of the ACL-IJCNLP 2009 Student Research Workshop