2020
pdf
bib
abs
Analyzing Word Embedding Through Structural Equation Modeling
Namgi Han

Katsuhiko Hayashi

Yusuke Miyao
Proceedings of the 12th Language Resources and Evaluation Conference
Many researchers have tried to predict the accuracies of extrinsic evaluation by using intrinsic evaluation to evaluate word embedding. The relationship between intrinsic and extrinsic evaluation, however, has only been studied with simple correlation analysis, which has difficulty capturing complex causeeffect relationships and integrating external factors such as the hyperparameters of word embedding. To tackle this problem, we employ partial least squares path modeling (PLSPM), a method of structural equation modeling developed for causal analysis. We propose a causal diagram consisting of the evaluation results on the BATS, VecEval, and SentEval datasets, with a causal hypothesis that linguistic knowledge encoded in word embedding contributes to solving downstream tasks. Our PLSPM models are estimated with 600 word embeddings, and we prove the existence of causal relations between linguistic knowledge evaluated on BATS and the accuracies of downstream tasks evaluated on VecEval and SentEval in our PLSPM models. Moreover, we show that the PLSPM models are useful for analyzing the effect of hyperparameters, including the training algorithm, corpus, dimension, and context window, and for validating the effectiveness of intrinsic evaluation.
pdf
bib
abs
A Greedy Bitflip Training Algorithm for Binarized Knowledge Graph Embeddings
Katsuhiko Hayashi

Koki Kishimoto

Masashi Shimbo
Findings of the Association for Computational Linguistics: EMNLP 2020
This paper presents a simple and effective discrete optimization method for training binarized knowledge graph embedding model BCP. Unlike the prior work using a SGDbased method and quantization of realvalued vectors, the proposed method directly optimizes binary embedding vectors by a series of bit flipping operations. On the standard knowledge graph completion tasks, the BCP model trained with the proposed method achieved comparable performance with that trained with SGD as well as stateoftheart realvalued models with similar embedding dimensions.
pdf
bib
abs
A System for Worldwide COVID19 Information Aggregation
Akiko Aizawa

Frederic Bergeron

Junjie Chen

Fei Cheng

Katsuhiko Hayashi

Kentaro Inui

Hiroyoshi Ito

Daisuke Kawahara

Masaru Kitsuregawa

Hirokazu Kiyomaru

Masaki Kobayashi

Takashi Kodama

Sadao Kurohashi

Qianying Liu

Masaki Matsubara

Yusuke Miyao

Atsuyuki Morishima

Yugo Murawaki

Kazumasa Omura

Haiyue Song

Eiichiro Sumita

Shinji Suzuki

Ribeka Tanaka

Yu Tanaka

Masashi Toyoda

Nobuhiro Ueda

Honai Ueoka

Masao Utiyama

Ying Zhong
Proceedings of the 1st Workshop on NLP for COVID19 (Part 2) at EMNLP 2020
The global pandemic of COVID19 has made the public pay close attention to related news, covering various domains, such as sanitation, treatment, and effects on education. Meanwhile, the COVID19 condition is very different among the countries (e.g., policies and development of the epidemic), and thus citizens would be interested in news in foreign countries. We build a system for worldwide COVID19 information aggregation containing reliable articles from 10 regions in 7 languages sorted by topics. Our reliable COVID19 related website dataset collected through crowdsourcing ensures the quality of the articles. A neural machine translation module translates articles in other languages into Japanese and English. A BERTbased topicclassifier trained on our articletopic pair dataset helps users find their interested information efficiently by putting articles into different categories.
2019
pdf
bib
abs
A Noncommutative Bilinear Model for Answering Path Queries in Knowledge Graphs
Katsuhiko Hayashi

Masashi Shimbo
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP)
Bilinear diagonal models for knowledge graph embedding (KGE), such as DistMult and ComplEx, balance expressiveness and computational efficiency by representing relations as diagonal matrices. Although they perform well in predicting atomic relations, composite relations (relation paths) cannot be modeled naturally by the product of relation matrices, as the product of diagonal matrices is commutative and hence invariant with the order of relations. In this paper, we propose a new bilinear KGE model, called BlockHolE, based on block circulant matrices. In BlockHolE, relation matrices can be noncommutative, allowing composite relations to be modeled by matrix product. The model is parameterized in a way that covers a spectrum ranging from diagonal to full relation matrices. A fast computation technique can be developed on the basis of the duality of the Fourier transform of circulant matrices.
2018
pdf
bib
abs
Neural Tensor Networks with Diagonal Slice Matrices
Takahiro Ishihara

Katsuhiko Hayashi

Hitoshi Manabe

Masashi Shimbo

Masaaki Nagata
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Although neural tensor networks (NTNs) have been successful in many NLP tasks, they require a large number of parameters to be estimated, which often leads to overfitting and a long training time. We address these issues by applying eigendecomposition to each slice matrix of a tensor to reduce its number of paramters. First, we evaluate our proposed NTN models on knowledge graph completion. Second, we extend the models to recursive NTNs (RNTNs) and evaluate them on logical reasoning tasks. These experiments show that our proposed models learn better and faster than the original (R)NTNs.
pdf
bib
abs
HigherOrder Syntactic Attention Network for Longer Sentence Compression
Hidetaka Kamigaito

Katsuhiko Hayashi

Tsutomu Hirao

Masaaki Nagata
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
A sentence compression method using LSTM can generate fluent compressed sentences. However, the performance of this method is significantly degraded when compressing longer sentences since it does not explicitly handle syntactic features. To solve this problem, we propose a higherorder syntactic attention network (HiSAN) that can handle higherorder dependency features as an attention distribution on LSTM hidden states. Furthermore, to avoid the influence of incorrect parse results, we trained HiSAN by maximizing jointly the probability of a correct output with the attention distribution. Experimental results on Google sentence compression dataset showed that our method achieved the best performance on F1 as well as ROUGE1,2 and L scores, 83.2, 82.9, 75.8 and 82.7, respectively. In human evaluation, our methods also outperformed baseline methods in both readability and informativeness.
pdf
bib
Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices
Tomoki Matsuno

Katsuhiko Hayashi

Takahiro Ishihara

Hitoshi Manabe

Yuji Matsumoto
Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation
2017
pdf
bib
abs
On the Equivalence of Holographic and Complex Embeddings for Link Prediction
Katsuhiko Hayashi

Masashi Shimbo
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
We show the equivalence of two stateoftheart models for link prediction/knowledge graph completion: Nickel et al’s holographic embeddings and Trouillon et al.’s complex embeddings. We first consider a spectral version of the holographic embeddings, exploiting the frequency domain in the Fourier transform for efficient computation. The analysis of the resulting model reveals that it can be viewed as an instance of the complex embeddings with a certain constraint imposed on the initial vectors upon training. Conversely, any set of complex embeddings can be converted to a set of equivalent holographic embeddings.
pdf
bib
abs
Kbest Iterative Viterbi Parsing
Katsuhiko Hayashi

Masaaki Nagata
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
This paper presents an efficient and optimal parsing algorithm for probabilistic contextfree grammars (PCFGs). To achieve faster parsing, our proposal employs a pruning technique to reduce unnecessary edges in the search space. The key is to conduct repetitively Viterbi inside and outside parsing, while gradually expanding the search space to efficiently compute heuristic bounds used for pruning. Our experimental results using the English Penn Treebank corpus show that the proposed algorithm is faster than the standard CKY parsing algorithm. In addition, we also show how to extend this algorithm to extract kbest Viterbi parse trees.
pdf
bib
abs
Supervised Attention for SequencetoSequence Constituency Parsing
Hidetaka Kamigaito

Katsuhiko Hayashi

Tsutomu Hirao

Hiroya Takamura

Manabu Okumura

Masaaki Nagata
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
The sequencetosequence (Seq2Seq) model has been successfully applied to machine translation (MT). Recently, MT performances were improved by incorporating supervised attention into the model. In this paper, we introduce supervised attention to constituency parsing that can be regarded as another translation task. Evaluation results on the PTB corpus showed that the bracketing Fmeasure was improved by supervised attention.
pdf
bib
abs
Hierarchical Word Structurebased Parsing: A Feasibility Study on UDstyle Dependency Parsing in Japanese
Takaaki Tanaka

Katsuhiko Hayashi

Masaaki Nagata
Proceedings of the 15th International Conference on Parsing Technologies
In applying wordbased dependency parsing such as Universal Dependencies (UD) to Japanese, the uncertainty of word segmentation emerges for defining a word unit of the dependencies. We introduce the following hierarchical word structures to dependency parsing in Japanese: morphological units (a short unit word, SUW) and syntactic units (a long unit word, LUW). An SUW can be used to segment a sentence consistently, while it is too short to represent syntactic construction. An LUW is a unit including functional multiwords and LUWbased analysis facilitates the capturing of syntactic structure and makes parsing results more precise than SUWbased analysis. This paper describes the results of a feasibility study on the ability and the effectiveness of parsing methods based on hierarchical word structure (LUW chunking+parsing) in comparison to single layer word structure (SUW parsing). We also show joint analysis of LUWchunking and dependency parsing improves the performance of identifying predicateargument structures, while there is not much difference between overall results of them. not much difference between overall results of them.
2016
pdf
bib
Empty element recovery by spinal parser operations
Katsuhiko Hayashi

Masaaki Nagata
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
pdf
bib
Empirical comparison of dependency conversions for RST discourse trees
Katsuhiko Hayashi

Tsutomu Hirao

Masaaki Nagata
Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue
2015
pdf
bib
Hybrid Approach to PDTBstyled Discourse Parsing for CoNLL2015
Yasuhisa Yoshida

Katsuhiko Hayashi

Tsutomu Hirao

Masaaki Nagata
Proceedings of the Nineteenth Conference on Computational Natural Language Learning  Shared Task
pdf
bib
Discriminative Preordering Meets Kendall’s 𝜏 Maximization
Sho Hoshino

Yusuke Miyao

Katsuhito Sudoh

Katsuhiko Hayashi

Masaaki Nagata
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
2013
pdf
bib
ShiftReduce Word Reordering for Machine Translation
Katsuhiko Hayashi

Katsuhito Sudoh

Hajime Tsukada

Jun Suzuki

Masaaki Nagata
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
pdf
bib
abs
Efficient Stacked Dependency Parsing by Forest Reranking
Katsuhiko Hayashi

Shuhei Kondo

Yuji Matsumoto
Transactions of the Association for Computational Linguistics, Volume 1
This paper proposes a discriminative forest reranking algorithm for dependency parsing that can be seen as a form of efficient stacked parsing. A dynamic programming shiftreduce parser produces a packed derivation forest which is then scored by a discriminative reranker, using the 1best tree output by the shiftreduce parser as guide features in addition to thirdorder graphbased features. To improve efficiency and accuracy, this paper also proposes a novel shiftreduce parser that eliminates the spurious ambiguity of arcstandard transition systems. Testing on the English Penn Treebank data, forest reranking gave a stateoftheart unlabeled dependency accuracy of 93.12.
2012
pdf
bib
Headdriven Transitionbased Parsing with Topdown Prediction
Katsuhiko Hayashi

Taro Watanabe

Masayuki Asahara

Yuji Matsumoto
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2011
pdf
bib
Thirdorder Variational Reranking on PackedShared Dependency Forests
Katsuhiko Hayashi

Taro Watanabe

Masayuki Asahara

Yuji Matsumoto
Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing
2010
pdf
bib
Hierarchical Phrasebased Machine Translation with Wordbased Reordering Model
Katsuhiko Hayashi

Hajime Tsukada

Katsuhito Sudoh

Kevin Duh

Seiichi Yamamoto
Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)