Chao Li


2020

pdf bib
How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention
Yue Guan | Jingwen Leng | Chao Li | Quan Chen | Minyi Guo
Proceedings of the 28th International Conference on Computational Linguistics

Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mechanism. As most of the research focus on probing tasks or hidden states, previous works have found some primitive patterns of attention head behavior by heuristic analytical methods, but a more systematic analysis specific on the attention patterns still remains primitive. In this work, we clearly cluster the attention heatmaps into significantly different patterns through unsupervised clustering on top of a set of proposed features, which corroborates with previous observations. We further study their corresponding functions through analytical study. In addition, our proposed features can be used to explain and calibrate different attention heads in Transformer models.

2019

pdf bib
Best Practices for Learning Domain-Specific Cross-Lingual Embeddings
Lena Shakurova | Beata Nyari | Chao Li | Mihai Rotaru
Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

Cross-lingual embeddings aim to represent words in multiple languages in a shared vector space by capturing semantic similarities across languages. They are a crucial component for scaling tasks to multiple languages by transferring knowledge from languages with rich resources to low-resource languages. A common approach to learning cross-lingual embeddings is to train monolingual embeddings separately for each language and learn a linear projection from the monolingual spaces into a shared space, where the mapping relies on a small seed dictionary. While there are high-quality generic seed dictionaries and pre-trained cross-lingual embeddings available for many language pairs, there is little research on how they perform on specialised tasks. In this paper, we investigate the best practices for constructing the seed dictionary for a specific domain. We evaluate the embeddings on the sequence labelling task of Curriculum Vitae parsing and show that the size of a bilingual dictionary, the frequency of the dictionary words in the domain corpora and the source of data (task-specific vs generic) influence performance. We also show that the less training data is available in the low-resource language, the more the construction of the bilingual dictionary matters, and demonstrate that some of the choices are crucial in the zero-shot transfer learning case.

2011

pdf bib
Automatically Generating Questions from Queries for Community-based Question Answering
Shiqi Zhao | Haifeng Wang | Chao Li | Ting Liu | Yi Guan
Proceedings of 5th International Joint Conference on Natural Language Processing

2010

pdf bib
Complete Syntactic Analysis Bases on Multi-level Chunking
Zhipeng Jiang | Yu Zhao | Yi Guan | Chao Li | Sheng Li
CIPS-SIGHAN Joint Conference on Chinese Language Processing