Published on Mon Jun 24 2019

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Daniel Loureiro, Alipio Jorge

contextual embeddings can be used to achieve unprecedented gains in Word Sense Disambiguation (WSD) tasks. Our approach focuses on creating sense-level embeddings with full-coverage of WordNet, and without recourse to explicit knowledge of sense distributions or task-specific modelling.

0
0
0
Abstract

Contextual embeddings represent a new generation of semantic representations learned from Neural Language Modelling (NLM) that addresses the issue of meaning conflation hampering traditional word embeddings. In this work, we show that contextual embeddings can be used to achieve unprecedented gains in Word Sense Disambiguation (WSD) tasks. Our approach focuses on creating sense-level embeddings with full-coverage of WordNet, and without recourse to explicit knowledge of sense distributions or task-specific modelling. As a result, a simple Nearest Neighbors (k-NN) method using our representations is able to consistently surpass the performance of previous systems using powerful neural sequencing models. We also analyse the robustness of our approach when ignoring part-of-speech and lemma features, requiring disambiguation against the full sense inventory, and revealing shortcomings to be improved. Finally, we explore applications of our sense embeddings for concept-level analyses of contextual embeddings and their respective NLMs.

Wed May 26 2021
NLP
LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information. We introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants.
1
3
7
Mon Jan 25 2021
NLP
PolyLM: Learning about Polysemy through Language Modeling
0
0
0
Tue Nov 05 2019
NLP
Incremental Sense Weight Training for the Interpretation of Contextualized Word Embeddings
A novel online algorithm learns the essence of each dimension in word embeddings by minimizing the within-group distance. Three state-of-the-art neural-based language models are used, ELMo, and BERT, to generate contextualized word embeddeddings.
0
0
0
Thu Nov 19 2015
NLP
sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings
Neural word representations have proven useful in Natural Language Processing(NLP) tasks. Most techniques model only onerepresentation per word, despite the fact that a single word can have multiple meanings or "senses" This paper presents a novel approach that models multiple embeddings for each
1
0
0
Thu Jan 21 2021
NLP
Multi-sense embeddings through a word sense disambiguation process
Natural Language Understanding has seen an increasing number of publications in the last few years. Traditional models often fall short in intrinsic issues of linguistics, such as polysemy andhomonymy. To mitigate such issues, we propose a novel approach called Most Suitable Sense Annotation.
0
0
0
Sat Feb 27 2021
Machine Learning
EDS-MEMBED: Multi-sense embeddings based on enhanced distributional semantic structures via a graph walk over word senses
Multi-sense embeddings (M-SE) can be exploited for this important requirement. M-SE seeks to represent each word by their distinct senses in order to resolve the conflation of meanings of words as used in different contexts.
0
0
0