Published on Fri Aug 16 2019

Learning Conceptual-Contextual Embeddings for Medical Text

Xiao Zhang, Dejing Dou, Ji Wu

We introduce a contextual text representation model called Conceptual-Contextual (CC) embeddings. Unlike entity embedding methods, our approach encodes a knowledge graph into a context model. Our model effectively encodes the huge UMLS database by leveraging semantic generalizability.

0
0
0
Abstract

External knowledge is often useful for natural language understanding tasks. We introduce a contextual text representation model called Conceptual-Contextual (CC) embeddings, which incorporates structured knowledge into text representations. Unlike entity embedding methods, our approach encodes a knowledge graph into a context model. CC embeddings can be easily reused for a wide range of tasks just like pre-trained language models. Our model effectively encodes the huge UMLS database by leveraging semantic generalizability. Experiments on electronic health records (EHRs) and medical text processing benchmarks showed our model gives a major boost to the performance of supervised medical NLP tasks.

Tue Oct 20 2020
Artificial Intelligence
UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus
UmlsBERT can encode clinical domain knowledge into word embeddings. It outperforms existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference clinical NLP tasks.
0
0
0
Fri Apr 17 2020
Machine Learning
Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
Word embeddings have been shown to capture the semantics of words based on their distributional properties. Embeddings of knowledge bases (KB) capture the explicit relations between entities denoted by words, but are not able to directly capture the syntagmatic properties of these words.
0
0
0
Fri Feb 22 2019
NLP
Enhancing Clinical Concept Extraction with Contextual Embeddings
Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks. There are no common best practices for how to integrate theserepresentations into clinical tasks. We demonstrate the potential of contextual embeddings through the state-of-the-art performance they achieve on clinical concept extraction.
0
0
0
Wed Apr 04 2018
Artificial Intelligence
Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data
Word embeddings are a popular approach to unsupervised learning of word relationships. In this article, we present a new set of embeddings for medical concepts learned using an extremely large collection of multimodal medical data. Our approach, called cui2vec, attains state-
0
0
0
Wed Apr 03 2019
NLP
Probing Biomedical Embeddings from Language Models
Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such as biomedical articles, further improves theirperformance.
1
3
1
Tue Oct 01 2019
Artificial Intelligence
Writing habits and telltale neighbors: analyzing clinical concept usage patterns with sublanguage embeddings
We present a method for characterizing the usage patterns of clinical concepts among different document types. We are able to measure divergences in concept usage while correcting for noise in embedding learning. Experiments on the MIMIC-III corpus demonstrate that our approach captures clinically-relevant differences.
0
0
0