Published on Fri Apr 09 2021

KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding

Keyur Faldu, Amit Sheth, Prashant Kikani, Hemang Akabari

KI-BERT-base model even outperforms BERT-large for domain-specific tasks like SciTail and academic subsets of QQP, QNLI, and MNLI.

3
0
2
Abstract

Contextualized entity representations learned by state-of-the-art deep learning models (BERT, GPT, T5, etc) leverage the attention mechanism to learn the data context. However, these models are still blind to leverage the knowledge context present in the knowledge graph. Knowledge context can be understood as semantics about entities, and their relationship with neighboring entities in knowledge graphs. We propose a novel and effective technique to infuse knowledge context from knowledge graphs for conceptual and ambiguous entities into models based on transformer architecture. Our novel technique project knowledge graph embedding in the homogeneous vector-space, introduces new token-types for entities, align entity position ids, and a selective attention mechanism. We take BERT as a baseline model and implement "KnowledgeInfused BERT" by infusing knowledge context from ConceptNet and WordNet, which significantly outperforms BERT over a wide range of NLP tasks over eight different GLUE datasets. KI-BERT-base model even outperforms BERT-large for domain-specific tasks like SciTail and academic subsets of QQP, QNLI, and MNLI.

Thu Oct 01 2020
Artificial Intelligence
CoLAKE: Contextualized Language and Knowledge Embedding
Contextualized Language and Knowledge Embedding (CoLAKE) learns contextualized representation for both language and knowledge. CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. Experimental results show that CoLAkeoutperforms previous counterparts
0
0
0
Mon Oct 12 2020
Artificial Intelligence
On the Complementary Nature of Knowledge Graph Embedding, Fine Grain Entity Types, and Language Modeling
We demonstrate the complementary natures of neural knowledge graph embedding and neural language modeling. We show that jointly modeling both structured knowledge tuples and language improves both.
0
0
0
Sat Sep 19 2020
Artificial Intelligence
Inductive Learning on Commonsense Knowledge Graph Completion
Commonsense knowledge graph (CKG) is a special type of knowledge graph where entities are composed of free-form text. Most existing CKG completion methods focus on the setting where all the entities are presented at training time. At test time, entities in CKGs can be unseen because they may have unseen text/names.
0
0
0
Wed Aug 12 2020
Artificial Intelligence
Evaluating the Impact of Knowledge Graph Context on Entity Disambiguation Models
Pretrained Transformer models have emerged as state-of-the-art approaches that learn contextual information from text. These models, albeit powerful, still require specialized knowledge in specific scenarios. We argue that context derived from a knowledge graph provides enough signals toform pretrained transformer models.
0
0
0
Fri Feb 16 2018
NLP
Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing
Machine Learning has been the quintessential solution for many AI problems. But learning is still heavily dependent on the specific training data. We propose to enhance learning models with world knowledge in the form of Knowledge Graph (KG) fact triples for Natural Language Processing tasks.
0
0
0
Wed Nov 13 2019
NLP
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In KEPLER, we code textual entity descriptions with a PLM as their embeddings, and then optimize the KE and language modeling objectives.
0
0
0