您当前访问的的浏览器版本过低,为了给您带来更好的体验,建议您升级至Edge浏览器或者推荐使用Google浏览器
取消
知识增强预训练模型
发布时间:2022-04-08  作者:王海峰,孙宇,吴华  阅读量:

知识增强预训练模型

王海峰, 孙宇, 吴华
(北京百度网讯科技有限公司,中国 北京 100193)

摘要:预训练模型主要从海量未标注、无结构化的数据中学习,但缺少外部知识指导,存在模型学习效率不高、模型效果不佳和知识推理能力受限等不足。如何在预训练模型中引入语言知识、世界知识等外部知识,提升模型效果以及知识记忆和推理能力是一个难题。本文从不同类型知识的引入、融合知识的方法、缓解知识遗忘的方法等角度,介绍知识增强预训练模型的发展,并以知识增强预训练模型百度文心为例,详细探讨知识增强预训练模型的原理和应用。  
关键词:自然语言处理;预训练模型;知识增强 


Knowledge-Enhanced Pre-Trained Models

WANG Haifeng, SUN Yu, WU Hua
(Beijing Baidu Netcom Science and Technology Co., Ltd., Beijing 100193, China)

Abstract: Pre-trained models can automatically learn from massive data without any manual labels. Nevertheless, the lack of guidance from external knowledge has dramatically hindered the learning efficiency and reasoning capacity. There are still challenges in incorporating external supervision such as linguistics and world knowledge to improve pre-trained models’ ability of knowledge memorization and reasoning. This paper provides a comprehensive review of knowledge-enhanced pre-trained models from various perspectives, such as multi-source knowledge incorporation, knowledge fusion, and knowledge forgetting alleviation. Here we take Baidu ERNIE as an example to describe the principles and applications of knowledge-enhanced pre-trained models.
Keywords: natural language processing; pre-trained model; knowledge enhanced

在线PDF浏览: PDF
本期相关文章