知识指导的预训练语言模型

知识指导的预训练语言模型

韩旭, 张正彦, 刘知远

（清华大学，中国 北京 100084）

摘要：作为典型的数据驱动方法，预训练语言模型仍然面临可解释性不强、鲁棒性差等难题。如何引入人类积累的丰富知识，是改进预训练模型性能的重要方向。系统介绍知识指导的预训练语言模型的最新进展与趋势，总结知识指导的预训练语言模型的典型范式，包括知识增强、知识支撑、知识约束和知识迁移，从输入、计算、训练、参数空间等多个角度阐释知识对于预训练语言模型的重要作用。  

关键词：自然语言处理；预训练语言模型；知识图谱  

Knowledge-Guided Pre-Trained Language Models

HAN Xu, ZHANG Zhengyan, LIU Zhiyuan

(Tsinghua University，Beijing 100084, China)

Abstract: As a typical data-driven method, pre-trained language models (PLMs) still face challenges such as poor interpretablility and robustness. Hence, it is important to introduce human knowledge into these models for better performance. The latest progress and trend of knowledge-guided PLMs are introduced and the framework of knowledge-guided PLMs is summarized, including knowledge augmentation, knowledge support, knowledge regularization, and knowledge transfer.

Keywords: natural language processing; PLM; knowledge graphs