您当前访问的的浏览器版本过低,为了给您带来更好的体验,建议您升级至Edge浏览器或者推荐使用Google浏览器
取消
自然语言处理技术发展
发布时间:2022-04-08  作者:王海宁  阅读量:

自然语言处理技术发展

王海宁
(英特尔(中国)有限公司,中国 北京 100013

摘要:基于神经网络和深度学习的预训练语言模型为自然语言处理技术带来了突破性发展。基于自注意力机制的Transformer模型是预训练语言模型的基础。GPT、BERT、XLNet等大规模预训练语言模型均基于Transformer模型进行堆叠和优化。认为目前依赖强大算力和海量数据的大规模预训练语言模型存在实用问题,指出轻量预训练语言模型是未来重要的发展方向。  
关键词:自然语言处理;预训练语言模型;Transformer;GPT;BERT;XLNet;模型优化


Development of Natural Language Processing Technology

WANG Haining
(Intel China Ltd., Beijing 100013, China)

Abstract: The pre-trained language model based on neural network and deep learning has brought breakthrough development for natural language processing technology. The Transformer model based on self-attention mechanism is the basis of the pre-trained language model. Large-scale pre-trained language models such as GPT, BERT, XLNet, etc. are based on the Transformer model or its optimization. However, the current large-scale pre-training language models that rely on powerful computing resources and massive data have practical problems. It is pointed out that lightweight pre-trained language models are an important development direction in the future.
Keywords: natural language processing; pre-trained language model; Transformer; GPT; BERT; XLNet; model optimization

在线PDF浏览: PDF
本期相关文章