（英特尔（中国）有限公司，中国 北京 100013）
Development of Natural Language Processing Technology
(Intel China Ltd., Beijing 100013, China)
Abstract: The pre-trained language model based on neural network and deep learning has brought breakthrough development for natural language processing technology. The Transformer model based on self-attention mechanism is the basis of the pre-trained language model. Large-scale pre-trained language models such as GPT, BERT, XLNet, etc. are based on the Transformer model or its optimization. However, the current large-scale pre-training language models that rely on powerful computing resources and massive data have practical problems. It is pointed out that lightweight pre-trained language models are an important development direction in the future.
Keywords: natural language processing; pre-trained language model; Transformer; GPT; BERT; XLNet; model optimization