사전 학습된 한국어 BERT의 전이학습을 통한 한국어 기계독해 성능개선에 관한 연구

• 영문명: A Study of Fine Tuning Pre-Trained Korean BERT for Question Answering Performance Development
• 발행기관: 한국IT서비스학회
• 저자명: 이치훈(Chi Hoon Lee) 이연지(Yeon Ji Lee) 이동희(Dong Hee Lee)
• 간행물 정보: 『한국IT서비스학회지』한국IT서비스학회지 제19권 제5호, 83~91쪽, 전체 9쪽
• 주제분류: 경제경영 > 경영학
• 파일형태: PDF
• 발행일자: 2020.10.30

영문 초록

Language Models such as BERT has been an important factor of deep learning-based natural language processing. Pre-training the transformer-based language models would be computationally expensive since they are consist of deep and broad architecture and layers using an attention mechanism and also require huge amount of data to train. Hence, it became mandatory to do fine-tuning large pre-trained language models which are trained by Google or some companies can afford the resources and cost. There are various techniques for fine tuning the language models and this paper examines three techniques, which are data augmentation, tuning the hyper paramters and partly re-constructing the neural networks. For data augmentation, we use no-answer augmentation and back-translation method. Also, some useful combinations of hyper parameters are observed by conducting a number of experiments. Finally, we have GRU, LSTM networks to boost our model performance with adding those networks to BERT pre-trained model. We do fine-tuning the pre-trained korean-based language model through the methods mentioned above and push the F1 score from baseline up to 89.66. Moreover, some failure attempts give us important lessons and tell us the further direction in a good way.

목차

1. 서 론

2. 관련 연구

3. 이론적 배경

4. 실험 및 결과

5. 결 론

참고문헌

키워드

Language Model Masked Language Model Question Answering BERT Fine-Tuning