Exploring the potential of DistilBERT architecture for automatic essay scoring task
Abstract
Automatic assessment of writing essays, or the process of using computers to evaluate and assign grades to written text, is very needed in the education system as an alternative to reduce human burden and time consumption, especially for large-scale tests. This task has received more attention in the last few years, being one of the major uses for natural language processing (NLP). Traditional automatic scoring systems typically rely on handcrafted features, whereas recent studies have used deep neural networks. Since the advent of transformers, pre-trained language models have performed well in many downstream tasks. We utilize the Kaggle benchmarking automated student assessment prize dataset to fine-tune the pre-trained DistilBERT in three different scenarios, and we compare results with the existing neural network-based approaches to achieve improved performance in the automatic essay scoring task. We utilize quadratic weighted Kappa (QWK) as the main metric to evaluate the performance of our proposed method. Results show that fine-tuning DistilBERT gives good results, especially with the scenario of training all parameters, which achieve 0.90 of QWK and outperform neural network models.
Keywords
Automatic essay scoring; DistilBERT; Fine-tuning; Natural language processing; Transformers
Full Text:
PDFDOI: http://doi.org/10.11591/ijeecs.v36.i2.pp1234-1241
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Indonesian Journal of Electrical Engineering and Computer Science (IJEECS)
p-ISSN: 2502-4752, e-ISSN: 2502-4760
This journal is published by the Institute of Advanced Engineering and Science (IAES) in collaboration with Intelektual Pustaka Media Utama (IPMU).