Exploring the potential of DistilBERT architecture for automatic essay scoring task

Abstract

Automatic assessment of writing essays, or the process of using computers to evaluate and assign grades to written text, is very needed in the education system as an alternative to reduce human burden and time consumption, especially for large-scale tests. This task has received more attention in the last few years, being one of the major uses for natural language processing (NLP). Traditional automatic scoring systems typically rely on handcrafted features, whereas recent studies have used deep neural networks. Since the advent of transformers, pre-trained language models have performed well in many downstream tasks. We utilize the Kaggle benchmarking automated student assessment prize dataset to fine-tune the pre-trained DistilBERT in three different scenarios, and we compare results with the existing neural network-based approaches to achieve improved performance in the automatic essay scoring task. We utilize quadratic weighted Kappa (QWK) as the main metric to evaluate the performance of our proposed method. Results show that fine-tuning DistilBERT gives good results, especially with the scenario of training all parameters, which achieve 0.90 of QWK and outperform neural network models.