A comparative review on deep learning models for text classification

Muhammad Zulqarnain, Rozaida Ghazali, Yana Mazwin Mohmad Hassim, Muhammad Rehan


Text classification is a fundamental task in several areas of natural language processing (NLP), including words semantic classification, sentiment analysis, question answering, or dialog management. This paper investigates three basic architectures of deep learning models for the tasks of text classification: Deep Belief Neural (DBN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), these three main types of deep learning architectures, are largely explored to handled various classification tasks. DBN have excellent learning capabilities to extracts highly distinguishable features and good for general purpose. CNN have supposed to be better at extracting the position of various related features while RNN is modeling in sequential of long-term dependencies. This paper work shows the systematic comparison of DBN, CNN, and RNN on text classification tasks. Finally, we show the results of deep models by research experiment. The aim of this paper to provides basic guidance about the deep learning models that which models are best for the task of text classification.


Deep Learning; CNN; RNN; DBN; Text Classification


Elman J. Finding structure in time* 1. Cogn Sci [Internet]. 1990;211(1 990):1–28. Available from: http://linkinghub.elsevier.com/retrieve/pii/036402139090002E

Tzortzis G, Likas A. Deep Belief Networks for Spam Filtering. 19th IEEE Int Conf Tools with Artif Intell 2007) [Internet]. 2007;306–9. Available from: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4410396

Chung J, Gulcehre C, Cho K, Bengio Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. 2014;1–9. Available from: http://arxiv.org/abs/1412.3555

Graves A. Generating Sequences With Recurrent Neural Networks. arXiv [Internet]. 2014;1–43. Available from: http://arxiv.org/abs/1308.0850

Liu T. A Novel Text Classification Approach Based on Deep Belief Network. springer [Internet]. 2010;314–21. Available from: http://link.springer.com/10.1007/978-3-642-17537-4_39

Vu NT, Adel H, Gupta P, Schütze H. Combining Recurrent and Convolutional Neural Networks for Relation Classification. 2016; Available from: http://arxiv.org/abs/1605.07333

Silva J, Coheur L, Mendes AC, Wichert A. From symbolic to sub-symbolic information in question classification. Artif Intell Rev. 2011;35(2):137–54.

Nakagawa T, Inui K, Kurohashi S. Dependency tree-based sentiment classification using CRFs with hidden variables. Proceeding HLT ’10 Hum Lang Technol 2010 Annu Conf North Am Chapter Assoc Comput Linguist. 2015;(June):786–94.

Zulqarnain M, Ghazali R, Mazwin Y, Saedi AKZ Al, Akram U. A comprehensive review on text classification using deep learning. J Phys Conf Ser. 2018;(29):1–7.

Song J, Qin S, Zhang P. Chinese Text Categorization Based on Deep Belief Networks. IEEE ICIS 2016. 2016;(June):1–5.

Vincent P. A Neural Probabilistic Language Model. 2003;3:1137–55.

Dey R, Salemt FM. Gate-variants of Gated Recurrent Unit (GRU) neural networks. Midwest Symp Circuits Syst. 2017;2017–Augus:1597–600.

Collobert R, Weston J. A unified architecture for natural language processing. Proc 25th Int Conf Mach Learn - ICML ’08 [Internet]. 2008;160–7. Available from: http://portal.acm.org/citation.cfm?doid=1390156.1390177

Prieto LP, Rodríguez-Triana MJ, Kusmin M, Laanpere M. Smart school multimodal dataset and challenges. CEUR Workshop Proc. 2016;1828:53–9.

Lai S, Xu L, Liu K, Zhao J. Recurrent Convolutional Neural Networks for Text Classification. Twenty-Ninth AAAI Conf Artif Intell. 2015;2267–73.

Aloysius N, Geetha M. A review on deep convolutional neural networks. 2017 Int Conf Commun Signal Process [Internet]. 2017;0588–92. Available from: http://ieeexplore.ieee.org/document/8286426/

arkhipenko AK, kozlov-ilya KI, integral TJ, kirillskorniakov SK, gomzin GA, turdakov TD. Comparison of neural network arChiteCtures for sentiment analysis of russian tweets. Dialogue Can Philos Assoc. 2016;

Mikolov T, Chen K, Corrado G, Dean J. 5021-Distributed-Representations-of-Words-and-Phrases-and-Their-Compositionality. ICML. 2013;1–9.

Dosovitskiy A, Springenberg JT, Brox T. An Empirical Exploration of Recurrent Network Architectures Rafal. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2015;07–12–June:1538–46.

Zeiler MD, Fergus R. Stochastic Pooling for Regularization of Deep Convolutional Neural Networks. 2013;1–9. Available from: http://arxiv.org/abs/1301.3557

Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High Confidence Predictionsfor Unrecognizable Images. Comput Vis Pattern Recognit [Internet]. 2015; Available from: https://scout.ai/story/the-rise-of-the-weaponized-ai-propaganda-machine

Rippel O, Snoek J, Adams RP. Spectral Representations for Convolutional Neural Networks. 2015;1–9. Available from: http://arxiv.org/abs/1506.03767

Gong Y, Wang L, Guo R, Lazebnik S. Multi-scale orderless pooling of deep convolutional activation features. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2014;8695 LNCS(PART 7):392–407.

Hochreiter S. 2 Previous Work. Neural Comput. 1997;9(8):1–32.

Socher R, Perelygin A, Wu J. Recursive deep models for semantic compositionality over a sentiment treebank. Proc … [Internet]. 2013;(October):1631–42. Available from: http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf%5Cnhttp://www.aclweb.org/anthology/D13-1170%5Cnhttp://aclweb.org/supplementals/D/D13/D13-1170.Attachment.pdf%5Cnhttp://oldsite.aclweb.org/anthology-new/D/D13/D13-1170.pdf

Kalchbrenner N, Grefenstette E, Blunsom P. A Convolutional Neural Network for Modelling Sentences. Proc 52nd Annu Meet Assoc Comput Linguist (Volume 1 Long Pap [Internet]. 2014;655–65. Available from: http://aclweb.org/anthology/P14-1062

Hendrickx I, Kim SN, Kozareva Z, Nakov P, Ó Séaghdha D, Padó S, et al. SemEval-2010 Task 8 : Multi-Way Classification of Semantic Relations Between Pairs of Nominals. Comput Linguist [Internet]. 2010;(June 2009):94–9. Available from: http://www.aclweb.org/anthology/S10-1006

Bowman SR, Angeli G, Potts C, Manning CD. A large annotated corpus for learning natural language inference. 2015; Available from: http://arxiv.org/abs/1508.05326

Sharif W, Samsudin NA, Deris MM, Aamir M. Improved relative discriminative criterion feature ranking technique for text classification. Int J Artif Intell. 2017;15(2):61–78.

Nam J, Kim J, Loza Mencía E, Gurevych I, Fürnkranz J. Large-scale multi-label text classification - Revisiting neural networks. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2014;8725 LNAI(PART 2):437–52.

DOI: http://doi.org/10.11591/ijeecs.v19.i1.pp%25p
Total views : 37 times


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

shopify stats IJEECS visitor statistics