A data-driven analysis to determine the optimal number of topics 'K' for latent Dirichlet allocation model

Astha Goyal, Indu Kashyap


Topic modeling is an unsupervised machine learning technique successfully used to classify and retrieve textual data. However, the performance of topic models is sensitive to selecting optimal hyperparameters, the number of topics 'K' and Dirichlet priors 'α' and 'β.' This data-driven analysis aims to determine the optimum number of topics, 'K,' within the latent Dirichlet allocation (LDA) model. This work utilizes three datasets, namely 20-Newsgroups news articles, Wikipedia articles, and Web of Science containing science articles, to assess and compare various 'K' values through the grid search approach. The grid search approach finds the best combination of hyperparameter values by trying all possible combinations to see which performs best. This research seeks to identify the 'K' that optimizes topic relevance, coherence, and model performance by leveraging statistical metrics, such as coherence scores, perplexity, and topic distribution quality. Through empirical analysis and rigorous evaluation, this work provides valuable insights for determining the ideal 'K' for LDA models.


Grid search approach; Latent dirichlet allocation; LDA hyperparameter tuning; Objective functions; Optimal number of topics; Topic modeling

Full Text:


DOI: http://doi.org/10.11591/ijeecs.v35.i1.pp310-322


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The Indonesian Journal of Electrical Engineering and Computer Science (IJEECS)
p-ISSN: 2502-4752, e-ISSN: 2502-4760
This journal is published by the Institute of Advanced Engineering and Science (IAES) in collaboration with Intelektual Pustaka Media Utama (IPMU).

shopify stats IJEECS visitor statistics