Development of a machine learning model with optuna and ensemble learning to improve performance on multiple datasets

Abstract

Machine learning, a subset of artificial intelligence (AI) is vital for its ability to learn from data and improve system performance. In Indonesia, advancements in ML have significant potential to boost competitiveness and foster sustainable development. However, issues like overfitting and suboptimal parameter settings can hinder model effectiveness. This study aims to improve the classification performance of ML models on various datasets. Advanced techniques like hyperparameter tuning with Optuna and ensemble learning with extreme gradient boosting (XGBoost) are integrated to enhance model performance. The study evaluates the performance of K nearest neighbors (KNN), support vector machine (SVM), and Gaussian naïve Bayes (GNB) algorithms across three datasets: academic records from the Islamic University of Riau (UIR), diabetes data from Kaggle, and Twitter data related to the 2024 elections. The findings reveal that the GNB algorithm outperforms KNN and SVM across all datasets, achieving the highest accuracy, precision, recall, and F1-score. Hyperparameter tuning with Optuna significantly improves model performance, demonstrating the value of systematic optimization. This study highlights the importance of advanced optimization techniques in developing high-performing ML models. The results suggest that robust algorithms like GNB, combined with hyperparameter tuning and ensemble learning, can significantly enhance classification performance.