Educational impact and ethical considerations in using Chatbots in Academia

Abstract

Chatbots are getting better every day due to the advancements in their capabilities in today’s technological age. This study aims to assess the efficacy of ChatGPT-4 and Gemini in producing scientific articles. Two types of prompts are given: direct questions and complete scenarios. Subsequently, we evaluate the educational and ethical aspects of the produced material by employing statistical analysis. We verify the credibility of references, detect any instances of plagiarism, and ensure the precision of the articles generated by the chatbot. In addition, we utilize topic modeling to assess the extent to which the content of the articles corresponds to the specified topic. According to the findings, Gemini outperformed ChatGPT-4, specifically in scenario questions, where it achieved an accuracy rate of 85%, while ChatGPT-4 only achieved 35% accuracy.