OpenAI ChatGPT vs Google Gemini: A study of AI chatbots’ writing quality evaluation and plagiarism checking
DOI:
https://doi.org/10.12928/eltej.v7i2.11572Keywords:
AI Chatbot, L2 Writing, Plagiarism, Writing Evaluation, Google Gemini, OpenAI ChatGPTAbstract
This study explores the writing quality of two AI chatbots, OpenAI ChatGPT and Google Gemini. The research assesses the quality of the generated texts based on five essay models using the T.E.R.A. software, focusing on ease of understanding, readability, and reading levels using the Flesch-Kincaid formula. Thirty essays were generated, 15 from each chatbot, and evaluated for plagiarism using two free detection tools—SmallSEOTools and Check-Plagiarism—as well as one paid tool, Turnitin. The findings revealed that both ChatGPT and Gemini performed well in terms of word concreteness but demonstrated weaknesses in narrativity. ChatGPT showed stronger performance in referential and deep cohesion, while Gemini excelled in narrativity, syntactic simplicity and word concreteness. However, a significant concern was the degree of plagiarism detected in texts from both AI tools, with ChatGPT's essays exhibiting a higher likelihood of plagiarism compared to Gemini’s. These findings highlight the potential limitations and risks associated with using AI-generated writing.
References
Ariyanti, A. (2016). Shaping students’ writing skills: The study of fundamental aspects in mastering academic writing. Indonesian Journal of EFL and Linguistics, 1(1), 63-77. Retrieved from http://www.indonesian-efl-journal.org
Aydın, Ö., & Karaarslan, E. (2022). OpenAI ChatGPT generated literature review: Digital twin in healthcare. Emerging Computer Technologies, 2, 22-31. İzmir Akademi Dernegi. http://dx.doi.org/10.2139/ssrn.4308687
Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52-62. https://doi.org/10.61969/jai.1337500
Barrot, J. S. (2023). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(6), 584–607. https://doi.org/10.1080/09588221.2021.1936071
Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/QRJ0902027
Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20, 43. https://doi.org/10.1186/s41239-023-00411-8
Check Plagiarism. (2024). Check plagiarism: Free online plagiarism checker. https://www.check-plagiarism.com/
Chompurach, W. (2021). “Please let me use Google Translate”: Thai EFL students’ behavior and attitudes toward Google Translate use in English writing. English Language Teaching, 14(12), 23-35. https://doi.org/10.5539/elt.v14n12p23
Davis, D. F., Golicic, S. L., & Boerstler, C. N. (2011). Benefits and challenges of conducting multiple methods research in marketing. Journal of the Academy of Marketing Science, 39, 467–479. https://doi.org/10.1007/s11747-010-0204-7
Espejel et al. (2023). GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Processing Journal, 5, 100032. https://doi.org/10.1016/j.nlp.2023.100032
Evans et al. (2023). ChatGPT impacts on access-efficiency, employment, education and ethics: The socio-economics of an AI language model. BizEcons Quarterly, 16, 1-17.
Fitria, T. N. (2023). Artificial intelligence (AI) technology in OpenAI ChatGPT application: A review of ChatGPT in writing English essay. ELT Forum: Journal of English Language Teaching, 12(1), 44-58. https://doi.org/10.15294/elt.v12i1.64069
Hasnawati, H., Mujahidin, E., & Tanjung, H. (2023). Analyzing students’ difficulties in writing English essay. International Journal of Social Science and Human Research, 6(10), 5954-5959. https://doi.org/10.47191/ijsshr/v6-i10-22
Hyland, K. (2003). Second language writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511667251
Hyland, K., & Jiang, F. K. (2017). Is academic writing becoming more informal? English for Specific Purposes, 45, 40-51. https://doi.org/10.1016/j.esp.2016.09.001
Jovanovic, M. (2022). Generative artificial intelligence: Trends and prospects. Computer, 55(10), 107-112. https://doi.org/10.1109/MC.2022.3192720
Khalil, M., & Er, E. (2023). Will ChatGPT get you caught? Rethinking of plagiarism detection. In Proceedings of the 25th International Conference on Human-Computer Interaction (HCII 2023) (Vol. 14040, pp. 475–487). https://arxiv.org/pdf/2302.04335
Lee, G. G., Shi, L., Latif, E., Gao, Y., Bewersdorf, A., Nyaaba, M., et al. (2023). Multimodality of AI for education: Towards artificial general intelligence. arXiv. https://arxiv.org/abs/2312.06037
Lee, H., & Lee, J. H. (2022). The effects of robot-assisted language learning: A meta-analysis. Educational Research Review, 35, 100425. https://doi.org/10.1016/j.edurev.2021.100425
Lee, J. & Park, J. (2023). AI as “Another I”: Journey map of working with artificial intelligence from AI-phobia to AI-preparedness. Organizational Dynamics, 52, 100994. https://doi.org/10.1016/j.orgdyn.2023.100994
Lingard, L. (2023). Writing with ChatGPT: It illustrates its capacity, limitations & implications for academic writers. Perspectives on Medical Education, 12(1), 261–270. https://doi.org/10.5334/pme.1072
McCarthy, J., Minsky, M., Rochester, N., & Shannon, C. (1955). A proposal for Dartmouth summer research project on artificial intelligence. AI Magazine, 27, 12.
McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press. https://doi.org/10.1017/CBO9780511894664
Meiranti, R. (2012). Improving students’ writing skills through field trip method. English Review: Journal of English Education, 1(1), 89-96.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2018). Qualitative data analysis: A methods sourcebook (4th ed.). SAGE Publications.
Mohammed, A., Al-ghazali, A. & Alqohfa, K. (2023). Exploring ChatGPT uses in higher studies: A case study of Arab postgraduates in India. Journal of English Studies in Arabia Felix, 2(2), 8–15. https://doi.org/10.56540/jesaf.v2i2.55
OpenAI. (2022). ChatGPT: Optimizing language models for dialogue. OpenAI. https://openai.com/blog/chatgpt/ (Accessed November 29, 2023)
Rane, N., Choudhary, S., & Rane, J. (2024). Gemini or ChatGPT? Capability, performance, and selection of cutting-edge generative artificial intelligence (AI) in business management. Studies in Economics and Business Relations, 5(1), 40-50. https://doi.org/10.48185/sebr.v5i1.1051
Rayhan, A. (2023). Mastering prompt engineering: Techniques for creating powerful and effective AI language models. Independently Published.
Seyyedrezaei, M. S., Amiryousefi, M., Gimeno-Sanz, A., & Tavakoli, M. (2022). A meta-analysis of the relative effectiveness of technology-enhanced language learning on ESL/EFL writing performance: Retrospect and prospect. Computer Assisted Language Learning, 1–34. https://doi.org/10.1080/09588221.2022.218782
Small SEO Tools. (n.d.). Small SEO tools: Text analysis tools. Small SEO tools. https://smallseotools.com/
Stahl, B. C. & Eke, D. (2023). The ethics of ChatGPT – Exploring the ethical issues of an emerging technology. International Journal of Information Management, 74, 102700. https://doi.org/10.1016/j.ijinfomgt.2023.102700
Suaverdez, J., & Suaverdez, U. (2023). Chatbots impact on academic writing. Global Journal of Business and Integral Security, (2). Retrieved from https://www.gbis.ch/index.php/gbis/article/view/150
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Turnitin. (n.d.). Turnitin: Empower students to do their best, original work. Turnitin. https://www.turnitin.com/
Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28, 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
Zhai, X. (2023). ChatGPT user experience: Implications for education. SSRN Electronic Journal. https://dx.doi.org/10.2139/ssrn.4312418
Zhang, S., Shan, C., Lee, J. S. Y., & et al. (2023). Effect of chatbot-assisted language learning: A meta-analysis. Education and Information Technologies, 28, 15223–15243. https://doi.org/10.1007/s10639-023-11805-6
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Siraprapa Kotmungkun, Wichuta Chompurach, Piriya Thaksanan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish in ELTEJ agree to the following terms: Authors retain copyright and grant the ELTEJ right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material) the work for any purpose, even commercially with an acknowledgement of the work's authorship and initial publication in ELTEJ. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in ELTEJ. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
