Effectiveness of Artificial Intelligence on Students’ Critical Thinking Skills: A Meta-Analysis Study

Authors

  • Santa Aulia Devi Rachmadhani Universitas Negeri Yogyakarta
  • Ali Muhson Universitas Negeri Yogyakarta
  • Risky Setiawan Universitas Negeri Yogyakarta

DOI:

https://doi.org/10.17977/um038v9i42026p499-511

Keywords:

Artificial Intelligence, Critical Thinking, Meta-analysis, Moderator Analysis, Effect Size

Abstract

Empirical evidence on the effect of artificial intelligence (AI) on students’ critical thinking skills remains fragmented and has not been systematically synthesized. This study aims to quantify the effect of AI-based interventions on critical thinking skills, test for publication bias, and examine four moderators: region, education level, treatment duration, and type of AI used. The analysis was limited to quasi-experimental studies with a control group. A meta-analysis was conducted on studies retrieved from Scopus, ScienceDirect, and ERIC, following the PRISMA guidelines. Effect sizes were calculated using Hedges’ g under a random-effects model. The results show a significant large effect with high heterogeneity, and no evidence of publication bias was found. Moderator analyses revealed significant differences in effect size across all four factors. The largest effects were observed in studies conducted in Africa, at the higher education level, with long-duration interventions, and with mixed AI use. Overall, AI-based interventions tend to improve students’ critical thinking skills. However, the magnitude of the effect depends on how the intervention is designed and implemented. Therefore, AI integration in learning should be purposefully structured according to pedagogical frameworks and supported by sufficiently long interventions. These findings strengthen the empirical evidence on the relationship between AI and critical thinking.

References

Abbas, R. M., Nser, H., & Farid, S. (2025). Exploring the Influence of AI-Generated Content on Critical Thinking Skills of University Students. Policy Research Journal, 3(5), 690–698. https://doi.org/10.5281/zenodo.15550335

Aladini, A., Ismail, S. M., Ahmad Saleem Khasawneh, M., & Shakibaei, G. (2025). Self-directed writing development across computer/AI-based tasks: Unraveling the traces on L2 writing outcomes, growth mindfulness, and grammatical knowledge. Computers in Human Behavior Reports, 17(December 2024), 100566. https://doi.org/10.1016/j.chbr.2024.100566

Alghamdi, L. H., & Alghizzi, T. M. (2025). Educators’ reflections on AI-automated feedback in higher education: a structured integrative review of potentials, pitfalls, and ethical dimensions. Frontiers in Education, 10(November), 1–17. https://doi.org/10.3389/feduc.2025.1704820

Andrian, R. D., Widianti, Y. M., & Wati, D. A. R. (2025). The Role of Artificial Intelligence in Enhancing Critical Thinking Skills of Educational Technology Students: The Moderating Influence of Digital Literacy and Usage Regulation. Journal on Smart Learning Technologies, 1(2), 109–120.

Arimbawa, G. P. A., Parwati, N. N., Agustini, K., Suartama, I. K., & Sudarma, I. K. (2025). Effectiveness Of Microlearning-Based E-Modules In Improving Critical Thinking Abilities Of Vocational High School Students. JKTP: Jurnal Kajian Teknologi Pendidikan, 8(3), 235–247. https://doi.org/10.17977/um038v8i32025p235-247

Ayanwale, M. A., & Omeh, C. B. (2026). AI-supported problem-based learning for enhancing computational thinking skills in STEM education. Computers in Human Behavior: Artificial Humans, 7(September 2025), 100263. https://doi.org/10.1016/j.chbah.2026.100263

Baroud, S., Alaoui, M. El, Idrissi, R. J., Boucetta, N., Ghariz, G., & Seghir, H. (2025). Improving biology learning in high school with ChatGPT: A case study in Morocco. Eurasia Journal of Mathematics, Science and Technology Education, 21(7). https://doi.org/10.29333/ejmste/16606

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2009a). Introduction to Meta-Analysis. John Wiley & Sons.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009b). Introduction to Meta-Analysis. In A Guide to the Scientific Career: Virtues, Communication, Research, and Academic Writing. https://doi.org/10.1002/9781118907283.ch68

Brazão, P., & Tinoca, L. (2025). Artificial intelligence and critical thinking: a case study with educational chatbots. Frontiers in Education, 10(September), 1–12. https://doi.org/10.3389/feduc.2025.1630493

Bueno-Picazo, M., & Tirado-Olivares, S. (2026). Generative AI in Geography Education: Content Creation and Conversational AI-Supported Learning to Promote Environmental Awareness. European Journal of Geography, 17(2), S.34-S.48. https://doi.org/10.48088/ejg.m.bue.17.2.034.048

Cai, J., Tan, Y. H., & Lin, S. E. (2026). Rethinking literature education: A quasi-experimental study of the production-oriented approach for english majors in China. Acta Psychologica, 266(March), 106808. https://doi.org/10.1016/j.actpsy.2026.106808

Cao, Y., & Gao, Q. (2026). Pedagogically Customized Machine Translation for Cultivating Critical Thinking in Foreign Language Education. Journal of Cases on Information Technology, 28(1), 1–17. https://doi.org/10.4018/JCIT.401921

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155

Dai, Y., Lin, Z., Liu, A., Dai, D., & Wang, W. (2024). Effect of an Analogy-Based Approach of Artificial Intelligence Pedagogy in Upper Primary Schools. Journal of Educational Computing Research, 61(8), 159–186. https://doi.org/10.1177/07356331231201342

Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629

Eltahir, M. E., & Babiker, F. M. E. (2024). The Influence of Artificial Intelligence Tools on Student Performance in e-Learning Environments: Case Study. Electronic Journal of E-Learning, 22(9), 91–110. https://doi.org/10.34190/ejel.22.9.3639

Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024). ChatGPT effects on cognitive skills of undergraduate students: Receiving instant responses from AI-based conversational large language models (LLMs). Computers and Education: Artificial Intelligence, 6(December 2023), 100198. https://doi.org/10.1016/j.caeai.2023.100198

Facione, P. A. (2011). Critical thinking: what it is and why it counts. Insight Assessment, 5(1), 1–23. https://www.researchgate.net/profile/Peter_Facione/publication/251303244_Critical_Thinking_What_It_Is_and_Why_It_Counts/links/5849b49608aed5252bcbe531/Critical-Thinking-What-It-Is-and-Why-It-Counts.pdf

Gao, J., Zhang, J., & Li, Y. (2025). Do AI chatbot-integrated writing tasks influence writing self-efficacy and critical thinking ability? An exploratory study. Computers and Education: Artificial Intelligence, 9(September 2024), 100472. https://doi.org/10.1016/j.caeai.2025.100472

García-Carmona, A. (2025). Scientific Thinking and Critical Thinking in Science Education : Two Distinct but Symbiotically Related Intellectual Processes. Science and Education, 34(1), 227–245. https://doi.org/10.1007/s11191-023-00460-5

Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1), 1–28. https://doi.org/10.3390/soc15010006

Glass, G. V. (1976). Primary, Secondary, and Meta-Analysis of Research.

Hassen, M. (2025). The Impact of AI on Students’ Reading, Critical Thinking, and Problem-Solving Skills. American Journal of Education and Information Technology, 9(2), 82–90. https://doi.org/10.11648/j.ajeit.20250902.12

Hedges, L. V., & Olkin, I. (1985). Statistical Methods for Meta-Analysis. Academic Press.

Heydarnejad, T. (2025). Unmasking the impacts of self-evaluation in AI-supported writing instruction on EFL learners’ emotion regulation, self-competence, motivation, and writing achievement. Computers and Education: Artificial Intelligence, 9(June), 100494. https://doi.org/10.1016/j.caeai.2025.100494

Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in knowledgebases. Journal of Intelligent Information Systems, 27(2), 159–184. https://doi.org/10.1007/s10844-006-2974-4

Hou, C., Zhu, G., & Sudarshan, V. (2025). The role of critical thinking on undergraduates’ reliance behaviours on generative AI in problem-solving. British Journal of Educational Technology, 56(5), 1919–1941. https://doi.org/10.1111/bjet.13613

İdil, Ş., Gülen, S., & Dönmez, İ. (2024). What Should We Understand from PISA 2022 Results? Journal of STEAM Education, 7(1), 1–9. https://doi.org/10.55290/steam.1415261

Ilgun Dibek, M., Sahin Kursad, M., & Erdogan, T. (2025). Influence of artificial intelligence tools on higher order thinking skills: a meta-analysis. Interactive Learning Environments, 33(3), 2216–2238. https://doi.org/10.1080/10494820.2024.2402028

Jose, B., Cherian, J., Verghis, A. M., Varghise, S. M., S, M., & Joseph, S. (2025). The cognitive paradox of AI in education: between enhancement and erosion. Frontiers in Psychology, 16. https://doi.org/10.3389/fpsyg.2025.1550621

Kamel, H. (2024). Understanding the impact of AI Hallucinations on the university community. Cybrarians Journal, 73, 111–134. https://doi.org/10.70000/cj.2024.73.622

Khan, I., Ahmad, A. R., Jabeur, N., & Mahdi, M. N. (2021). An artificial intelligence approach to monitor student performance and devise preventive measures. Smart Learning Environments, 8(1). https://doi.org/10.1186/s40561-021-00161-y

Kooli, C., Yusuf, N., & Sarhan, M. Y. (2026). Colloquial engagement theory with AI awareness (CET-AIA): A new creative pedagogical framework for ethical assessment in the age of artificial intelligence. Thinking Skills and Creativity, 60(November 2025), 102051. https://doi.org/10.1016/j.tsc.2025.102051

Lee, C. C., & Low, M. Y. H. (2024). Using genAI in education: the case for critical thinking. Frontiers in Artificial Intelligence, 7. https://doi.org/10.3389/frai.2024.1452131

Li, C., & Long, J. (2025). Comparing AI-Assisted and Traditional Teaching in College English: Pedagogical Benefits and Learning Behaviors. Information (Switzerland), 16(10), 1–19. https://doi.org/10.3390/info16100895

Li, T., Krajcik, J. S., & Spiro, R. (2025). The Transformative Collaboration of Human Intelligence and Artificial Intelligence in Designing Knowledge-in-Use Science Assessment for Learning. Journal of Science Education and Technology. https://doi.org/10.1007/s10956-025-10275-4

Li, Y., Kong, H., & Sciences, S. (2025). Education Studies Addressing “ Hallucinations ” in AI ‑ Generated Content : Strategies for Developing Student Fact ‑ Checking and. 1(2), 48–62.

Linsih, R. A., Soepriyanto, Y., & Praherdhiono, H. (2025). The Effect Of Collaborative Inquiry Based Learning Model And Learning Motivation On Critical Thinking Skills Of Elementary School Students. JKTP: Jurnal Kajian Teknologi Pendidikan, 8(2), 118–127. https://doi.org/10.17977/um038v8i22025p118

Liu, Q., & Tu, C. C. (2024). Improving Critical Thinking through AI-Supported Socio-Scientific Issues Instruction. Journal of Logistics, Informatics and Service Science, 11(3), 52–65. https://doi.org/10.33168/JLISS.2024.0304

Liu, Z., Gao, Y., Yang, Y., Kong, X., & Zhao, L. (2024). Exploring the influence of students’ perceptions of face-to-face collaboration on cognitive engagement and learning outcomes in collaborative programming. Acta Psychologica, 248(July), 104393. https://doi.org/10.1016/j.actpsy.2024.104393

Luan, J., Halim, H. A., & Saad, M. R. M. (2026). The Effect of AI and Multiple Intelligences in Project-Based Learning on English Achievement. International Journal of Computer-Assisted Language Learning and Teaching, 16(1), 1–28. https://doi.org/10.4018/IJCALLT.400912

Mai, H. T. (2026). Evaluating the impact of using ChatGPT on EFL students’ argumentative skills. Discover Education, 5(1), 1–20. https://doi.org/10.1007/s44217-026-01153-1

Maia, J. de S. Z., Bueno, A. P. A., & Sato, J. R. (2023). Applications of Artificial Intelligence Models in Educational Analytics and Decision Making: A Systematic Review. World, 4(2), 288–313. https://doi.org/10.3390/world4020019

Najah, E. F., Syaharuddin, Gatan, P. R., Mohammed, R. A., & Abdillah. (2024). The role of Information Literacy in Enhancing Students’ Critical Thinking Skills. Jurnal Literasi Perpustakaan Dan Informasi UHO: Jurnal Penelitian Kajian Perpustakaan Dan Informasi, 4(3), 129–139. https://doi.org/10.52423/jlpi.v4i3.15

Nusivera, E., Hikmat, A., & Ghani, A. R. A. (2025). Integration of Chat-GPT Usage in Language Learning Model to Improve Argumentation Skills, Complex Comprehension Skills, and Critical Thinking Skills. International Journal of Learning, Teaching and Educational Research, 24(2), 375–390. https://doi.org/10.26803/ijlter.24.2.19

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Bmj, 372. https://doi.org/10.1136/bmj.n71

Park, S. A., & Kim, H. Y. (2025). Development and effects of a scenario-based labor nursing simulation education program using an artificial intelligence tutor: a quasi-experimental study. Women’s Health Nursing, 31(2), 143–154. https://doi.org/10.4069/whn.2025.06.18

Partnership for 21 st Century Skills. (2015). Partnership for 21St Century Skills-Core Content Integration. Ohio Department of Education, 1. www.P21.org.

Piaget, J. (1970). Science of Education and the Psychology of the Child. Orion Press. https://books.google.co.id/books?id=ZwedAAAAMAAJ

Sinaga, A. I., Nasution, A. F., & Albina, M. (2025). Effect Of Artificial Intelligence Use On Creativity In Writing Scientific Works. JKTP: Jurnal Kajian Teknologi Pendidikan, 8(3), 226–234. https://doi.org/10.17977/um038v8i32025p226-234

Song, D., Zhang, P., Zhu, Y., Qi, S., Yang, Y., Gong, L., & Zhou, L. (2025). Effects of generative artificial intelligence on higher-order thinking skills and artificial intelligence literacy in nursing undergraduates: A quasi-experimental study. Nurse Education in Practice, 88(81), 104549. https://doi.org/10.1016/j.nepr.2025.104549

Sun, X., Liu, C., Li, P., & Cai, Z. (2026). Effects of Generative AI Tools On Students’ Critical Thinking: A Three-Level Meta-Analysis. Educational Psychology Review, 38(1), 84. https://doi.org/10.1007/s10648-026-10180-1

Sun, Y., Sheng, D., Zhou, Z., & Wu, Y. (2024). AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content. Humanities and Social Sciences Communications, 11(1), 1–14. https://doi.org/10.1057/s41599-024-03811-x

Tan, X., Cheng, G., & Ling, M. H. (2025). Enhancing teachers’ AI competency: A professional development intervention study based on intelligent-TPACK framework. Computers and Education: Artificial Intelligence, 9(November), 100521. https://doi.org/10.1016/j.caeai.2025.100521

Tian, J., & Zhang, R. (2025). Learners’ AI dependence and critical thinking: The psychological mechanism of fatigue and the social buffering role of AI literacy. Acta Psychologica, 260(October), 105725. https://doi.org/10.1016/j.actpsy.2025.105725

Vygotsky, L. S. (1978). Mind in Society: Development of Higher Psychological Processes. Harvard University Press. https://books.google.co.id/books?id=RxjjUefze_oC

Wiboolyasarin, W., Wiboolyasarin, K., Suwanwihok, K., Jinowat, N., & Muenjanchoey, R. (2024). Synergizing collaborative writing and AI feedback: An investigation into enhancing L2 writing proficiency in wiki-based environments. Computers and Education: Artificial Intelligence, 6(March), 100228. https://doi.org/10.1016/j.caeai.2024.100228

Yas, N., Abdulrahim, H., Rabadi, I. G., & Abdijabar, Z. (2025). Protecting the civil rights of the author from the risks of artificial intelligence. Research Journal in Advanced Humanities, 6(1), 1–9. https://doi.org/10.58256/s7rhdm72

Zablotska, L., Chernii, L., Meleshchenko, V., Olendr, T., & Tsar, I. (2026). Developing Students’ Critical Thinking through AI-Enhanced Foreign Language Instruction. Arab World English Journal, 3, 132–146. https://doi.org/10.24093/awej/ai3.9

Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: a systematic review. Smart Learning Environments, 11(1). https://doi.org/10.1186/s40561-024-00316-7

Zhao, Y., Yue, Y., Sun, Z., Jiang, Q., & Li, G. (2025). Does Generative Artificial Intelligence Improve Students’ Higher-Order Thinking? A Meta-Analysis Based on 29 Experiments and Quasi-Experiments. Journal of Intelligence, 13(12), 1–21. https://doi.org/10.3390/jintelligence13120160

Zheng, K., Shen, Z., Chen, Z., Che, C., & Zhu, H. (2024). Application of AI-empowered scenario-based simulation teaching mode in cardiovascular disease education. BMC Medical Education, 24(1). https://doi.org/10.1186/s12909-024-05977-z

Downloads

Published

2026-11-11

How to Cite

Rachmadhani, S. A. D., Muhson, A., & Setiawan, R. (2026). Effectiveness of Artificial Intelligence on Students’ Critical Thinking Skills: A Meta-Analysis Study. JKTP: Jurnal Kajian Teknologi Pendidikan, 9(4), 499–511. https://doi.org/10.17977/um038v9i42026p499-511