Researchers from China assess the capabilities of ChatGPT and analyze how it could assist both teachers and students in educational activities.

SHANGHAI, Jan. 9, 2024 /PRNewswire/ — Generative artificial intelligence (AI) platforms such as ChatGPT have made a significant impact since being released in 2022. Due to their comprehensive capabilities, generative AI-powered models could vastly reshape various academic and applied fields, with education being no exception.

While certain industry experts are of the view ChatGPT may be a hinderance to traditional education, others foresee unprecedented opportunities to make learning a more diverse and adaptable experience for teachers and students alike. However, there is a lack of systematic evidence demonstrating ChatGPT’s potential in this field.

Against this backdrop, a team of researchers led by Dr. Xiaozhe Yang from the Institute of Curriculum and Instruction, East China Normal University, conducted several tests to investigate the educational capabilities of ChatGPT’s latest iteration. Their findings were made available online on November 5, 2023, in  ECNU Review of Education.

First, the research team evaluated ChatGPT-4’s higher-order logical abilities using the Watson-Glaser Critical Thinking Appraisal (WGCTA) test. WGCTA is used to assess participants’ common sense, logic, and reasoning skills based on their answers to several questions. “With an overall accuracy rate of 82.5%, ChatGPT outperformed average undergraduate students in terms of critical thinking. It analyzed arguments with 100% accuracy and identified hypotheses, deductions, and explanations with an accuracy of 87.5% or higher,” states Dr. Yang. He further adds, “However, its performance in reasoning ability was average.“

To analyze ‘thinking patterns’ of the AI models, the research team used the Five Core Competencies Questionnaire. The results indicated that ChatGPT achieved high scores in metacognition, willingness to collaborate and communicate, and problem solving. However, it exhibited a low inclination toward creativity, lacking the ability to create independently. Nonetheless, in practice, users can use appropriate prompts to elicit creative answers.

Subsequently, the team examined ChatGPT’s potential for educational applications through National Teacher Certificate Examination (NTCE) for teaching Chinese at the high-school level. ChatGPT-4 performed quite well at this test, achieving an overall accuracy of 71%. “Passing the NTCE examination signifies that ChatGPT has sufficient critical thinking to handle simple logic problems and a tendency to include core competencies in problem solving, collaboration, communication, and metacognition. Relying on a rich and extensive text dataset, ChatGPT could master relevant educational policies, teaching methods, and educational principles, as well as the professional ethics and conduct of teachers, proving its competence for performing educational work,” highlights Dr. Yang.

Overall, the findings of this study suggest that generative AI is poised to become a valuable educational tool that will likely reshape the structure of modern curricula. These models could help promote educational equity by adjusting to the needs of each student. A hopeful Dr. Yang says, “Education has entered a new era of exploration as we pursue the coexistence and co-creation of human and computer systems through iterations of generative AI.”

Reference

Title of original paper: Assessing ChatGPT’s Educational Capabilities and Application Potential

Journal: ECNU Review of Education

DOI: https://doi.org/10.1177/20965311231210006

Media Contact:
Li You
+86 (21) 62224545
[email protected] 

SOURCE ECNU Review of Education