AN INVESTIGATION ON THE EFFECTIVENESS OF CHATBOTS IN EVALUATING WRITING ASSIGNMENTS IN EFL CONTEXTS

Saliha Toscu

Makale detayları

AN INVESTIGATION ON THE EFFECTIVENESS OF CHATBOTS IN EVALUATING WRITING ASSIGNMENTS IN EFL CONTEXTS

Yazarlar:

Saliha Toscu

Yayıncı:

Burdur Mehmet Akif Ersoy University

Ana Başlıklar:

Eğitim Teknolojisi ve Bilgi İşlem

Dergi:

Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi Issue 72, pp.295-329

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001329671

pISSN:

2146-5983

Anahtar kelimeler:

Özet:

 This study aims to explore the effectiveness of chatbots, specifically ChatGPT, in evaluating writing assignments. 51 English as a foreign language EFL learners from Türkiye took part in the study voluntarily. Based on a quasi-experimental research design, the study involved dividing the participants into two non-equivalent groups. The participants in the control group received feedback on their writing from a teacher, while the experimental group was given feedback by the ChatGPT. Thus, the research indicated how Artificial Intelligence AI-generated feedback was influential in evaluating writing assignments compared to human-generated feedback. The feedback given by the teacher and the ChatGPT was analyzed via a content analysis, and the results showed that the ChatGPT and the teacher feedback had differences in the language style, register, content, and accuracy. Besides, students perceptions of the feedback were revealed through the Student Perceptions of Writing Feedback Scale of Marss 2016 and analyzed using Independent Samples t-Tests on the SPSS. The statistical tests indicated that the students perceptions did not vary whether the feedback was AI or human-generated. This study suggests that AI-generated feedback can match human feedback in terms of students perceptions. Still, the differences regarding feedback features show the strengths of a teacher and a bot for giving feedback by suggesting a more complementary role for AI technology in educational settings.

 This study aims to explore the effectiveness of chatbots, specifically ChatGPT, in evaluating writing assignments. 51 English as a foreign language EFL learners from Türkiye took part in the study voluntarily. Based on a quasi-experimental research design, the study involved dividing the participants into two non-equivalent groups. The participants in the control group received feedback on their writing from a teacher, while the experimental group was given feedback by the ChatGPT. Thus, the research indicated how Artificial Intelligence AI-generated feedback was influential in evaluating writing assignments compared to human-generated feedback. The feedback given by the teacher and the ChatGPT was analyzed via a content analysis, and the results showed that the ChatGPT and the teacher feedback had differences in the language style, register, content, and accuracy. Besides, students perceptions of the feedback were revealed through the Student Perceptions of Writing Feedback Scale of Marss 2016 and analyzed using Independent Samples t-Tests on the SPSS. The statistical tests indicated that the students perceptions did not vary whether the feedback was AI or human-generated. This study suggests that AI-generated feedback can match human feedback in terms of students perceptions. Still, the differences regarding feedback features show the strengths of a teacher and a bot for giving feedback by suggesting a more complementary role for AI technology in educational settings.

 This study aims to explore the effectiveness of chatbots, specifically ChatGPT, in evaluating writing assignments. 51 English as a foreign language EFL learners from Türkiye took part in the study voluntarily. Based on a quasi-experimental research design, the study involved dividing the participants into two non-equivalent groups. The participants in the control group received feedback on their writing from a teacher, while the experimental group was given feedback by the ChatGPT. Thus, the research indicated how Artificial Intelligence AI-generated feedback was influential in evaluating writing assignments compared to human-generated feedback. The feedback given by the teacher and the ChatGPT was analyzed via a content analysis, and the results showed that the ChatGPT and the teacher feedback had differences in the language style, register, content, and accuracy. Besides, students perceptions of the feedback were revealed through the Student Perceptions of Writing Feedback Scale of Marss 2016 and analyzed using Independent Samples t-Tests on the SPSS. The statistical tests indicated that the students perceptions did not vary whether the feedback was AI or human-generated. This study suggests that AI-generated feedback can match human feedback in terms of students perceptions. Still, the differences regarding feedback features show the strengths of a teacher and a bot for giving feedback by suggesting a more complementary role for AI technology in educational settings.

 Bu çalışma, sohbet botlarının, özellikle ChatGPTin, yazma ödevlerini değerlendirmedeki etkinliğini ortaya çıkarmayı hedeflemektedir. Türkiyedeki özel bir okuldaki 51 yabancı dil olarak İngilizce öğrencisi çalışmaya gönüllü olarak katılmıştır. Yarı deneysel araştırma desenine dayalı olan çalışmada katılımcılar eşit olmayan iki gruba ayrılmıştır. Kontrol grubundaki katılımcıların yazma ödevlerine öğretmen, deney grubundakilere ChatGPT tarafından dönüt verilmiştir. Böylece, çalışma yapay zekâ tarafından üretilen geri bildirimin insan tarafından üretilen geri bildirimle karşılaştırıldığında yazma ödevlerini değerlendirmede etkinliğini göstermiştir. Öğretmen ve ChatGPT tarafından verilen dönüt içerik analizi yoluyla incelenmiştir ve sonuçlar ChatGPT ve öğretmen dönütü arasında dil stili, dil biçimi, içerik ve dil bilgisi doğruluğu konularında farklılıklar olduğunu göstermiştir. Ayrıca, çalışmada öğrencilerin geri bildirime yönelik algıları Marssın 2016 Öğrenci Yazma Geri Bildirim Algısı Ölçeği yoluyla ortaya çıkarılmıştır ve SPSSte Bağımsız Grup t-Test kullanılarak analiz edilmiştir. İstatistiksel testler, öğrencilerin algılarının geri bildirimin yapay zekâ veya insan tarafından üretildiğinde değişmediğini göstermiştir. Bu çalışma, öğrenci algısı bakımından yapay zekâ teknolojisi ve insan geri bildiriminin eşleşebileceğini önermektedir, ancak dönütün özelliklerine ilişkin farklılıklar yapay zekâ teknolojisinin eğitim ortamlarında daha tamamlayıcı bir rol oynamasını önererek öğretmenin ve botun geri bildirim vermede güçlü yönlerini göstermektedir.

Makale detayları

AN INVESTIGATION ON THE EFFECTIVENESS OF CHATBOTS IN EVALUATING WRITING ASSIGNMENTS IN EFL CONTEXTS

Yazarlar:

Saliha Toscu

Yayıncı:

Burdur Mehmet Akif Ersoy University

Ana Başlıklar:

Eğitim Teknolojisi ve Bilgi İşlem

Dergi:

Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi Issue 72, pp.295-329

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001329671

pISSN:

2146-5983

Chatbots

ChatGPT

EFL writing

feedback

evaluationdeğerlendirme

Sohbet Robotları

chatgpt

yabancı dil olarak İngilizce yazma

geri bildirim

Sayı

Makale detayları

Yazarlar: Saliha Toscu

Yayıncı: Burdur Mehmet Akif Ersoy University

Ana Başlıklar: Eğitim Teknolojisi ve Bilgi İşlem

Dergi: Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi Issue 72, pp.295-329

Belge Tipi: Makale

Yayın Tarihi: 10/2024

Belge Dili: İngilizce

Document ID (Org No.): 2001329671

pISSN: 2146-5983

Sayı

Yazarlar:

Saliha Toscu

Yayıncı:

Burdur Mehmet Akif Ersoy University

Ana Başlıklar:

Eğitim Teknolojisi ve Bilgi İşlem

Dergi:

Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi Issue 72, pp.295-329

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001329671

pISSN:

2146-5983