Inter-Rater Reliability Analysis in Performance-Based Assessment: A Comparison of Generalizability Coefficients and Rater Consistency

Mustafa Köroğlu

Makale detayları

Inter-Rater Reliability Analysis in Performance-Based Assessment: A Comparison of Generalizability Coefficients and Rater Consistency

Yazarlar:

Mustafa Köroğlu

Yayıncı:

Necmettin Erbakan University

Ana Başlıklar:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Dergi:

Ahmet Keleşoğlu Eğitim Fakültesi Dergisi Vol.7 Issue 2, pp.218-234

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001359745

pISSN:

2687-1750

Anahtar kelimeler:

Özet:

 This study investigates the reliability of a performance-based assessment tool used to evaluate university students basic statistical skills within the framework of Generalizability Theory GT. A total of 80 students from the Guidance and Psychological Counseling program participated in a two-hour examination consisting of 10 applied tasks. The tasks were scored independently by two raters using a detailed analytic rubric. The scores were analyzed using a fully crossed design p  i  r, with variance components estimated via the maximum likelihood method, and 95 confidence intervals calculated using a cluster bootstrap procedure 1,000 resamples. Results showed that 50.2 of the total variance was attributable to students, 25.6 to items, and 16.6 to raters, while interaction terms remained at low levels. The initial relative generalizability coefficient was calculated as .98, and the absolute decision coefficient Φ was .81. When the number of items was increased to 15 and the number of raters to five, the Φ coefficient improved to .90, and absolute error variance decreased to .45. Findings indicated that true performance differences among students were strongly captured, although rater effects could not be completely eliminated. Expanding task coverage and increasing the number of raters were found to be effective strategies for reducing both absolute and relative error variances. The study supports the importance of rubric use, investment in rater training, a multi-taskmulti-rater approach, and GT-based revision cycles in high-stakes performance assessments. The findings are expected to inform practical assessment strategies aimed at improving statistical literacy in teacher education programs. Additionally, it is recommended that the study be replicated with larger and more diverse samples across disciplines to enhance internal validity. Future directions may include implementing rater feedback cycles through online platforms and integrating rubric-supported scoring systems. Keywords: Generalizability theory, performance-based assessment, rater reliability.

 This study investigates the reliability of a performance-based assessment tool used to evaluate university students basic statistical skills within the framework of Generalizability Theory GT. A total of 80 students from the Guidance and Psychological Counseling program participated in a two-hour examination consisting of 10 applied tasks. The tasks were scored independently by two raters using a detailed analytic rubric. The scores were analyzed using a fully crossed design p  i  r, with variance components estimated via the maximum likelihood method, and 95 confidence intervals calculated using a cluster bootstrap procedure 1,000 resamples. Results showed that 50.2 of the total variance was attributable to students, 25.6 to items, and 16.6 to raters, while interaction terms remained at low levels. The initial relative generalizability coefficient was calculated as .98, and the absolute decision coefficient Φ was .81. When the number of items was increased to 15 and the number of raters to five, the Φ coefficient improved to .90, and absolute error variance decreased to .45. Findings indicated that true performance differences among students were strongly captured, although rater effects could not be completely eliminated. Expanding task coverage and increasing the number of raters were found to be effective strategies for reducing both absolute and relative error variances. The study supports the importance of rubric use, investment in rater training, a multi-taskmulti-rater approach, and GT-based revision cycles in high-stakes performance assessments. The findings are expected to inform practical assessment strategies aimed at improving statistical literacy in teacher education programs. Additionally, it is recommended that the study be replicated with larger and more diverse samples across disciplines to enhance internal validity. Future directions may include implementing rater feedback cycles through online platforms and integrating rubric-supported scoring systems. Keywords: Generalizability theory, performance-based assessment, rater reliability.

 This study investigates the reliability of a performance-based assessment tool used to evaluate university students basic statistical skills within the framework of Generalizability Theory GT. A total of 80 students from the Guidance and Psychological Counseling program participated in a two-hour examination consisting of 10 applied tasks. The tasks were scored independently by two raters using a detailed analytic rubric. The scores were analyzed using a fully crossed design p  i  r, with variance components estimated via the maximum likelihood method, and 95 confidence intervals calculated using a cluster bootstrap procedure 1,000 resamples. Results showed that 50.2 of the total variance was attributable to students, 25.6 to items, and 16.6 to raters, while interaction terms remained at low levels. The initial relative generalizability coefficient was calculated as .98, and the absolute decision coefficient Φ was .81. When the number of items was increased to 15 and the number of raters to five, the Φ coefficient improved to .90, and absolute error variance decreased to .45. Findings indicated that true performance differences among students were strongly captured, although rater effects could not be completely eliminated. Expanding task coverage and increasing the number of raters were found to be effective strategies for reducing both absolute and relative error variances. The study supports the importance of rubric use, investment in rater training, a multi-taskmulti-rater approach, and GT-based revision cycles in high-stakes performance assessments. The findings are expected to inform practical assessment strategies aimed at improving statistical literacy in teacher education programs. Additionally, it is recommended that the study be replicated with larger and more diverse samples across disciplines to enhance internal validity. Future directions may include implementing rater feedback cycles through online platforms and integrating rubric-supported scoring systems. Keywords: Generalizability theory, performance-based assessment, rater reliability.

 Bu çalışma, üniversite öğrencilerinin temel istatistik becerilerini değerlendirmek için kullanılan performansa dayalı bir ölçme aracının güvenirliğini Genellenebilirlik Kuramı GK çerçevesinde incelemektedir. Rehberlik ve Psikolojik Danışmanlık programından toplam 80 öğrenci, 10 uygulamalı görevden oluşan iki saatlik bir sınava katılmıştır. Görevler, ayrıntılı bir analitik rubrik kullanılarak iki puanlayıcı tarafından bağımsız olarak puanlanmıştır. Puanlar, tam çaprazlanmış bir desen b  m  d kullanılarak analiz edilmiş; varyans bileşenleri maksimum olabilirlik yöntemiyle tahmin edilmiş ve 95 güven aralıkları 1.000 tekrar içeren küme bootstrap prosedürüyle hesaplanmıştır. Sonuçlar, toplam varyansın 50,2sinin öğrencilere, 25,6sının maddelere ve 16,6sının puanlayıcılara atfedildiğini, etkileşim terimlerinin ise düşük düzeyde kaldığını göstermiştir. İlk göreli genellenebilirlik katsayısı .98, mutlak karar katsayısı Φ ise .81 olarak hesaplanmıştır. Madde sayısı 15e, puanlayıcı sayısı ise beşe çıkarıldığında Φ katsayısı .90a yükselmiş ve mutlak hata varyansı .45e düşmüştür. Bulgular, öğrenciler arasındaki gerçek performans farklılıklarının güçlü bir şekilde yakalandığını, ancak puanlayıcı etkilerinin tamamen ortadan kaldırılamadığını göstermektedir. Görev kapsamını genişletmek ve puanlayıcı sayısını artırmak, hem mutlak hem de göreli hata varyanslarını azaltmada etkili stratejiler olarak bulunmuştur. Çalışma, rubrik kullanımının, puanlayıcı eğitimine yatırım yapmanın, çoklu görevçoklu puanlayıcı yaklaşımının ve GK temelli revizyon döngülerinin yüksek riskli performans değerlendirmelerinde önemini desteklemektedir. Bulguların, öğretmen eğitimi programlarında istatistiksel okuryazarlığı geliştirmeyi amaçlayan pratik ölçme stratejilerine katkı sağlaması beklenmektedir. Ayrıca, çalışmanın iç geçerliliğini artırmak için farklı disiplinlerde daha büyük ve çeşitli örneklemlerle tekrarlanması önerilmektedir. Gelecek araştırmalar, çevrim içi platformlar aracılığıyla puanlayıcı geri bildirim döngülerinin uygulanmasını ve rubrik destekli puanlama sistemlerinin entegrasyonunu içerebilir. Anahtar Kelimeler: Genellenebilirlik kuramı, performansa dayalı değerlendirme, puanlayıcı güvenirliği.

Makale detayları

Inter-Rater Reliability Analysis in Performance-Based Assessment: A Comparison of Generalizability Coefficients and Rater Consistency

Yazarlar:

Mustafa Köroğlu

Yayıncı:

Necmettin Erbakan University

Ana Başlıklar:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Dergi:

Ahmet Keleşoğlu Eğitim Fakültesi Dergisi Vol.7 Issue 2, pp.218-234

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001359745

pISSN:

2687-1750

Generalizability theory

performance-based assessment

rater reliabilityGenellenebilirlik kuramı

performansa dayalı değerlendirme

puanlayıcı güvenirliği

Sayı

Makale detayları

Inter-Rater Reliability Analysis in Performance-Based Assessment: A Comparison of Generalizability Coefficients and Rater Consistency

Yazarlar: Mustafa Köroğlu

Yayıncı: Necmettin Erbakan University

Ana Başlıklar: Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Dergi: Ahmet Keleşoğlu Eğitim Fakültesi Dergisi Vol.7 Issue 2, pp.218-234

Belge Tipi: Makale

Yayın Tarihi: 9/2025

Belge Dili: İngilizce

Document ID (Org No.): 2001359745

pISSN: 2687-1750

Generalizability theory

performance-based assessment

rater reliabilityGenellenebilirlik kuramı

performansa dayalı değerlendirme

puanlayıcı güvenirliği

Sayı

Yazarlar:

Mustafa Köroğlu

Yayıncı:

Necmettin Erbakan University

Ana Başlıklar:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Dergi:

Ahmet Keleşoğlu Eğitim Fakültesi Dergisi Vol.7 Issue 2, pp.218-234

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001359745

pISSN:

2687-1750