EXAMINING THE INTERRATER RELIABILITY ACCORDING TO DIFFERENT SCALING TYPES, NUMBER OF RATERS AND NUMBER OF RATED SUBJECTS

Yılmaz Orhun Gürlük; Mediha Korkmaz; Gizem Cömert; Ömer Emre Can Alagöz

Makale detayları

EXAMINING THE INTERRATER RELIABILITY ACCORDING TO DIFFERENT SCALING TYPES, NUMBER OF RATERS AND NUMBER OF RATED SUBJECTS

Yazarlar:

Yılmaz Orhun Gürlük | Mediha Korkmaz | Gizem Cömert | Ömer Emre Can Alagöz

Yayıncı:

Dokuz Eylul University

Dergi:

Dokuz Eylül Üniversitesi Sosyal Bilimler Enstitüsü Dergisi Vol.25 Issue 1, pp.227-245

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001285154

pISSN:

1308-0911

Anahtar kelimeler:

Özet:

 In this study, it was aimed to compare the coefficients of interrater agreement according to classical statistic theories. The difference between the calculated agreement coefficients according to different scaling types has been focused and the importance of deciding on the scaling type has been revealed. It was examined whether the change in the number of raters and rateds affected the values, and the most appropriate sample size to be used for the interrater agreement was calculated by using the optimization analysis of the generalizability theory. In the study, 9 cards of the Bender-Gestalt motor skill test, which can be seen by everyone, were applied to 35 children in total, and the measurements were evaluated by 8 raters blindly to each other. Accordingly, the highest agreement value was calculated in the intra-class correlation coefficient and this value was followed by Krippendorff alpha, Fleiss kappa and Cohen kappa, respectively. It has been determined that as both the number of rateds and raters decrease, the agreement values tend to decrease. On the other hand, it was determined that the difficulty level of the cards did not have a significant effect. The high generalizability coefficients showed that the test was reliably scored by the raters. When the optimization analysis was examined, it was seen that the most suitable sample size for this test was 50. Having more than 50 participants did not increase agreement.

 In this study, it was aimed to compare the coefficients of interrater agreement according to classical statistic theories. The difference between the calculated agreement coefficients according to different scaling types has been focused and the importance of deciding on the scaling type has been revealed. It was examined whether the change in the number of raters and rateds affected the values, and the most appropriate sample size to be used for the interrater agreement was calculated by using the optimization analysis of the generalizability theory. In the study, 9 cards of the Bender-Gestalt motor skill test, which can be seen by everyone, were applied to 35 children in total, and the measurements were evaluated by 8 raters blindly to each other. Accordingly, the highest agreement value was calculated in the intra-class correlation coefficient and this value was followed by Krippendorff alpha, Fleiss kappa and Cohen kappa, respectively. It has been determined that as both the number of rateds and raters decrease, the agreement values tend to decrease. On the other hand, it was determined that the difficulty level of the cards did not have a significant effect. The high generalizability coefficients showed that the test was reliably scored by the raters. When the optimization analysis was examined, it was seen that the most suitable sample size for this test was 50. Having more than 50 participants did not increase agreement.

 In this study, it was aimed to compare the coefficients of interrater agreement according to classical statistic theories. The difference between the calculated agreement coefficients according to different scaling types has been focused and the importance of deciding on the scaling type has been revealed. It was examined whether the change in the number of raters and rateds affected the values, and the most appropriate sample size to be used for the interrater agreement was calculated by using the optimization analysis of the generalizability theory. In the study, 9 cards of the Bender-Gestalt motor skill test, which can be seen by everyone, were applied to 35 children in total, and the measurements were evaluated by 8 raters blindly to each other. Accordingly, the highest agreement value was calculated in the intra-class correlation coefficient and this value was followed by Krippendorff alpha, Fleiss kappa and Cohen kappa, respectively. It has been determined that as both the number of rateds and raters decrease, the agreement values tend to decrease. On the other hand, it was determined that the difficulty level of the cards did not have a significant effect. The high generalizability coefficients showed that the test was reliably scored by the raters. When the optimization analysis was examined, it was seen that the most suitable sample size for this test was 50. Having more than 50 participants did not increase agreement.

 Bu araştırmada klasik kuramlara göre puanlayıcılar arası uyum katsayılarını karşılaştırmak amaçlanmıştır. Farklı ölçekleme türlerine göre elde edilen katsayılar üzerinden hesaplanan değerler arasındaki farka odaklanılmış ve ölçekleme türüne karar vermenin önemi ortaya konmuştur. Puanlanan ve puanlayıcı sayısının değişmesinin değerleri etkileyip etkilemediğine bakılmış ve genellenebilirlik kuramının optimizasyon analizi kullanılarak puanlayıcılar arası uyum için kullanılacak en uygun örneklem büyüklüğü hesaplanmıştır. Araştırmada toplamda 35 çocuğa Bender Görsel Motor Gestalt II testinin yaş gruplarında ortak olan 9 kopyalama kartı uygulanmış ve alınan ölçümler toplamda 8 puanlayıcı tarafından birbirlerine kör olarak değerlendirilmiştir. Sonuçlara göre en yüksek uyum değeri sınıf içi korelasyon katsayısında hesaplanmış ve bu değeri sırasıyla Krippendorff alfa, Fleiss kappa ve Cohen kappa takip etmiştir. Hem puanlanan hem de puanlayıcı sayısı azaldıkça uyum değerlerinin düşme eğiliminde olduğu tespit edilmiştir. Öte yandan kartların zorluk düzeyinin anlamlı bir etkisi olmadığı saptanmıştır. Genellenebilirlik katsayılarının yüksek çıkması testin puanlayıcılar tarafından güvenilir şekilde puanlandığını göstermiştir. Optimizasyon analizi incelendiğinde bu test için en uygun örneklem büyüklüğünün 50 olduğu görülmüştür. Katılımcı sayısının 50den fazla olması ise uyumu arttırmamıştır.

Makale detayları

EXAMINING THE INTERRATER RELIABILITY ACCORDING TO DIFFERENT SCALING TYPES, NUMBER OF RATERS AND NUMBER OF RATED SUBJECTS

Yazarlar:

Yılmaz Orhun Gürlük | Mediha Korkmaz | Gizem Cömert | Ömer Emre Can Alagöz

Yayıncı:

Dokuz Eylul University

Dergi:

Dokuz Eylül Üniversitesi Sosyal Bilimler Enstitüsü Dergisi Vol.25 Issue 1, pp.227-245

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001285154

pISSN:

1308-0911

Interrater Agreement

Scaling Types

Intraclass Correlation

Kappa

Krippendorff Alpha

Generalizability Theory

Puanlayıcılar Arası Uyum

Ölçekleme Türü

Sınıf İçi Korelasyon

Kappa

Krippendorff Alfa

Genellenebilirlik Kuramı

Sayı

Makale detayları

Yazarlar: Yılmaz Orhun Gürlük | Mediha Korkmaz | Gizem Cömert | Ömer Emre Can Alagöz

Yayıncı: Dokuz Eylul University

Dergi: Dokuz Eylül Üniversitesi Sosyal Bilimler Enstitüsü Dergisi Vol.25 Issue 1, pp.227-245

Belge Tipi: Makale

Yayın Tarihi: 3/2023

Belge Dili: Türkçe

Document ID (Org No.): 2001285154

pISSN: 1308-0911

Sayı

Yazarlar:

Yılmaz Orhun Gürlük | Mediha Korkmaz | Gizem Cömert | Ömer Emre Can Alagöz

Yayıncı:

Dokuz Eylul University

Dergi:

Dokuz Eylül Üniversitesi Sosyal Bilimler Enstitüsü Dergisi Vol.25 Issue 1, pp.227-245

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001285154

pISSN:

1308-0911