Validity of Simulation Studies: A Case Research in the Context of Differential Item Functioning Detection

Özkan Saatçioğlu

Makale detayları

Validity of Simulation Studies: A Case Research in the Context of Differential Item Functioning Detection

Yazarlar:

Özkan Saatçioğlu

Yayıncı:

Ataturk University

Ana Başlıklar:

Test | Ölçme ve Psikometri (Diğer)

Dergi:

Journal of Psychometric Research Vol.3 Issue 1, pp.24-40

Belge Tipi:

Makale

Yayın Tarihi:

3/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001342639

pISSN:

3023-4646

Anahtar kelimeler:

Özet:

 The aim of this study is to examine the simulation validity by determining whether the simulation process produces results that are realistically close to expectations, through the generation of artificial data containing Differential Item Functioning DIF and assessing whether the data were accurately generated. In the study, which involves one reference group and two focal groups, 2250 different conditions were simulated by considering factors such as the sample size of the reference group, the sample size ratios of the focal groups, the amount of DIF, and the DIF technique. During the data generation process, random data for difficulty and discrimination parameters were generated using the Two-Parameter Logistic Model 2PLM, and it was planned that 20 of the items in the test would contain DIF. To test the validity of the simulation, mean absolute bias and RMSE values for the difficulty and discrimination parameters were calculated both at the item level and by considering the relevant factors. The analysis results revealed that the mean absolute bias and RMSE values calculated for the difficulty and discrimination parameters were low and close to zero. This indicates that estimation errors were minimal and supports the validity of the results. Additionally, it was found that the sample size of the reference group and the sample size ratios of the focal groups had a statistically significant effect on the mean absolute bias and RMSE values for both difficulty and discrimination parameters, and it was observed that as the sample size increased, the mean absolute bias and RMSE values decreased. However, it was concluded that the amount of DIF added to the focal groups did not have a significant effect on the accuracy of parameter estimations. The findings demonstrate that sample size plays a critical role in the accuracy of parameter estimations, while the amount of DIF does not significantly impact this process, and the results of the study are consistent with relevant research in the literature. As a result of this research, it has been recommended that validity evidence for the simulation should be provided not only in DIF investigation studies but also in simulation studies conducted in various subject areas within the field of psychometrics.

 The aim of this study is to examine the simulation validity by determining whether the simulation process produces results that are realistically close to expectations, through the generation of artificial data containing Differential Item Functioning DIF and assessing whether the data were accurately generated. In the study, which involves one reference group and two focal groups, 2250 different conditions were simulated by considering factors such as the sample size of the reference group, the sample size ratios of the focal groups, the amount of DIF, and the DIF technique. During the data generation process, random data for difficulty and discrimination parameters were generated using the Two-Parameter Logistic Model 2PLM, and it was planned that 20 of the items in the test would contain DIF. To test the validity of the simulation, mean absolute bias and RMSE values for the difficulty and discrimination parameters were calculated both at the item level and by considering the relevant factors. The analysis results revealed that the mean absolute bias and RMSE values calculated for the difficulty and discrimination parameters were low and close to zero. This indicates that estimation errors were minimal and supports the validity of the results. Additionally, it was found that the sample size of the reference group and the sample size ratios of the focal groups had a statistically significant effect on the mean absolute bias and RMSE values for both difficulty and discrimination parameters, and it was observed that as the sample size increased, the mean absolute bias and RMSE values decreased. However, it was concluded that the amount of DIF added to the focal groups did not have a significant effect on the accuracy of parameter estimations. The findings demonstrate that sample size plays a critical role in the accuracy of parameter estimations, while the amount of DIF does not significantly impact this process, and the results of the study are consistent with relevant research in the literature. As a result of this research, it has been recommended that validity evidence for the simulation should be provided not only in DIF investigation studies but also in simulation studies conducted in various subject areas within the field of psychometrics.

 The aim of this study is to examine the simulation validity by determining whether the simulation process produces results that are realistically close to expectations, through the generation of artificial data containing Differential Item Functioning DIF and assessing whether the data were accurately generated. In the study, which involves one reference group and two focal groups, 2250 different conditions were simulated by considering factors such as the sample size of the reference group, the sample size ratios of the focal groups, the amount of DIF, and the DIF technique. During the data generation process, random data for difficulty and discrimination parameters were generated using the Two-Parameter Logistic Model 2PLM, and it was planned that 20 of the items in the test would contain DIF. To test the validity of the simulation, mean absolute bias and RMSE values for the difficulty and discrimination parameters were calculated both at the item level and by considering the relevant factors. The analysis results revealed that the mean absolute bias and RMSE values calculated for the difficulty and discrimination parameters were low and close to zero. This indicates that estimation errors were minimal and supports the validity of the results. Additionally, it was found that the sample size of the reference group and the sample size ratios of the focal groups had a statistically significant effect on the mean absolute bias and RMSE values for both difficulty and discrimination parameters, and it was observed that as the sample size increased, the mean absolute bias and RMSE values decreased. However, it was concluded that the amount of DIF added to the focal groups did not have a significant effect on the accuracy of parameter estimations. The findings demonstrate that sample size plays a critical role in the accuracy of parameter estimations, while the amount of DIF does not significantly impact this process, and the results of the study are consistent with relevant research in the literature. As a result of this research, it has been recommended that validity evidence for the simulation should be provided not only in DIF investigation studies but also in simulation studies conducted in various subject areas within the field of psychometrics.

 Bu çalışmanın amacı, simülasyon sürecinin beklenen şekilde gerçeğe yakın sonuçlar ortaya çıkarıp çıkarmadığını belirlemek amacıyla Değişen Madde Fonksiyonu DMF içeren yapay verilerin doğru bir şekilde üretilip üretilmediğine yönelik simülasyon geçerliğinin incelenmesidir. Bir referans iki odak olmak üzere üç grubun ele alındığı araştırmada, referans grubun örneklem büyüklüğü, odak grupların örneklem büyüklüğü oranları, DMF miktarı ve DMF tekniği faktörleri dikkate alınarak 2250 farklı koşul simüle edilmiştir. Veri üretim sürecinde, İki Parametreli Lojistik Model 2PLM ile güçlük ve ayırt edicilik parametreleri için rastgele veriler oluşturulmuş ve testteki maddelerin 20sinin DMF içermesi planlanmıştır. Simülasyonun geçerliğini test etmek amacıyla, güçlük ve ayırt edicilik parametrelerine ilişkin ortalama mutlak yanlılık ve RMSE değerleri hem madde düzeyinde hem de ilgili faktörler dikkate alınarak hesaplanmıştır. Analizler sonucunda, güçlük ve ayırt edicilik parametreleri için hesaplanan ortalama mutlak yanlılık ve RMSE değerlerinin düşük ve sıfıra yakın olduğu bulunmuştur. Bu durum kestirim hatalarının az olduğunu ve sonuçların geçerliğinin desteklendiğini ortaya koymuştur. Ayrıca referans grubun örneklem büyüklüğünün ve odak grupların örneklem büyüklüğü oranlarının hem güçlük hem de ayırt edicilik parametreleri için ortalama mutlak yanlılık ve RMSE değerleri üzerinde istatistiksel olarak manidar bir etkiye sahip olduğu belirlenmiş ve örneklem büyüklüğü arttıkça ortalama mutlak yanlılık ve RMSE değerlerinin azaldığı tespit edilmiştir. Bununla birlikte, odak gruplara eklenen DMF miktarlarının, parametre kestirimlerinin doğruluğu üzerinde anlamlı bir etki oluşturmadığı sonucuna ulaşılmıştır. Elde edilen bulgular, örneklem büyüklüğünün parametre kestirimlerinin doğruluğu üzerinde kritik bir rol oynadığını ve DMF miktarının bu süreçte anlamlı bir etki yaratmadığını ortaya koymuş ve çalışmanın bulguları alanyazındaki ilgili araştırmalar ile tutarlılık göstermiştir. Yapılan bu araştırma sonucunda DMF inceleme çalışmalarının yanı sıra psikometrinin farklı konu alanlarında yapılacak olan simülasyon çalışmalarında da simülasyonun geçerlik kanıtlarının sunulması gerektiği önerilmiştir.

Makale detayları

Validity of Simulation Studies: A Case Research in the Context of Differential Item Functioning Detection

Yazarlar:

Özkan Saatçioğlu

Yayıncı:

Ataturk University

Ana Başlıklar:

Test | Ölçme ve Psikometri (Diğer)

Dergi:

Journal of Psychometric Research Vol.3 Issue 1, pp.24-40

Belge Tipi:

Makale

Yayın Tarihi:

3/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001342639

pISSN:

3023-4646

Simulation

Validity of Simulation

Differential Item Functioning

Item Difficulty Parameter

Item Discrimination ParameterSimülasyon

Simülasyonun Geçerliği

Değişen Madde Fonksiyonu

Madde Güçlük Parametresi

Madde Ayırt Edicilik Parametresi

Sayı

Makale detayları

Yazarlar: Özkan Saatçioğlu

Yayıncı: Ataturk University

Ana Başlıklar: Test | Ölçme ve Psikometri (Diğer)

Dergi: Journal of Psychometric Research Vol.3 Issue 1, pp.24-40

Belge Tipi: Makale

Yayın Tarihi: 3/2025

Belge Dili: İngilizce

Document ID (Org No.): 2001342639

pISSN: 3023-4646

Sayı

Yazarlar:

Özkan Saatçioğlu

Yayıncı:

Ataturk University

Ana Başlıklar:

Test | Ölçme ve Psikometri (Diğer)

Dergi:

Journal of Psychometric Research Vol.3 Issue 1, pp.24-40

Belge Tipi:

Makale

Yayın Tarihi:

3/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001342639

pISSN:

3023-4646