Design of information retrieval experiments: the sufficient topic set size for providing an adequate level of confidence

Bekir Taner DİNÇER

Makale detayları

Design of information retrieval experiments: the sufficient topic set size for providing an adequate level of confidence

Yazarlar:

Bekir Taner DİNÇER

Yayıncı:

TUBITAK

Dergi:

Turkish Journal of Electrical Engineering and Computer Science 2013 Vol.21 Issue Sup.2, pp.2218-2232

Belge Tipi:

Makale

Yayın Tarihi:

11/2012

Belge Dili:

Türkçe

Document ID (Org No.):

2000743425

Sıklık:

İki Aylık

eISSN:

1303-6203

pISSN:

1300-0632

Özet:

 

In the current design of information retrieval (IR) experiments, a sample of  50 topics is generally agreed to be sufficient in size to perform dependable  system evaluations. This article presents the detailed and formal  explanation of how the second fundamental theorem of probability, the  central limit theorem, can be used for the estimation of the sufficient size  of a topic sample. The research performed in this article, using past Text  Retrieval Conference data, reveals that, on average, 50 topics will be  sufficient to provide a confidence level at or above 95% if the null  hypothesis of an equal population mean average precision (MAP) (H0) is  rejected for 2 IR systems having an observed difference in the MAP of 0.035  or more, whereas, in contrast, previous empirical research suggests a  difference in the MAP of 0.05 or more. This study also shows that, for  individual system pairs, the sample size required to provide 95% confidence on a declared significance may range from a size as small as 10  to a size as large as 722. Thus, for the design of IR experiments, it agrees  with the common view that relying on average figures as a rule of thumb may  well be misleading.

 

In the current design of information retrieval (IR) experiments, a sample of  50 topics is generally agreed to be sufficient in size to perform dependable  system evaluations. This article presents the detailed and formal  explanation of how the second fundamental theorem of probability, the  central limit theorem, can be used for the estimation of the sufficient size  of a topic sample. The research performed in this article, using past Text  Retrieval Conference data, reveals that, on average, 50 topics will be  sufficient to provide a confidence level at or above 95% if the null  hypothesis of an equal population mean average precision (MAP) (H0) is  rejected for 2 IR systems having an observed difference in the MAP of 0.035  or more, whereas, in contrast, previous empirical research suggests a  difference in the MAP of 0.05 or more. This study also shows that, for  individual system pairs, the sample size required to provide 95% confidence on a declared significance may range from a size as small as 10  to a size as large as 722. Thus, for the design of IR experiments, it agrees  with the common view that relying on average figures as a rule of thumb may  well be misleading.

Makale detayları

Design of information retrieval experiments: the sufficient topic set size for providing an adequate level of confidence

Yazarlar: Bekir Taner DİNÇER

Yayıncı: TUBITAK

Dergi: Turkish Journal of Electrical Engineering and Computer Science 2013 Vol.21 Issue Sup.2, pp.2218-2232

Belge Tipi: Makale

Yayın Tarihi: 11/2012

Belge Dili: Türkçe

Document ID (Org No.): 2000743425

Sıklık: İki Aylık

eISSN: 1303-6203

pISSN: 1300-0632

Sayı

Yazarlar:

Bekir Taner DİNÇER

Yayıncı:

TUBITAK

Dergi:

Turkish Journal of Electrical Engineering and Computer Science 2013 Vol.21 Issue Sup.2, pp.2218-2232

Belge Tipi:

Makale

Yayın Tarihi:

11/2012

Belge Dili:

Türkçe

Document ID (Org No.):

2000743425

Sıklık:

İki Aylık

eISSN:

1303-6203

pISSN:

1300-0632