Study on Inter-raters Reliability under the Latent Trait Model through Derangement
Date Issued
2009
Date
2009
Author(s)
Liu, Shan-Pang
Abstract
We investigate the inter-rater reliability when the ability of large number of examinees is classified to ordinal grade by raters through derangement. The polychoric correlation coefficient is used as inter-rater reliability when the latent trait model (LTM) is assumed.o ensure at least hundreds examinees is graded by two raters when the number of raters is around a few hundred and the number of examinees is around three hundred thousand, we consider assigning examinees to raters through derangement. Under this setting, it is found that all raters are grouped into several cycles. Through analytic argument and simulation, it is found that the number of group is often not more than ten, the probability of getting at least one cycle of size 2 or 3 is close to 0.59, and the size of largest cycle is often exceeding one hundred. It also finds that the distributions of latent trait of examinees by different raters are close to each other up to a location shift.nder the assumption of the LTM, the discriminate parameter in models can be regard as the accuracy of rating.The correlation between the grades given by raters and the latent trait of examinees was affected by the interaction of the thresholds and discriminate parameter. The correlation coefficient of perspective latent trait variables of two raters is the product of their discriminate parameter, and polychoric correlation coefficient can be estimated by two stages method. The parameter of the thresholds of raters were estimated by the proportion of rating, while as discriminate parameter can be estimates through appropriate derangement.inally according to the result of research, we propose the summary and some suggestions.
Subjects
reliability
polychoric correlation coefficient
latent trait model
thresholds
derangement
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-98-R92221028-1.pdf
Size
23.53 KB
Format
Adobe PDF
Checksum
(MD5):59ef9b1d82d591871801f007ece1a0da