Investigating the score dependability and decision dependability of the GEPT listening test: A multivariate generalizability theory approach
Journal
English Teaching and Learning
Journal Volume
40
Journal Issue
1
Pages
79-111
Date Issued
2016
Author(s)
Abstract
Classical test theory (CTT) reliability estimates have been commonly used and accepted as a useful way of estimating reliability. However, several limitations in the CTT model for reliability estimates have been identified, particularly within the context of criterion-referenced tests. Generalizability (G) theory is one of the useful techniques which can address these limitations and help to examine dependability issues. The purpose of this paper is to investigate the score dependability and decision dependability of the listening section of the General English Proficiency Test (GEPT). One form of the intermediate-level GEPT listening test was administered to 609 Taiwanese college freshmen. The responses were then coded for the ability to understand literal meaning (operationalized as literal-explicit items) and the ability to understand pragmatic meaning (operationalized as pragmatic-implicit items). A multivariate G theory analysis was conducted to examine the relative effects of multiple sources of variance in the listening scores as well as the dependability of the composite scores for the listening test and decisions being made at the current cut-off score. It was found that both literal-explicit and pragmatic-implicit items worked almost equally well. The current predetermined cut-off score yielded a moderately high dependability of decisions.
SDGs
Type
journal article