Sie befinden Sich nicht im Netzwerk der Universität Paderborn. Der Zugriff auf elektronische Ressourcen ist gegebenenfalls nur via VPN oder Shibboleth (DFN-AAI) möglich. mehr Informationen...
An Examination of Rater Drift Within a Generalizability Theory Framework
Ist Teil von
Journal of educational measurement, 2009-03, Vol.46 (1), p.43-58
Ort / Verlag
Malden, USA: Blackwell Publishing Inc
Erscheinungsjahr
2009
Quelle
Applied Social Sciences Index & Abstracts (ASSIA)
Beschreibungen/Notizen
The present study examined the long-term usefulness of estimated parameters used to adjust the scores from a performance assessment to account for differences in rater stringency. Ratings from four components of the USMLE® Step 2 Clinical Skills Examination data were analyzed. A generalizability-theory framework was used to examine the extent to which rater-related sources of error could be eliminated through statistical adjustment. Particular attention was given to the stability of these estimated parameters over time. The results suggest that rater stringency estimates obtained at a point in time and then used to adjust ratings over a period of months may substantially decrease in usefulness. In some cases, over several months, the use of these adjustments may become counterproductive. Additionally, it is hypothesized that the rate of deterioration in the usefulness of estimated parameters may be a function of the characteristics of the scale.