Individual Feedback to Enhance Rater Training: Does It Work?  

Oleh:

Elder, Cathie ; Knoch, Ute ; Barkhuizen, Gary ; Randow, Janet von

Jenis: Article from Journal - e-Journal
Dalam koleksi: Language Assessment Quarterly vol. 2 no. 3 (2005), page 175 — 196.
Topik: Study Investigating; Analytically Scored Writing; Enhance Rater Training
Fulltext: LANGUAGE ASSESSMENT QUARTERLY, 2(3), 175–196.pdf (643.02KB)

Isi artikel

Research on the utility of feedback to raters in the form of performance reports has produced mixed findings (Lunt, Morton, & Wigglesworth, 1994; Wigglesworth, 1993) and has thus far been trialled only in oral assessment contexts. This article reports on a study investigating raters’ attitudes and responsiveness to feedback on their ratings of an analytically scored writing task administered in the context of a large-scale university writing assessment program. After participating in an online refresher training program, 50 scripts were rated independently by 8 raters.Amultifaceted Rasch analysis (using FACETS; Linacre&Wright, 1993) was used to generate individualized reports on each rater’s pattern of bias with respect to particular categories of the rating scale as well as their overall consistency and relative severity in relation to others in the group. Reports were explicated at a group briefing session, and raters were asked to attend to the feedbackwhenscoring a further batch of 60 scripts. Qualitative data on rater attitudes to the feedback were also elicited via questionnaire. Results showedthat most raters found the feedback useful and showedenhanced awareness of their rating behaviour. A comparison of ratings before and after feedback (again using Rasch analysis) revealed that many were able to modify their scoring, resulting in greater intragroup consistency and a reduced incidence of item bias. There was nevertheless variation in receptivity to feedback, and questionnaire responses are used to suggest reasons for this. Gains in rater reliability moreover had the effect of reducing the test’s discriminatory power suggesting that the costs of implementing this rather elaborate approach to training may outweigh the benefits in this instance. Further refinements to this approach to training are recommended, and research into the effects of feedback in different testing contexts is advocated, particularly on high stakes tests where the consequences of rater inconsistency and bias are more grave.

Opini Anda

Klik untuk menuliskan opini Anda tentang koleksi ini!

Kembali