Investigating the Feasibility of Generic Scoring Models of E-rater for TOEFL iBT Independent Writing Tasks

• 영문명: Investigating the Feasibility of Generic Scoring Models of E-rater for TOEFL iBT Independent Writing Tasks
• 발행기관: 팬코리아영어교육학회(구 영남영어교육학회)
• 저자명: YongWon Lee
• 간행물 정보: 『영어교육연구』제28권 1호, 101~122쪽, 전체 22쪽
• 주제분류: 사회과학 > 교육학
• 파일형태: PDF
• 발행일자: 2016.03.31

영문 초록

The current study reports the findings from Phase 2 of a larger research study undertaken toinvestigate the feasibility of using generic scoring models for e-rater in the context of scoringessays for independent writing tasks for TOEFL CBT and TOEFL iBT. In Phase 1, sixdifferent variants of generic and hybrid scoring models of e-rater were created based ontransformed writing data from three different samples of TOEFL CBT prompts (n1=20, n2=20,n3= 40) with the help of ETS (Educational Testing Service) staff and then evaluated on aseparate sample of seven TOEFL CBT prompts (Lee, 2016). In the present investigation, thesesix generic/hybrid models were used, along with prompt-specific models, to score a total of3,126 essays written for two TOEFL iBT independent writing tasks from a field study and theirperformance was evaluated. Results of the analysis showed that (a) there were relatively smallscore variations among different automated scoring models and (b) similar levels of scoreagreement were achieved between the human-human rater pair and various human-automatedrater pairs, although the prompt-specific model behaved most similarly to the human raters. Interms of criterion-related validity of scores, the human rater scores turned out to be somewhatbetter indicators of test-takers’ overall ESL (English as a Second Language) languageproficiency than the automated scores in general. Nevertheless, the comparative advantage invalidity of human rater scores (over automated scores) seemed to diminish significantly, whenmore direct writing measures, such as scores for TOEFL CBT independent writing tasks, wereused as criterion measures.

목차

I. INTRODUCTION

II. MODEL BUILDING FOR AUTOMATED ESSAY SCORING

III. METHOD

IV. RESULT AND DISCUSSION

V. SUMMARY AND CONCLUSIONS

ACKNOWLEDGEMENTS

REFERENCES