Abstract
While large-scale language and writing assessments benefit from a wealth of literature on the reliability and validity of specific tests and rating procedures, there is comparatively less literature that explores the specific language of second language writing rubrics. This paper provides an analysis of the language of performance descriptors for the public versions of the TOEFL and IELTS writing assessment rubrics, with a focus on linguistic agency encoded by agentive verbs and language of ability encoded by modal verbs can and cannot. While the IELTS rubrics feature more agentive verbs than the TOEFL rubrics, both pairs of rubrics feature uneven syntax across the band or score descriptors with either more agentive verbs for the highest scores, more nominalization for the lowest scores, or language of ability exclusively in the lowest scores. These patterns mirror similar patterns in the language of college-level classroom-based writing rubrics, but they differ from patterns seen in performance descriptors for some large-scale admissions tests. It is argued that the lack of syntactic congruity across performance descriptors in the IELTS and TOEFL rubrics may reflect a bias in how actual student performances at different levels are characterized.
References
Ahearn, Laura M. 2001. Language and agency. Annual Review of Anthropology 30. 109–137. DOI: 10.1146/annurev.anthro.30.1.109
Alderson, J. Charles. 1991. Bands and scores. In J. Charles Alderson & Brian North (eds.), Language testing in the 1990s: The communicative legacy, 71–86. London: Modern English Publications/British Council.
Alderson, J. Charles, Neus Figueras, Henk Kuijper, Guenter Nold, Sauli Takala & Claire Tardieu. 2004. The development of specifications for item development and classification within the Common European Framework of Reference for Languages: Learning, teaching, assessment: Reading and listening: Final report of the Dutch CEF Construct Project. Lancaster University. http://eprints.lancs.ac.uk/44/1/final_report.pdf.
Bachman, Lyle F. 1990. Fundamental considerations in language testing. Oxford: Oxford University Press.
Banerjee, Jayanti, Xun Yan, Mark Chapman & Heather Elliott. 2015. Keeping up with the times: Revising and refreshing a rating scale. Assessing Writing 26. 5–19. DOI: 10.1016/j.asw.2015.07.001
Becker, Anthony. 2010. Examining rubrics used to measure writing performance in US intensive English programs. The CATESOL Journal 22(1). 113–130.
Billig, Michael. 2008. The language of critical discourse analysis: The case of nominalization. Discourse & Society 19(6). 783–800. DOI: 10.1177/0957926508095894
Brindley, Geoff. 1998. Describing language development? Rating scales and SLA. In Lyle F. Bachman & Andrew D. Cohen (eds.), Interfaces between second language acquisition and language testing research, 112–140. New York: Cambridge University Press.
Bucholtz, Mary & Kira Hall. 2005. Identity and interaction: A sociocultural linguistic approach. Discourse Studies 7(4–5). 585–614. DOI: 10.1177/1461445605054407
Calkins, Lucy McCormick. 1994. The art of teaching writing (new ed.). Portsmouth: Heinemann.
Chakroff, Aleksandr, Kyle A. Thomas, Omar S. Haque & Liane Young. 2015. An indecent proposal: The dual functions of indirect speech. Cognitive Science 39(1). 199–211. DOI: 10.1111/cogs.12145
Cotton, Fiona & Kate Wilson. 2011. An investigation of examiner rating of coherence and cohesion in the IELTS Academic Writing Task 2. https://www.ielts.org/-/media/research-reports/ielts_rr_volume12_report6.ashx
Covill, Amy E. 2012. College students’ use of a writing rubric: Effect on quality of writing, self-efficacy, and writing practices. Journal of Writing Assessment 5(1). http://journalofwritingassessment.org/article.php?article=60
Crusan, Deborah. 2010. Assessment in the second language writing classroom. Ann Arbor: University of Michigan Press.
Davies, Alan. 2008. Assessing academic English. Testing English proficiency 1950–89: The IELTS solution. Cambridge: Cambridge University Press.
Dowty, David. 1991. Thematic proto-roles and argument selection. Language 67(3). 547–619.
Dryer, Dylan. 2013. Scaling writing ability: A corpus-driven inquiry. Written Communication 30(1). 3–35. DOI: 10.1177/0741088312466992
Duranti, Alessandro. 2004. Agency in language. In Alessandro Duranti (ed.), A companion to linguistic anthropology, 451–473. Malden, MA: Blackwell. DOI: 10.1002/9780470996522.ch20
Ehrlich, Susan. 2001. Representing rape: Language and sexual consent. New York: Routledge.
Fausey, Caitlin M. & Lera Boroditsky. 2010. Subtle linguistic cues influence perceived blame and financial liability. Psychonomic Bulletin & Review 17(5). 644–650. DOI: 10.3758/PBR.17.5.644
Fausey, Caitlin M., Bria L. Long, Aya Inamori & Lera Boroditsky. 2010. Constructing agency: The role of language. Frontiers in Psychology 1. 162. DOI: 10.3389/fpsyg.2010.00162
Fillmore, Charles J. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences 280. 20–32. DOI: 10.1111/j.1749-6632.1976.tb25467.x
Fillmore, Charles J. & Collin Baker. 2010. A frames approach to semantic analysis. In Bernd Heine & Heiko Narrog (eds.), The Oxford handbook of linguistic analysis, 313–339. Oxford: Oxford University Press. DOI: 10.1093/oxfordhb/9780199544004.013.0013
Fowler, Roger, Bob Hodge, Günther Kress & Tony Trew. 1979. Language and control. London: Routledge.
Fowler, Roger. 1991. Language in the news: Discourse and ideology in the press. London: Routledge.
Fox, Janna D. 2007. Language testing reconsidered. Ottawa: University of Ottawa Press.
Golder, Katherine, Kenneth Reeder & Sarah Fleming. 2012. Determination of appropriate IELTS Writing and Speaking Band Scores for admission into two programs at a Canadian post-secondary polytechnic institution. Canadian Journal of Applied Linguistics/Revue canadienne de linguistique appliquée 14(1). 222–250.
Hambleton, Ronald K.& Mary Pitoniak. 2006. Setting performance standards. In Robert L. Brennan (ed.), Educational measurement (4th ed.), 433–470. Westport, CT: Praeger.
Hawkey, Roger & Fiona Barker. 2004. Developing a common scale for the assessment of writing. Assessing Writing 9(2). 122–159. DOI: 10.1016/j.asw.2004.06.001
Henley, Nancy M., Michelle Miller & Jo Anne Beazley. 1995. Syntax, semantics, and sexual violence: Agency and the passive voice. Journal of Language and Social Psychology 14(1–2). 60–84. DOI: 10.1177/0261927X95141004
Jeffery, Jill V. 2009. Constructs of writing proficiency in US state and national writing assessments: Exploring variability. Assessing Writing 14(1). 3–24. DOI: 10.1016/j.asw.2008.12.002
Knoch, Ute. 2007. ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing 12(2). 108–128. DOI: 10.1016/j.asw.2007.07.002
Knoch, Ute. 2009. Diagnostic assessment of writing: A comparison of two rating scales. Language Testing 26(2). 275–304. DOI: 10.1177/0265532208101008
Knoch, Ute. 2011. Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from? Assessing Writing 16(2). 81–96. DOI: 10.1016/j.asw.2011.02.003
Knoch, Ute, Susy Macqueen & Sally O'Hagan. 2014. An investigation of the effect of task type on the discourse produced by students at various score levels in the TOEFL iBT® writing test. ETS Research Report Series 2014(2). DOI: 10.1002/ets2.12038
Kuiken, Folkert & Ineke Vedder. 2014. Raters’ decisions, rating procedures and rating scales. Language Testing. 31(3). 279–284. DOI: 10.1177/0265532214526179
LaFrance, Marianne & Eugene Hahn. 1994. The disappearing agent: Gender stereotypes, interpersonal verbs and implicit causality. In Camille Roman, Suzanne Juhasz & Cristianne Miller (eds.), The women and language debate: A sourcebook, 348–362. New Brunswick, NJ: Rutgers University Press.
Li, Jinrong & Peggy Lindsey. 2015. Understanding variations between student and teacher application of rubrics. Assessing Writing 26. 67–79. DOI: 10.1016/j.asw.2015.07.003
Lumley, Tom. 2002. Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing 19(3). 246–276. DOI:10.1191/0265532202lt230oa
Matsuda, Paul Kei, & Jill V. Jeffery. 2012. Voice in student essays. In Ken Hyland & Carmen Sancho Guinda (eds.), Stance and voice in written academic genres, 151–165. New York: Palgrave Macmillan. DOI: 10.1057/9781137030825_10
Messick, Samuel. 1988. Meaning and values in test validation: The science and ethics of assessment. ETS Research Report Series 1988(2). DOI:10.1002/j.2330-8516.1988.tb00303.x
Morales, Meghan Corella & Jin Sook Lee. 2015. Stories of assessment: Spanish–English bilingual children's agency and interactional competence in oral language assessments. Linguistics and Education 29. 32–45. DOI: 10.1016/j.linged.2014.10.008
North, Brian. 2007. The CEFR illustrative descriptor scales. The Modern Language Journal 91(4). 656–659. DOI: 10.1111/j.1540-4781.2007.00627_3.x
North, Brian& Günther Schneider. 1998. Scaling descriptors for language proficiency scales. Language Testing 15(2). 217–262. DOI: 10.1177/026553229801500204
[OED =] Oxford English Dictionary (3rd ed.). Oxford: Oxford University Press. https://www.oed.com/
Schaefer, Edward. 2008. Rater bias patterns in an EFL writing assessment. Language Testing 25(4). 465–493. DOI: 10.1177/0265532208094273
Spandel, Vicki. 2006. In defense of rubrics. English Journal 96(1). 19–22.
Upshur, John A. & Carolyn E. Turner. 1995. Constructing rating scales for second language tests. ELT Journal49(1). 3–12. DOI: 10.1093/elt/49.1.3
Winke, Paula & Hyojung Lim. 2015. ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing 25. 38–54. DOI: 10.1016/j.asw.2015.05.002
Wodak, Ruth & Michael Meyer (eds.). 2009. Methods of critical discourse analysis (2nd ed.). Los Angeles, CA: Sage.