Designing and Validating an Integrated Reading-Writing (IRW) Test for First-year Students in the Humanities and Social Sciences in New Zealand

<p dir="ltr">Academic reading and writing are dominant literacy skills important for students' success at university. L1 and EAL students entering university should be prepared for the requirements of courses, and competent in reading academic texts and engaging with texts in writing. Assessing students' academic reading and writing skills is critical to evaluating their preparedness for university courses, especially for first-year EAL students. Academic tests for such purposes generally aim to evaluate students' preparedness for first-year courses. However, closer inspection reveals that tests often do not authentically align with the requirements of TLU domains or consider the dynamic interrelationship of reading and writing skills. Therefore, such tests may not be sufficient to gauge students’ readiness for university-level reading and writing. To remedy this, some tests use integrated reading-writing (IRW) assessment tasks which are considered authentic, as they require writing based on reading sources. However, there are many weaknesses with these tests.</p><p dir="ltr">This study develops an IRW test closely aligning with TLU domain requirements to evaluate students' preparedness for first-year humanities and social science (HSS) courses at Victoria University of Wellington (VUW). The test’s primary purposes are: as a post-entry language assessment (PELA) after admission to the university to assist targeted support; to offer suggestions for EAP exit testing. It is hoped that assessing students' reading-writing abilities using a content-responsible IRW test would raise students' awareness of first-year HSS course demands.</p><p dir="ltr">Weir's (2005) socio-cognitive framework informed test development and validation. Context, cognitive, and scoring validity parameters were investigated through document analysis, lecturer interviews, test-taker interviews, task representation questionnaires, raters’ think-aloud protocols, and multi-faceted Rasch measurement analysis. Domain analysis revealed a need for student engagement in high-level reading skills, incorporating source ideas into writing for effective argument construction. These requirements informed the IRW test, which asked test-takers to write an essay based on two source texts adapted from a chapter and a journal article. An analytical marking rubric was developed, validated, and revised.</p><p dir="ltr">It was found that students’ previous test experiences created difficulties in applying reading and writing skills in an academic context, highlighting the importance of tests that are aligned with TLU. The rating scale generally functioned effectively, despite some rater bias in reading-associated criteria. Raters were affected by latent criteria in some instances. Furthermore, evaluation of the relationship between test-takers' reading and IRW test scores suggests that reading comprehension was important but not sufficient for success. The IRW test required higher-order skills that elude evaluation through reading comprehension alone. IRW tests aiming to evaluate students' higher-order literacies for university readiness should include tasks that are similar to TLU tasks.</p>