New Measures of Academic Collocation Knowledge: Wordlist-Based Test Development and Argument-Based Validation
Knowledge of academic vocabulary is essential for second or foreign language learners who are preparing for studying at English-medium universities. This vocabulary comprises single words (e.g., approximate, component and establish) and multiword units, including frequent two-word academic collocations. These items occur across different disciplines (e.g., ultimate goal and key element) and have been identified through corpus-based research which has resulted in several word lists (e.g., Ackermann & Chen, 2013; Lei & Liu, 2018). While there are tests which target knowledge of general academic vocabulary based on single word lists, there is a lack of tests of academic collocation knowledge. Such assessment is beneficial for troubleshooting problems with academic collocations at an early stage so that support for learning these items can be provided in a timely manner. This study aims to fill this gap by developing and validating two separate measures of recognition and recall knowledge of general academic collocations for diagnostic purposes.
To that end, this research first adapted an existing framework from Nation (2016) to evaluate two published lists of academic collocations. The evaluation was to select the most representative items for developing two Academic Collocation Tests (ACTs): the recognition test (multiple-choice format) and the recall test (gap-fill format). The test development was guided by an evidence-centred design framework (Mislevy & Yin, 2013). The validation process then employed an argument-based approach (Kane, 2013) to collect validity evidence. A total of 343 tertiary students (233 in Vietnam and 110 in New Zealand) took part in this study. They completed a background questionnaire which included demographic information and language proficiency (e.g., IELTS scores and learning experience). They took both of the ACTs and also completed the Vocabulary Size Test (VST) (Nation & Beglar, 2007), which was used as a measure of general vocabulary knowledge. Forty-four of the participants took part in a post-test interview to share their reflections on the tests and re-took the ACTs verbally for the assessment of test-retest reliability. Data gathering took place via online platforms because of the COVID-19 pandemic.
Five main findings arose from this thesis. First, results of the wordlist evaluation process indicated that the Academic Collocation List (Ackermann & Chen, 2013) provided the best source of items for testing purposes in the present study. Second, statistical analyses showed that test items developed from that list worked well together to measure the intended construct. Third, reflections from test-takers revealed that the ACTs allowed them to demonstrate their knowledge of academic collocations, although the online test-taking condition was not ideal. Fourth, the ACTs were found to be highly reliable, as evidenced by high reliability indices. Finally, scores on the ACTs were positively correlated with scores on other tests of similar constructs, including the VST and IELTS. The relationship between ACT scores and time spent studying English was also significant. That said, ACT scores were not significantly correlated with the frequency of academic collocations and time spent studying in an English-speaking context.
Based on these findings, this thesis offers pedagogical implications to support English for Academic Purposes (EAP) teaching and learning with improving academic collocation knowledge. This study advances the field of vocabulary assessment by applying test development and validation frameworks to create rigorous tests for EAP. It also provides a model for evaluating word lists of multiword units, which lays the foundation for a similar practice in wordlist studies and supports the further application of wordlist-based test development.