Mapping the KKU-AELT Reading Test to the CEFR: A Standard-Setting Study

Sivakorn Tangsakul; Kornwipa Poonpon

doi:10.66947/pasaa.v73ispc.2262

Authors

Sivakorn Tangsakul
Kornwipa Poonpon Khon Kaen University https://orcid.org/0000-0002-8349-2834

DOI:

https://doi.org/10.66947/pasaa.v73ispc.2262

Keywords:

CEFR, academic reading, language test, Thai higher education, KKU-AELT

Abstract

This standard-setting study aimed to align the Khon Kaen University Academic English Reading Test (KKU-AELT), a locally developed reading test, to the CEFR. The study adopted the staged approach (Council of Europe, 2009), involving familiarization, specification, standardization training, standard setting, and validation. The item-descriptor-matching method, an item response theory-based and test-centered approach (Ferrara et al., 2008), was employed. Five panelists participated in the standard setting process. The instruments included a judgment sheet for the ordered-item booklet, a judgment sheet for the booklet, and three feedback questionnaires. The data obtained were analyzed using classical statistics, i.e., frequency, percentage, mean, standard deviation, and standard error judgment. The intraclass correlation coefficient and Cronbach’s alpha were also employed for rater agreement. A high level of inter-rater consistency was found across three rounds of judgment. The results revealed the final cut-off scores for the reading test, with the score ranges corresponding to the CEFR levels delineated as A1 (1-15), A2 (16-31), B1 (32-54), B2 (55-83), and C1 (84-100). The CEFR interpretation of academic reading scores provides key insights for the ongoing improvement of students’ English proficiency.

References

Alderson, J. C., Figueras, N., Kuijper, H., Nold, G., Takala, S., & Tardieu, C. (2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: The experience of the Dutch CEFR Construct Project. Language Assessment Quarterly, 3(1), 3 –30. https://eprints.lancs.ac.uk/id/eprint/3909/

Ativorakun, C. & Wudthayagorn, J. (2018). Mapping Srinakharinwirot University –Standardized English Test (SWU-SET) onto the Common European Framework of Reference (CEFR). Suranaree Journal of Social Science, 12(2), 69 –84. https://doi.org/10.55766/CTUU4836

Brunfaut, T. & Harding, L. (2014). Linking the GEPT listening test to the Common European Framework of Reference. Taiwan: Language Training and Testing Centre. https://www.lttc.ntu.edu.tw/lttc-gept-grants/RReport/RG05.pdf

Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press.

Council of Europe. (2009). Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment: Manual. Council of Europe, Language Policy Division.

Ferrara, S., Perie, M. & Johnson, E. (2008). Matching the judgmental task with standard setting panelist expertise: The item-descriptor (ID) matching method. Journal of Applied Testing Technology, 9(1). 1–20.

Figueras, N. (2012). The impact of the CEFR. ELT Journal, 66(4), 477–485. https://doi.org/10.1093/elt/ccs037

Figueras, N., North, B., Takala, S., Verhelst, N., & Van Avermaet, P. (2005). Relating examinations to the Common European Framework: A manual. Language Testing, 22(3), 261–279. https://doi.org/10.1191/0265532205lt308oa

Fulcher, G. (2010). Practical language testing. Hodder Education.

Graduate School of Khon Kaen University. (2022). Graduate student manual 2022. https://app.gs.kku.ac.th/images/img/news/announcement/2-1GS-77-66.pdf

Harsch, C., & Martin, G. (2012). Comparing CEF-related and independent writing proficient ratings: An empirical study. Language Testing, 29(3), 345–366

Jaturapitakkul, N. & Todd, R.W. (2018). Test of English for Thai engineers and technologists (TETET). Technical Report. https://sola.kmutt.ac.th/tetet/doc/TETET_Technical_report_NJ_RT_Complete.pdf

Jin, Y., Wu, Z., Alderson, C. & Song, W. (2017). Developing the China Standard of English: Challenges at macropolitical and micropolitical levels. Language Testing in Asia, 7(1), 1–19. https://doi.org/10.1186/s40468-017-0032-5

Kaftandjieva, F. (2004). Standard setting. Section B of the Reference Supplement to the preliminary version of the Manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Council of Europe.

Kaftandjieva, F., & Takala, S. (2002). Relating the Finnish Matriculation Examination English Test results to the CEF scales. Paper presented at the Helsinki Seminar on Linking Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment, Helsinki, Finland.

Knoch, U., & Frost, K. (2016). Linking the GEPT writing sub-test to the Common European Framework of Reference (CEFR). Language Training and Testing Centre. https://www.lttc.ntu.edu.tw/lttc-gept-grants/RReport/RG08.pdf

Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55.

Nakanitanon, P. (2021). Linking an English proficiency test to the CEFR: Setting validity cut scores. The New English Teacher, 15(1), 53–64.

Nickerson, R. S. (2004). Cognition and chance: The psychology of probabilistic reasoning. Lawrence Erlbaum Associates.

North, B. (2007). The CEFR illustrative scales. Modern Language Journal, 94(4), 656–659. http://www.jstor.org/stable/4626092

Office of the Basic Education Commission. (2014). Guidelines for practices by Ministry of Education: English education policy reform. Office of the Basic Education Commission, Ministry of Education. http://old.drs.ac.th/ext/tch_data/tch_ 02.pdf

Office of the Higher Education Commission. (2024). Policy of upgrading English education standards of higher education institutions. Office of the Higher Education Commission, Ministry of Higher Education, Science, Research and Innovation. https://www.ops.go.th/th/ches-downloads/edu-standard/item/9625-2567

O’Sullivan, B. (2015). Linking the Aptis reporting scales to the CEFR. Technical Report. https://www.britishcouncil.org/exam/aptis/research/

publications/reporting-scales

Poonpon, K. (2021). Test takers’ perceptions of design and implementation of an online language testing system at a Thai university during the COVID-19 pandemic. PASAA, 62, 1–28. https://doi.org/10.58837/CHULA.PASAA.62.1.1

Reynolds, C. R., Altmann, R. A., & Allen, D. N. (2021). The meaning of test scores. In C. R. Reynolds, R. A. Altmann, & D. N. Allen (Eds.), Mastering modern psychological testing (pp. 91–131). Springer. https://doi.org/10.1007/978-3-030-59455-8_3

Rovinelli, R.J. & Hambleton, R.K. (1997). On the use of content specialists in the assessment of criterion-referenced test item validity. Dutch Journal of Educational Research, 2, 49–60.

Shin, S. Y., Sanonguthai, S., & Tangkiengsirisin, S. (2022). Aligning the CEFR levels with the TU-GET CBT scores [Technical report]. Thammasat University. https://drive.google.com/drive/folders/1XDcUnxZwBpMaCdtjuZpn8i6bHx489ynM?fbclid=IwAR0UW-09Zu0uwWTYlfzz9etUAXCKk2kcqggCAMAWfKyDWBiAVGpB5q22SA

Srisawat, C. & Poonpon, K. (2023). Revision of an academic English writing rubric for a graduate school admission test. PASAA, 65, 234–262. https://doi.org/10.58837/CHULA.PASAA.65.1.9

Tannenbaum, R. J., & Baron, P. A. (2011). Mapping TOEFL® ITP scores onto the Common European Framework of Reference (Research Memorandum ETS RM-11-33). ETS.

Tannenbaum, R. J., & Baron, P. A. (2015). Mapping scores from the TOEFL Junior Comprehensive Test onto the Common European Framework of Reference (TOEFL Research Report No. RM-15-13). ETS.

Tannenbaum, R. J., & Wylie, E. C. (2005). Mapping English language proficiency test scores onto the Common European Framework (TOEFL Research Report No. RR-80). ETS.

Tannenbaum, R. J., & Wylie, E. C. (2008). Linking English-language test scores onto the Common European Framework of Reference: An application of standard-setting methodology (TOEFL iBT Research Report iBT-06). ETS.

Thongyoi, K., & Poonpon, K. (2020). Phrasal complexity measures as predictors of EFL university students’ English academic writing proficiency. rEFLections, 27(1), 44–61. https://doi.org/10.61508/refl.v27i1.241750

Wu, J. R., & Wu, R. Y. F. (2010). Relating the GEPT reading comprehension tests to the CEFR. In W. Martyniuk (Ed.), Aligning tests with the CEFR (Studies in Language Testing, Vol. 33, pp. 204–224). Cambridge University Press.

Wudthayagorn, J. (2018). Mapping the CU-TEP to the Common European Framework of Reference (CEFR). LEARN Journal: Language Education and Acquisition Research Network, 11(2), 163–180. https://so04.tci-thaijo.org/index.php/LEARN/article/view/161641