Measuring up: Rasch analysis of English reading comprehension test for informal education learners
DOI:
https://doi.org/10.12928/eltej.v8i1.12747Keywords:
Informal Education , Learners , Rasch Analysis , Reading ComprehensionAbstract
This study aims to evaluate the quality of English reading comprehension test instruments used in informal learning, especially as English literacy tests. With a quantitative approach, the analysis was carried out using the Rasch model through the Quest program on 30 multiple-choice questions given to 30 grade IX students from informal educational institutions in Bantul. The results of the analysis showed that although all questions were included in the fit category for the Rasch model, the level of reliability was relatively low, that was 0.52 for items and 0.39 for participants. In addition, 13.3% of the questions showed inconsistent results (misfit), this means that there is inconsistency in the results and quality of the questions that need to be improved. The analysis of the level of difficulty also showed that there were questions that were too easy or too difficult. These findings highlight the importance of revising the test items and the need to increase the number of participants and items to obtain more accurate measurement results. This study also provides practical implications regarding the need for continuous and planned instrument development in the context of informal education, to provide valid and reliable evaluation tools to measure students' literacy skills.
References
Anggia, H., & Habók, A. (2023). Textual complexity adjustments to the English reading comprehension test for undergraduate EFL students. Heliyon, 9(1). https://doi.org/10.1016/j.heliyon.2023.e12891
Aryadoust, V., Ng, L. Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487
Brown, H. D., & Abeywickrama, P. (2018). Language assessment: Principles and Classroom Practices (3rd ed.). Pearson/Longman.
Butterfuss, R., Kim, J., & Kendeou, P. (2020). Reading Comprehension. In Oxford Research Encyclopedia of Education. Oxford University Press. https://doi.org/10.1093/acrefore/9780190264093.013.865
Chong, S. W., & Reinders, H. (2022). Autonomy of English language learners: A scoping review of research and practice. Language Teaching Research. https://doi.org/10.1177/13621688221075812
Christensen, K. S., & Ammentorp, J. (2024). Rasch analysis of the self-efficacy (SE-12) questionnaire measuring clinical communication skills. PEC Innovation, 4. https://doi.org/10.1016/j.pecinn.2024.100296
Creswell, J. W. (2014). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (4th ed.). Sage Publications.
Dunn, K. J. (2024). Random-item Rasch models and explanatory extensions: A worked example using L2 vocabulary test item responses. Research Methods in Applied Linguistics, 3(3). https://doi.org/10.1016/j.rmal.2024.100143
Faradillah, A., & Febriani, L. (2021). Mathematical Trauma Students’ Junior High School Based on Grade and Gender. Infinity Journal, 10(1), 53–68. https://doi.org/10.22460/infinity.v10i1.p53-68
Johnson, M., & Majewska, D. (2022). Formal, non-formal, and informal learning: What are they, and how can we research them? https://www.cambridge.org/
Jones, I. D., & Brady, G. (2022). Informal Education Pedagogy Transcendence from the ‘Academy’ to Society in the Current and Post COVID Environment. Education Sciences, 12(1). https://doi.org/10.3390/EDUCSCI12010037
Luber, L., Fögele, J., & Mehren, R. (2020). How Do Students Experience a Deprived Urban Area in Berlin? Empirical Reconstruction of Students’ Orientations. Review of International Geographical Education Online, 10(4), 500–532. https://doi.org/10.33403/rigeo.763170
Morea, N., Kasprowicz, R. E., Morrison, A., & Silvestri, C. (2024). Diverse population, homogenous ability: The development of a new receptive vocabulary size test for young language learners in England using Rasch analysis. Research Methods in Applied Linguistics, 3(3). https://doi.org/10.1016/j.rmal.2024.100166
Nguyen, H. T. M. (2022). Reshaping Content Validation: A Case Study of a Reading Achievement Test for Third-year English Majors. International Journal of Language Testing, 12(1), 26-58.
Noroozi, S., & Karami, H. (2024). A Rasch-based validation of the University of Tehran English Proficiency Test (UTEPT). Language Testing in Asia, 14(1). https://doi.org/10.1186/s40468-024-00290-4
Polat, M. (2022). A Rasch Analysis of Rater Behaviour in Speaking Assessment. International Online Journal of Education and Teaching (IOJET), 7(3), 1126–1141. https://iojet.org/index.php/IOJET/article/view/902
Prabowo, M. Y., & Rahmadian, S. (2023). Equivalency Evidence of the English Competency Test Across Different Modes: A Rasch Analysis. Teflin Journal, 34(2), 301–319. https://doi.org/10.15639/teflinjournal.v34i2/301-319
Priyani, T., & Sugiharto, B. (2024). Analysis of biology midterm exam items using a comparison of the classical theory test and the Rasch model. JPBI (Jurnal Pendidikan Biologi Indonesia), 10(3), 939–958. https://doi.org/10.22219/jpbi.v10i3.34345
Rafatbakhsh, E., & Ahmadi, A. (2023). Predicting the difficulty of EFL reading comprehension tests based on linguistic indices. Asian-Pacific Journal of Second and Foreign Language Education, 8(1). https://doi.org/10.1186/s40862-023-00214-4
Rizbudiani, A. D., Jaedun, A., Rahim, A., & Nurrahman, A. (2021). Rasch model item response theory (IRT) to analyze the quality of mathematics final semester exam test on system of linear equations in two variables (SLETV). Al-Jabar: Jurnal Pendidikan Matematika, 12(2), 399-412. https://doi.org/10.24042/ajpm.v12i2.9939
Smith, A. (2021). COVID-19 and Informal Education: Considerations for Informal Learning During the Pandemic. International Journal of Multidisciplinary Perspectives in Higher Education, 6(1), 122–127.
Smith, A., & Seal, M. (2021). The contested terrain of critical pedagogy and teaching informal education in higher education. In Education Sciences 11(9). MDPI. https://doi.org/10.3390/educsci11090476
Srisang, P., Everatt, J., & Th, A. (2021). Lower and Higher-Level Comprehension Skills of Undergraduate EFL Learners and Their Reading Comprehension. LEARN Journal: Language Education and Acquisition Research Network, 14(1), 427–454. https://so04.tci-thaijo.org/index.php/LEARN/index
Stenner, A. J. (2023). Explanatory Models, Unit Standards, and Personalized Learning in Educational Measurement. In Explanatory Models, Unit Standards, and Personalized Learning in Educational Measurement. Springer Nature Singapore. https://doi.org/10.1007/978-981-19-3747-7
Subagja, S., Rubini, B., & Kurniasih, S. (2023). Development and Validation of a Test of Science Process Skills for Secondary Students on Cellular Living System Organization Matter: Rasch Model Analysis. Jurnal Penelitian Pendidikan IPA, 9(10), 9056–9062. https://doi.org/10.29303/jppipa.v9i10.5177
Tkacová, H., Králik, R., Tvrdoň, M., Jenisová, Z., & Martin, J. G. (2022). Credibility and Involvement of social media in Education—Recommendations for Mitigating the Negative Effects of the Pandemic among High School Students. International Journal of Environmental Research and Public Health, 19(5). https://doi.org/10.3390/ijerph19052767
Toker, T., & Seıdel, K. (2023). A Mixture Rasch Model Analysis of Data from a Survey of Novice Teacher Core Competencies. International Journal of Contemporary Educational Research, 10(1), 147–156. https://doi.org/10.52380/ijcer.2023.10.1.349
William, F. K. A. (2024). Mastering validity and reliability in academic research: Meaning and significance. International Journal of Research Publications, 144(1), 287-292. https://doi.org/10.47119/ijrp1001441320246160
Winarti, A., & Mubarak, A. (2020). Rasch Modeling: A Multiple-Choice Chemistry Test. Indonesian Journal on Learning and Advanced Education (IJOLAE), 2(1), 1–9. https://doi.org/10.23917/ijolae.v2i1.8985
Wright, B. D., & Mok, M. M. C. (2004). An overview of the family of Rasch measurement models. In E. V. Smith Jr. & R. M. Smith (Eds.). In Introduction to Rasch measurement theory, models and applications 1(1), 1-24.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Arandha May Rachmawati, Agus Widyantoro
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish in ELTEJ agree to the following terms: Authors retain copyright and grant the ELTEJ right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material) the work for any purpose, even commercially with an acknowledgement of the work's authorship and initial publication in ELTEJ. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in ELTEJ. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
