Bridging HOTS and character education in reading assessments: A study on validity, reliability, and Pancasila integration
DOI:
https://doi.org/10.33474/j-reall.v7i1.24707Keywords:
character education, HOTS, Pancasila, reading comprehension, R&D methodAbstract
Very little research on developing reading assessments has systematically integrated higher-order thinking skills (HOTS) and character building in an original, valid, and reliable reading comprehension test in English. This study addresses that gap by modeling the development of such an instrument through a research and development (R&D) design comprising four stages: preliminary study, development, evaluation-revision, and pre-implementation. Within these stages, a test blueprint based on reading sub-skills and HOTS indicators was created, and Pancasila student profile guided the selection of the four reading texts. The 40 multiple-choice questions were analyzed for readability, reliability, item difficulty, and item discrimination. Participants included 12 undergraduate students enrolled in an English reading course, with feedback from the course lecturer providing face validity through a reviewer checklist. The try-out results showed that the test passed the face validity and reliability check, and the majority of the questions were found to have moderate difficulty (48%) and satisfactorily discriminating (38%). To improve item quality, several questions were revised and their options refined. This study contributes a replicable model for teachers and teacher educators, demonstrating how classroom-based reading assessments can be designed to foster evaluative thinking and character formation. While limited by its small sample size and less stringent item revision criteria, the findings provide practical guidance for assessment design and inform teacher training and professional development programs. Future research could implement the revised test in larger-scale contexts, gather students' perspectives, and examine the relationship between HOTS levels and item difficulty to refine assessment practices further.
References
Abdulkarim, A., Komalasari, K., Saripudin, D., Ratmaningsih, N., & Anggraini, D. N. (2020). Development of a unity in diversity-based Pancasila education text book for Indonesian universities. International Journal of Instruction, 13(1), 371–386. https://doi.org/10.29333/iji.2020.13125a
Acar, A., & İŞi̇Sağ, K. U. (2017). Readability and comprehensibility in translation using reading ease and grade indices. International Journal of Comparative Literature and Translation Studies, 5(2), 47. https://doi.org/10.7575/aiac.ijclts.v.5n.2p.47
Afflerbach, P. (2016). Reading assessment: Looking ahead. The Reading Teacher, 69(4), 413–419. https://doi.org/10.1002/trtr.1430
Ahmadi, F., Rochmad, R., Lestari, F. P., & Harjunowibowo, D. (2021). The development of mathematics comic containing Pancasila values to develop character of elementary school students: A case study of Indonesia. Journal of Innovation in Educational and Cultural Research, 2(1), 25–34. https://doi.org/10.46843/jiecr.v2i1.20
Amali, L. N., Anggani Linggar Bharati, D., & Rozi, F. (2022). The implementation of high order thinking skills (HOTS) assessment to evaluate the students’ reading comprehension achievement. English Education Journal, 12(1), 10–18. https://doi.org/10.15294/eej.v12i1.52571
Ardiana, A. P., & As Sabiq, A. H. (2020). HOTS-based analysis on English reading comprehension formative assessment. Tarling : Journal of Language Education, 4(1), 19–36. https://doi.org/10.24090/tarling.v4i1.3975
Arifin, I., Wilujeng, I., & Jumadi, J. (2020). The effect of quick on the draw model assisted by the physics learning book integrated Pancasila values on critical thinking skill. Jurnal Penelitian & Pengembangan Pendidikan Fisika, 6(1), 121–130. https://doi.org/10.21009/1.06114
Arshad, A. (2020). A review on the principles of a reading comprehension test construction to assess the test takers at different levels. Psychology and Education, 57(8), 1290-1302.
Atjonen, P. (2014). Teachers’ views of their assessment practice. The Curriculum Journal, 25(2), 238–259. https://doi.org/10.1080/09585176.2013.874952
Atjonen, P., Pöntinen, S., Kontkanen, S., & Ruotsalainen, P. (2022). In enhancing preservice teachers’ assessment literacy: Focus on knowledge base, conceptions of assessment, and teacher learning. Frontiers in Education, 7, 891391. https://doi.org/10.3389/feduc.2022.891391
Bachman, L. F. (2010). Fundamental considerations in language testing (1. publ., 11. [print.]). Oxford University Press.
Berkowitz, M. W. (2012). Understanding Effective Character Education. CSEE Connections, the Center for Spiritual and Ethical Education. https://www.researchgate.net/publication/265068585
Borg, W. R., & Gall, M. D. (1983). Educational research: An introduction. Longman
Bower, M., Torrington, J., Lai, J. W. M., Petocz, P., & Alfano, M. (2024). How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey. Education and Information Technologies. https://doi.org/10.1007/s10639-023-12405-0
Brata, D. P. N., Utomo, E. S., & Sukardi, S. (2022). The analysis of students’ attitudes construction based on Pancasila profile to be integrated with teacher’s lesson plan in junior high school in pandemic era. 2nd International Conference on Education and Technology (ICETECH 2021). https://doi.org/10.2991/assehr.k.220103.045
Brown, H. D., & Lee, H. (2015). Teaching by principles: An interactive approach to language pedagogy. Pearson Education. https://books.google.co.id/books/about/Teaching_by_Principles.html?hl=id&id=4XEbrgEACAAJ&redir_esc=y
Butler, A. C. (2018). Multiple-choice testing in education: Are the best practices for assessment also good for learning? Journal of Applied Research in Memory and Cognition, 7(3), 323–331. https://doi.org/10.1016/j.jarmac.2018.07.002
Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. Cengage Learning
DeLuca, C., & Bellara, A. (2013). The current state of assessment education: Aligning policy, standards, and teacher education curriculum. Journal of Teacher Education, 64(4), 356–372. https://doi.org/10.1177/0022487113488144
Faradella, N. E., Wiyaka, W., & Egar, N. (2024). Teachers’ strategies and challenges in conducting HOTS-based activities in Merdeka Curriculum era. Indonesian Journal of Education and Pedagogy, 1(2), 75–90. https://doi.org/10.61251/ijoep.v1i2.64
George, D., & Mallery, P. (2020). IBM SPSS statistics 26 step by step. Routledge
Gopal, R., Maniam, M., Madzlan, N. A., Shukor, S. S. B., & Neelamegam, K. (2021). Readability formulas: An analysis into reading index of prose forms. Studies in English Language and Education, 8(3), 972–985. https://doi.org/10.24815/siele.v8i3.20373
Grabe, W., & Jiang, X. (2013). Assessing reading. In A. J. Kunnan (Ed.), The companion to language assessment (1st ed., pp. 185–200). Wiley. https://doi.org/10.1002/9781118411360.wbcla060
Gustiani, S. (2019). Research and development (R&D) method as a model design in educational research and its alternatives. Holistics (Hospitality and Linguistics): Jurnal Ilmiah Bahasa Inggris, 11(2). 11(2).
Hanafi, H. (2016). Developing reading comprehension test for the first semester students of english department. ELLITE: Journal of English Language, Literature, and Teaching, 1(1). https://doi.org/10.32528/ellite.v1i1.164
Haniah, A. R., Aman, A., & Setiawan, R. (2020). Integration of strengthening of character education and higher order thinking skills in history learning. Journal of Education and Learning (EduLearn), 14(2), 183–190. https://doi.org/10.11591/edulearn.v14i2.15010
Hidayat, M., Rozak, R. W. A., Hakam, K. A., Kembara, M. D., & Parhan, M. (2021). Character education in Indonesia: How is it internalized and implemented in virtual learning? Jurnal Cakrawala Pendidikan, 41(1), 186–198. https://doi.org/10.21831/cp.v41i1.45920
Hoang, L. P., Le, P. A., Le, H. T., Nguyen, D. T., Phan, T. C., Le, H. T., Van Tran, H., & Tran-Duong, Q. H. (2025). Evaluating educational assessment competence of pre-service teachers: Extended standards in the context of digital classroom assessment transformation. Education and Information Technologies, 30(12), 16347–16374. https://doi.org/10.1007/s10639-025-13467-y
Khairani, A. Z., & Shamsuddin, H. (2016). Assessing item difficulty and discrimination indices of teacher-developed multiple-choice tests. In S. F. Tang & L. Logonnathan (Eds.), Assessment for learning within and beyond the classroom (pp. 417–426). Springer Singapore. https://doi.org/10.1007/978-981-10-0908-2_35
Khorsand, N. (2009). Cognitive levels of questions used by Iranian EFL teachers in advanced reading comprehension tests. https://www.semanticscholar.org/paper/Cognitive-Levels-of-Questions-Used-by-Iranian-EFL-Khorsand/e173875b81e2036cfc68b5c569dbea07a2ceb523
Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2
Lickona, T., Schaps, E., & Lewis, C. (2002). Eleven principles of effective character education. Special Topics, General. https://digitalcommons.unomaha.edu/slcestgen/50
Magliano, J. P., Millis, K., Ozuru, Y., & McNamara, D. S. (2007). A multidimensional framework to evaluate reading assessment tools. In Reading comprehension strategies: Theories, interventions, and technologies (pp. 107–136). Lawrence Erlbaum Associates Publishers.
Miller, M. D., Linn, R. L., Gronlund, N. E., & Linn, R. L. (2009). Measurement and assessment in teaching (10th ed). Merrill/Pearson.
Nenohai, J. (2021). Internalizing character education (CE) into English language teaching: Teachers’ perspectives. Jurnal Manajemen Pendidikan, 8(2).
Newton, P. M., & Essex, K. (2024). How common is cheating in online exams and did it increase during the COVID-19 pandemic? A systematic review. Journal of Academic Ethics, 22(2), 323–343. https://doi.org/10.1007/s10805-023-09485-5
Ningrum, A. S. B., & Sudarwati, E. (2022). Is the test sensible? Developing a critical reading test for Indonesian EFL learners. JEELS (Journal of English Education and Linguistics Studies), 9(1), 209–227. https://doi.org/10.30762/jeels.v9i1.4005
Nitko, A. J., & Brookhart, S. M. (2014). Educational assessment of students. Pearson Education.
Oktarina, Y., Inderawati, R., & Petrus, I. (2022). Developing local culture-based EFL reading materials for the 21st-century learning. Studies in English Language and Education, 9(3), 1128–1147. https://doi.org/10.24815/siele.v9i3.24660
Oktriani, D. R., Hufad, A., & Utami, N. F. (2023). Overcoming the character crisis in children: strategies, outcomes, and evaluations of Bina Desa program. Utamax: Journal of Ultimate Research and Trends in Education, 5(2). https://doi.org/10.31849/utamax.v5i2.13758
Pastore, S. (2023). Teacher assessment literacy: A systematic review. Frontiers in Education, 8, 1217167. https://doi.org/10.3389/feduc.2023.1217167
Putra, T. K., & Abdullah, D. F. (2019). Higher-order thinking skill (HOTS) questions in English national examination in Indonesia. Jurnal Bahasa Lingua Scientia, 11(1), 145–160. https://doi.org/10.21274/ls.2019.11.1.145-160
Rachmawati, D. L., & Purwati, O. (2021). EFL teachers’ attitudes and competence in developing HOTS-based formative assessment. JEES (Journal of English Educators Society), 6(2). https://doi.org/10.21070/jees.v6i2.1060
Rahmiati, I. I., & Emaliana, I. (2017). Developing reading test using lower to higher order of thinking for ESP students. Language in India, 17(11), 124-144.
Ratrisari, A. A., Subyantoro, & Wagiran, W. (2024). Teachers’ perceptions of the development of Indonesian language assessment guidelines containing HOTS and Pancasila student profiles. Seloka: Jurnal Pendidikan Bahasa Dan Sastra Indonesia, 13(1), 27–36.
Rofii, A., Murtadho, F., & Rahmat, A. (2018). Model of contextual-based academic writing learning module. English Review: Journal of English Education, 6(2), 51. https://doi.org/10.25134/erjee.v6i2.1242
Sackett, P. R. (2012). Cognitive tests, constructs, and content validity: A commentary on Schmidt. International Journal of Selection and Assessment, 20(1), 24–27. https://doi.org/10.1111/j.1468-2389.2012.00576.x
Sari, A. M., & Ariswan, A. (2021). The integrated physics learning e-module with Pancasila character values in work and energy subjects as solution to improve students’ critical thinking ability and independence: Is it effective? Jurnal Ilmiah Pendidikan Fisika Al-Biruni, 10(1), 85. https://doi.org/10.24042/jipfalbiruni.v10i1.7749
Sari, A. M., Wilujeng, I., & Satriana, A. (2020). The development of optical devices learning book integrated with Pancasila practice values. Journal of Physics: Conference Series, 1440(1), 012036. https://doi.org/10.1088/1742-6596/1440/1/012036
Sato, T., & Ikeda, N. (2015). Test-taker perception of what test items measure: A potential impact of face validity on student learning. Language Testing in Asia, 5(1), 10. https://doi.org/10.1186/s40468-015-0019-z
Seifert, S., & Paleczek, L. (2021). Digitally assessing text comprehension in grades 3-4: Test development and validation. Electronic Journal of E-Learning, 19(5), 336-348. https://doi.org/10.34190/ejel.19.5.2467
Setiawan, H., Khair, B. N., Ratnadi, R., Hakim, M., & Istiningsih, S. (2019). Developing HOTS-based assessment instrument for primary schools. Proceedings of the 1st Annual Conference on Education and Social Sciences (ACCESS 2019). 1st Annual Conference on Education and Social Sciences (ACCESS 2019). https://doi.org/10.2991/assehr.k.200827.054
Shofa, N. A. (2021). Strengthening the profile of Pancasila students through project-based learning in learning history. In Prosiding Seminar Nasional & Internasional.
Sudarwati, E., Fatimah, F., Astuti, Y., & Ubaidillah, M. F. (2021). Developing online learning assessment instrument for English sentence structure course during Covid-19 pandemic. Langkawi: Journal of The Association for Arabic and English, 7(2), 170. https://doi.org/10.31332/lkw.v7i2.3122
Tyas, M. A., Nurkamto, J., Marmanto, S., & Laksani, H. (2019). Developing higher order thinking skills (HOTS)–Based Questions: Indonesian EFL Teachers’ Challenges. 52–63. https://doi.org/10.17501/26307413.2019.2106
Ware, J., & Vik, T. (2009). Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations. Medical Teacher, 31(3), 238–243. https://doi.org/10.1080/01421590802155597
Weir, C. J. (2005). Language testing and validation: An evidence-based approach. hampshire. Palgrave McMillan, 10, 9780230514577.
Wijayanti, P. S., Jamilah, F., Herawati, T. R., & Kusumaningrum, R. N. (2022). Penguatan penyusunan modul projek profil pelajar Pancasila pada sekolah penggerak jenjang SMA [Strengthening the development of Pancasila student profile project modules in activator schools at the senior high school level]. ABDIMAS NUSANTARA: Jurnal Pengabdian Kepada Masyarakat, 3(2), 43–49.
Wuryandani, W., Sartono, E. K. E., & Fathurrohman. (2020). Development of Big Book media to increase understanding of Pancasila values in early class. Proceedings of the International Conference on Educational Research and Innovation (ICERI 2019). International Conference on Educational Research and Innovation (ICERI 2019). https://doi.org/10.2991/assehr.k.200204.023
Zakso, A., Agung, I., Sofyatiningrum, E., & Capnary, M. C. (2022). Factors affecting character education in the development of the profile of Pancasila students: The case of Indonesia. 6(2), 2254–2273.
Özdemir, E. Ç., & Akyol, H. (2019). The development of a reading comprehension test. Universal Journal of Educational Research, 7(2), 563–570. https://doi.org/10.13189/ujer.2019.070229
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Imelda Gozali, Faizatul Istiqomah, Ly Gia Ngan Vo
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors can separately make additional contractual arrangements for non-exclusive distribution published by the journal (e.g., publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are allowed and encouraged to send their work via online (e.g., in the institutional repositories or their website) after published by the journal.
