Adaptive Assessment and Guessing Detection Implementation

Akbar Noto Ponco Bimantoro, Umi Laili Yuhana

Abstract


Computerized adaptive testing (CAT) is a context-based adaptive assessment. However, the assessment result may not be valid because the examinee might cheat or guess the answers. Although there are many guessing detection methods, there are not many discussions about their implementation into CAT. Therefore, this paper presents an example of a modification of an existing software so the newly modified software can detect guessed answers and be able to select questions adaptively. The system can detect assuming behavior by recording the examinee’s answer time. Also, the designed system can like questions adaptively by connecting Fuzzy logic, which calculates what level the question should select for the next iteration. The system is responded well by elementary and college students. A total of 56.6% felt the system was straightforward to use. The detection methods can detect guessing behavior of about 73%. However, the system’s sensitivity is low if the method is forced to classify answers which answered in a long response time / general guessing. Nevertheless, when we limit the data classified within 10s response time (rapid-guessing), the method’s sensitivity rises to 68.78%.

Keywords


Adaptive Assessment; Computerized Adaptive Testing; Guessing Behavior; Question Selection

Full Text:

Full Text

References


Kennedy KJ, Lee JCK. The Changing Role of Schools in Asian Societies. Routledge; 2007. https://www.taylorfrancis.com/books/9781134127306.

Kemendikbud. Panduan Penilaian Untuk Sekolah Dasar (SD). Kementrian Pendidikan dan Kebudayaan; 2016.

Kemendikbud. Panduan Penilaian oleh Pendidik dan Satuan Pendidikan Atas. Kementrian Pendidikan dan Kebudayaan; 2017.

Kemendikbud. Panduan Penilaian Untuk SMA. Kementrian Pendidikan dan Kebudayaan; 2015.

Peng SS, Lee JCK. Educational evaluation in East Asia: Emerging issues and challenges. Nova Science Publishers, UK; 2011.

Hwang GJ. A conceptual map model for developing intelligent tutoring systems. Computers & Education 2003 4;40:217–235. https://www.sciencedirect.com/science/article/abs/pii/S0360131502001215https://linkinghub.elsevier.com/retrieve/pii/S0360131502001215.

Hwang GJ, Sung HY, Chang SC, Huang XC. A fuzzy expert system-based adaptive learning approach to improving students’ learning performances by considering affective and cognitive factors. Computers and Education: Artificial Intelligence 2020;1:100003. https://doi.org/10.1016/j.caeai.2020.100003https://linkinghub.elsevier.com/retrieve/pii/S2666920X20300035.

Tseng WT. Measuring English vocabulary size via computerized adaptive testing. Computers & Education 2016 6;97:69–85. https://linkinghub.elsevier.com/retrieve/pii/S0360131516300501.

Louhab FE, Bahnasse A, Talea M. Towards an Adaptive Formative Assessment in Context-Aware Mobile Learning. Procedia Computer Science 2018;135:441–448. https://doi.org/10.1016/j.procs.2018.08.195.

Pan CC, Lin CC. Designing and implementing a computerized adaptive testing system with an MVC framework: A case study of the IEEE floating-point standard. Proceedings of 4th IEEE International Conference on Applied System Innovation 2018, ICASI 2018 2018;p. 609–612.

Vega YLP, Bolanos JCG, Nieto GMF, Baldiris SM. Application of item response theory (IRT) for the generation of adaptive assessments in an introductory course on object-oriented programming. Proceedings - Frontiers in Education Conference, FIE 2012;p. 0–3.

Uto M, Ueno M. Item Response Theory for Peer Assessment. IEEE Transactions on Learning Technologies 2016 4;9:157–170. http://ieeexplore.ieee.org/document/7243342/.

Cui W, Xue Z, Shen J, Sun G, Li J. The Item Response Theory Model for an AI-based Adaptive Learning System. In: 2019 18th International Conference on Information Technology Based Higher Education and Training (ITHET) IEEE; 2019. p.1–6. https://ieeexplore.ieee.org/document/8937383/.

Chrysafiadi K, Troussas C, Virvou M. Combination of fuzzy and cognitive theories for adaptive e-assessment. Expert Systems with Applications 2020;161:113614. https://doi.org/10.1016/j.eswa.2020.113614.

Delgado-Gómez D, Laria JC, Ruiz-Hernández D. Computerized adaptive test and decision trees: A unifying approach. Expert Systems with Applications 2019;117:358–366.

Rodríguez-Cuadrado J, Delgado-Gómez D, Laria JC, Rodríguez-Cuadrado S. Merged Tree-CAT: A fast method for building precise computerized adaptive tests based on decision trees. Expert Systems with Applications 2020 4;143:113066. https://linkinghub.elsevier.com/retrieve/pii/S0957417419307833.

Ridwan W, Wiranto I, Dako RDR. Ability estimation in computerized adaptive test using Mamdani Fuzzy Inference System. IOP Conference Series: Materials Science and Engineering 2020 5;850:012004. https://iopscience.iop.org/article/10.1088/1757-899X/850/1/012004.

Lendyuk T, Sachenko S, Rippa S, Sapojnyk G. Fuzzy rules for tests complexity changing for individual learning path construction. In: 2015 IEEE 8th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), vol. 2 IEEE; 2015. p. 945–948. http://ieeexplore.ieee.org/document/7341443/.

Wise SL. An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education 2006;19:95–114.

Pastor DA, Ong TQ, Strickman SN. Patterns of Solution Behavior across Items in Low-Stakes Assessments. Educational Assessment 2019 7;24:189–212.

Demars CE. Changes in Rapid-Guessing Behavior Over a Series of Assessments. Educational Assessment 2007 4;12:23–45.

DeMars CE, Wise SL. Can differential rapid-guessing behavior lead to differential item functioning? International Journal of Testing 2010;10:207–229.

Setzer JC, Wise SL, van den Heuvel JR, Ling G. An Investigation of Examinee Test-Taking Effort on a Large-Scale Assessment. Applied Measurement in Education 2013 1;26:34–49. http://www.tandfonline.com/doi/abs/10.1080/08957347.2013.739453.

Wise SL, Kong X. Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests. Applied Measurement in Education 2005 4;18:163–183. http://www.tandfonline.com/doi/abs/10.1207/s15324818ame1802_2.

Silm G, Must O, Täht K. Test-taking effort as a predictor of performance in low-stakes tests. Trames Journal of the Humanities and Social Sciences 2013;17:433. http://www.kirj.ee/?id=23010&tpl=1061&c_tpl=1064.

Wise SL, Ma L, Kingsbury GG, Hauser C. An investigation of the relationship between time of testing and test-taking effort. National Council on Measurement in Education 2010;p. 1–18.

Wise SL. Rapid-Guessing Behavior: Its Identification, Interpretation, and Implications. Educational Measurement: Issues and Practice 2017 12;36:52–61. https://onlinelibrary.wiley.com/doi/10.1111/emip.12165.

Zadeh LA. Fuzzy sets. Information and Control 1965 6;8:338–353. https://linkinghub.elsevier.com/retrieve/pii/S001999586590241X.

Mamdani EH. Application of Fuzzy Algorithms for Control of Simple Dynamic Plant. Proceedings of the Institution of Electrical Engineers 1974;121:1585–1588.

Schnipke DL. Assessing Speededness in Computer-based tests using item response times. Dissertation 1995.

Lin CK. Effects of Removing Responses With Likely Random Guessing Under Rasch Measurement on a Multiple-Choice Language Proficiency Test. Language Assessment Quarterly 2018 10;15:406–422. https://www.tandfonline.com/doi/full/10.1080/15434303.2018.1534237.




DOI: http://dx.doi.org/10.12962/j20882033.v33i1.12027

Refbacks

  • There are currently no refbacks.


Creative Commons License

IPTEK Journal of Science and Technology by Lembaga Penelitian dan Pengabdian kepada Masyarakat, ITS is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://iptek.its.ac.id/index.php/jts.