Developing a Multi-Rater Indonesian Essay Scoring Dataset for Automated Essay Scoring

Pietra Wahyu  Utomo; Zainal  Abidin

doi:10.70716/reswara.v4i4.908

Authors

Pietra Wahyu Utomo Master of Informatics, UIN Maulana Malik Ibrahim, Malang, Indonesia Author
Zainal Abidin Master of Informatics, UIN Maulana Malik Ibrahim, Malang, Indonesia Author

DOI:

https://doi.org/10.70716/reswara.v4i4.908

Keywords:

automated essay scoring, multi-rater annotation, inter-rater reliability, educational dataset

Abstract

Automated Essay Scoring (AES) research requires high-quality annotated datasets that provide reliable human reference scores, particularly for languages and educational contexts that remain underrepresented in existing resources. This study develops and validates a multi-rater Indonesian essay assessment dataset designed to support future AES and educational natural language processing research. The dataset comprises 3,600 response-level records collected from 360 senior high school students answering 10 Indonesian language essay questions. Each response was independently and blindly evaluated by three qualified Indonesian language teachers using a common analytical scoring rubric. Individual rater scores were preserved alongside an aggregated reference score to maintain the multi-rater structure of the dataset. Annotation reliability was evaluated using the average-measures absolute-agreement Intraclass Correlation Coefficient (ICC(A,k)), complemented by question-level reliability analysis and maximum pairwise inter-rater score differences. The overall ICC(A,k) was 0.9905, indicating excellent inter-rater reliability, while question-level ICC values ranged from 0.9803 to 0.9934, with a mean of 0.9899. Furthermore, 96.83% of responses exhibited a maximum inter-rater score difference of no more than one point. These findings demonstrate that the dataset provides highly consistent human annotations and a reliable basis for constructing reference scores. By documenting the dataset structure, scoring rubric, annotation protocol, rater qualifications, and reliability evidence, this study provides a documented research resource for future Indonesian AES research. The preserved multi-rater annotations may support subsequent investigations of human–AI agreement, rater variability, scoring severity, and alternative reference-score construction, subject to dataset access restrictions.

References

Artstein, R., & Poesio, M. (2008). Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596. https://doi.org/10.1162/coli.07-034-R2

Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12(2), 86–107. https://doi.org/10.1016/j.asw.2007.07.001

Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6, 587–604. https://doi.org/10.1162/tacl_a_00041

Beseiso, M. H., Alzubi, O. A., & Rashaideh, H. (2021). A novel automated essay scoring approach for reliable higher educational assessments. Journal of Computing in Higher Education, 33(3), 727–746. https://doi.org/10.1007/s12528-021-09283-1

Bridgeman, B., & Ramineni, C. (2017). Design and evaluation of automated writing evaluation models: Relationships with writing in naturalistic settings. Assessing Writing, 34, 62–71. https://doi.org/10.1016/j.asw.2017.10.001

Buzick, H., Oliveri, M. E., Attali, Y., & Flor, M. (2016). Comparing human and automated essay scoring for prospective graduate students with learning disabilities and/or ADHD. Applied Measurement in Education, 29(3), 161–172. https://doi.org/10.1080/08957347.2016.1171765

Cohen, Y., Levi, E., & Ben-Simon, A. (2018). Validating human and automated scoring of essays against “true” scores. Applied Measurement in Education, 31(3), 241–250. https://doi.org/10.1080/08957347.2018.1464450

Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1).

Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., ... & Gardner, M. (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 conference on empirical methods in natural language processing (pp. 1286-1305). https://doi.org/10.18653/v1/2021.

East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing. Assessing Writing, 14(2), 88–115. https://doi.org/10.1016/j.asw.2009.04.001

Ferrara, S., & Qunbar, S. (2022). Validity arguments for AI-based automated scores: Essay scoring as an illustration. Journal of Educational Measurement, 59(3), 288–313. https://doi.org/10.1111/jedm.12333

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723

Gwet, K. L. (2021). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters: Vol. 2. Analysis of quantitative ratings (5th ed.). Advanced Analytics, LLC.

Hannah, L., Jang, E. E., Shah, M., & Gupta, V. (2023). Validity arguments for automated essay scoring of young students’ writing traits. Language Assessment Quarterly, 20(4–5), 399–420. https://doi.org/10.1080/15434303.2023.2288253

Huang, J., & Chen, G. (2022). Individualized feedback to raters in language assessment: Impacts on rater effects. Assessing Writing, 52, 100623. https://doi.org/10.1016/j.asw.2022.100623

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

Leckie, G., & Baird, J.-A. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399–418. https://doi.org/10.1111/j.1745-3984.2011.00152.x

Li, X., Yang, H., Hu, S., Geng, J., Lin, K., & Li, Y. (2022). Enhanced hybrid neural network for automated essay scoring. Expert Systems, 39(10), e13068. https://doi.org/10.1111/exsy.13068

Ludwig, S., Mayer, C., Hansen, C., Eilers, K., & Brandt, S. (2021). Automated essay scoring using transformer models. Psych, 3(4), 897–915. https://doi.org/10.3390/psych3040056

Page, E. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 47, 238–243.

Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2002). Stumping e-rater: Challenging the validity of automated essay scoring. Computers in Human Behavior, 18(2), 103–134. https://doi.org/10.1016/S0747-5632(01)00052-8

Powers, D. E., Escoffery, D. S., & Duchnowski, M. P. (2015). Validating automated essay scoring: A (modest) refinement of the “gold standard.” Applied Measurement in Education, 28(2), 130–142. https://doi.org/10.1080/08957347.2014.1002920

Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring system: A systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527. https://doi.org/10.1007/s10462-021-10068-2

Shermis, M. D. (2022). Anchoring validity evidence for automated essay scoring. Journal of Educational Measurement, 59(3), 314–337. https://doi.org/10.1111/jedm.12336

Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of automated essay evaluation: Current applications and new directions. Routledge. https://doi.org/10.4324/9780203122761

Shermis, M. D., Lottridge, S., & Mayfield, E. (2015). The impact of anonymization for automated essay scoring. Journal of Educational Measurement, 52(4), 419–436. https://doi.org/10.1111/jedm.12093

Sun, J., Song, T., Peng, W., & Song, J. (2025). A survey of automated essay scoring: Challenges, advances, and future. Neurocomputing, 650, 130916. https://doi.org/10.1016/j.neucom.2025.130916

Vo, Y., Rickels, H., Welch, C., & Dunbar, S. (2023). Human scoring versus automated scoring for English learners in a statewide evidence-based writing assessment. Assessing Writing, 56, 100719. https://doi.org/10.1016/j.asw.2023.100719

Voss, E. (2025). Comparison of traditional machine learning and neural network approaches for automated scoring of second language English essays. Language Testing, 42(4), 369–396. https://doi.org/10.1177/02655322251348959

Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., ... Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18

Wind, S. A., Wolfe, E. W., Engelhard, G., Jr., Foltz, P., & Rosenstein, M. (2018). The influence of rater effects in training sets on the psychometric quality of automated scoring for writing assessments. International Journal of Testing, 18(1), 27–49. https://doi.org/10.1080/15305058.2017.1361426

Wiseman, C. S. (2012). Rater effects: Ego engagement in rater decision-making. Assessing Writing, 17(3), 150–173. https://doi.org/10.1016/j.asw.2011.12.001

Zhang, J. (2016). Same text different processing? Exploring how raters’ cognitive and meta-cognitive strategies influence rating accuracy in essay scoring. Assessing Writing, 27, 37–53. https://doi.org/10.1016/j.asw.2015.11.001