Transparent Algorithmic Integration in the Systematic Review (IATSR): A Methodological Proposal for Educational Research in the Age of Artificial Intelligence
Keywords:
Systematic review, PRISMA 2020, PRISMA-trAIce, Artificial Intelligence, Human-in-the-loop, Transparency, Reproducibility, Educational research, Research trainingAbstract
This article proposes, describes and argues for a methodological structure (IATSR, Transparent Algorithmic Integration in the Systematic Review) for conducting systematic reviews assisted by algorithmic tools in the non-bibliometric fields of educational research. The proposal stems from a problem now felt in editorial practice and documented in the international literature: the spread of undeclared use of so-called generative artificial intelligence in scholarly production, with effects on the quality, verifiability and equity of educational research. We argue that algorithmic integration, if made constitutive and declared within a protocol built on PRISMA 2020, can be carried out rigorously, transparently and verifiably. The workflow unfolds in seven phases, adopts the items already present in PRISMA-trAIce, and treats the human-in-the-loop principle as unavoidable. The contribution delivers five theoretical-methodological products – the formalisation of the workflow in seven phases, the R1/R2 classification of sources, the epistemological risk matrix, the mapping onto the PRISMA 2020 checklist items and the adapted flow diagram – which remain usable independently of the full adoption of the workflow. The proposal is accompanied by explicit qualitative validity judgements stated as expert a priori estimates rather than the outcome of a pilot study or inter-rates validation: the empirical testing of the protocol constitutes the main line of future research. The contribution thus positions itself as a contribution to the methodology of educational research, to research training and to the debate on the transparent assessment of scholarly production.
References
Aloe, A. M., Dewidar, O., Hennessy, E. A., Pigott, T., Stewart, G., Welch, V., Wilson, D. B., & Campbell MECCIR Working Group. (2024). Campbell standards: Modernizing Campbell’s methodologic expectations for Campbell Collaboration intervention reviews (MECCIR). Campbell Systematic Reviews, 20, Article e1445. https://doi.org/10.1002/cl2.1445
American Psychological Association. (2024). APA PsycInfo [Research database]. APA PsycNet. https://www.apa.org/pubs/databases/psycinfo
Aromataris, E., Lockwood, C., Porritt, K., Pilla, B., & Jordan, Z. (Eds.). (2024). JBI manual for evidence synthesis. JBI. https://doi.org/10.46658/JBIMES-24-01
Pullin, A. S., Frampton, G. K., Livoreil, B., & Petrokofsky, G. (Eds.). (2022). Guidelines and standards for evidence synthesis in environmental management (Version 5.1). Collaboration for Environmental Evidence. https://www.environmentalevidence.org/information-for-authors
Consensus NLP, Inc. (2024). Consensus [AI-powered academic search engine]. https://consensus.app
COPE Council. (2023). COPE position statement on AI tools in scholarly publishing. https://doi.org/10.24318/cCVRZBms
Culbert, J. H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M., Donner, P., & Mayr, P. (2025). Reference coverage analysis of OpenAlex compared to Web of Science and Scopus. Scientometrics, 130, 2475–2492. https://doi.org/10.1007/s11192-025-05293-3
Elsevier. (2024). Scopus AI (AI Discovery) [Generative AI research tool]. https://www.elsevier.com/products/scopus/scopus-ai
Flemyng, E., Noel-Storr, A., Macura, B., Gartlehner, G., Thomas, J., Meerpohl, J. J., Jordan, Z., Minx, J., Eisele-Metzger, A., Hamel, C., Jemioło, P., Porritt, K., & Grainger, M. (2025). Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025. Environmental Evidence, 14, Article 20. https://doi.org/10.1186/s13750-025-00374-5
Glynn, A. (2024). Suspected undeclared use of artificial intelligence in the academic literature: An analysis of the Academ-AI dataset [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2411.15218
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Gray, A. (2024). ChatGPT “contamination”: Estimating the prevalence of LLMs in the scholarly literature [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2403.16887
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2024). Cochrane handbook for systematic reviews of interventions (Version 6.5, updated August 2024). Cochrane. https://training.cochrane.org/handbook
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
Holst, D., Moenck, K., Koch, J., Schmedemann, O., & Schüppstuhl, T. (2025). Transparent reporting of AI in systematic literature reviews: Development of the PRISMA-trAIce checklist. JMIR AI, 4, Article e80247. https://doi.org/10.2196/80247
Incorrect terminology in Table 2 [Correction]. (2019). JAMA, 322(20), 2026. https://doi.org/10.1001/jama.2019.18307
Institute of Education Sciences. (2024). Education Resources Information Center (ERIC) [Bibliographic database]. U.S. Department of Education. https://eric.ed.gov
International Committee of Medical Journal Editors. (2023). Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. https://www.icmje.org/recommendations/
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025). Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances, 11(27), Article eadt3813. https://doi.org/10.1126/sciadv.adt3813
Kung, J. (2023). Elicit (product review). Journal of the Canadian Health Libraries Association / Journal de l’Association des bibliothèques de la santé du Canada, 44(1), 15–18. https://doi.org/10.29173/jchla29657
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., & Messer, M. (2025). Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: Experimental study. JMIR AI, 4, Article e80371. https://doi.org/10.2196/80371
Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson Education.
McInnes, M. D. F., Moher, D., Thombs, B. D., McGrath, T. A., Bossuyt, P. M., & the PRISMA-DTA Group. (2018). Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: The PRISMA-DTA statement. JAMA, 319(4), 388–396. https://doi.org/10.1001/jama.2017.19163
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS Medicine, 6(7), Article e1000097. https://doi.org/10.1371/journal.pmed.1000097
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews, 4, Article 5. https://doi.org/10.1186/2046-4053-4-5
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Topaz, M., Roguin, N., Gupta, P., Zhang, Z., & Peltonen, L.-M. (2026). Fabricated citations: An audit across 2·5 million biomedical papers. The Lancet, 407, 1779–1781. https://doi.org/10.1016/S0140-6736(26)00603-3
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Xiao, Y., & Watson, M. (2019). Guidance on conducting a systematic literature review. Journal of Planning Education and Research, 39(1), 93–112. https://doi.org/10.1177/0739456X17723971
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education: Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0
Downloads
Published
How to Cite
License
Copyright (c) 2026 Maurizio Gentile, Piergiuseppe Ellerani
This work is licensed under a Creative Commons Attribution 4.0 International License.
Formazione & insegnamento is distributed under Attribution 4.0 International (CC BY 4.0).
For further details, please refer to our Repository & Archiving Policy, as well as our Copyright & Licensing Terms.
