Exploring Higher Education Students' Experience with AI-powered Educational Tools: The Case of an Early Warning System
DOI:
https://doi.org/10.7346/-fei-XXII-01-24_09Keywords:
Student's Experience, Artificial intelligence, Early Warning System, Higher Education, Thematic AnalysisAbstract
AI-powered educational tools (AIEd) include early warning systems (EWS) to identify at-risk undergraduates, offering personalized assistance. Revealing students' subjective experiences with EWS could contribute to a deeper understanding of what it means to engage with AI in areas of human life, like teaching and learning. Our investigation hence explored students' subjective experiences with EWS, characterizing them according to students’ profiles, self-efficacy, prior experience, and perspective on data ethics. The results show that students, largely senior workers with strong academic self-efficacy, had limited experience with this method and minimal expectations. But, using the EWS inspired meaningful reflections. Nonetheless, a comparison between the Computer Science and Economics disciplines demonstrated stronger trust and expectation regarding the system and AI for the former. The study emphasized the importance of helping students’ additional experiences and comprehension while embracing AI systems in education to ensure the quality, relevance, and fairness of their educational experience overall.
References
Akhtar, S., Warburton, S., & Xu, W. (2017). The use of an online learning and teaching system for monitoring computer-aided design student participation and predicting student success. International Journal of Technology and Design Education, 27(2). https://doi.org/10.1007/s10798-015-9346-8
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. ACM International Conference Proceeding Series, 267–270. https://doi.org/10.1145/2330601.2330666
Bañeres, D., Karadeniz, A., Guerrero-Roldán, A.-E., & Elena Rodríguez, M. (2021). A Predictive System for Supporting At-Risk Students’ Identification. In K. Arai, S. Kapoor, & R. Bhatia (Eds.), Proceedings of the Future Technologies Conference (FTC) 2020, Volume 1 (Vol. 1288, pp. 891–904). Springer International Publishing. https://doi.org/10.1007/978-3-030-63128-4_67
Bañeres, D., Rodríguez, M. E., Guerrero-Roldán, A. E., & Karadeniz, A. (2020). An Early Warning System to Detect At-Risk Students in Online Higher Education. Applied Sciences, 10(13), 4427. https://doi.org/10.3390/app10134427
Bhattacherjee, A., & Premkumar, G. (2004). Understanding changes in belief and attitude toward information technology usage: A theoretical model and longitudinal test. MIS Quarterly: Management Information Systems, 28(2), 229–254. https://doi.org/10.2307/25148634
Bisdas, S., Topriceanu, C. C., Zakrzewska, Z., Irimia, A. V., Shakallis, L., Subhash, J., Casapu, M. M., Leon-Rojas, J., Pinto dos Santos, D., Andrews, D. M., Zeicu, C., Bouhuwaish, A. M., Lestari, A. N., Abu-Ismail, L., Sadiq, A. S., Khamees, A., Mohammed, K. M. G., Williams, E., Omran, A. I., … Ebrahim, E. H. (2021). Artificial Intelligence in Medicine: A Multinational Multi-Center Survey on the Medical and Dental Students’ Perception. Frontiers in Public Health, 9. https://doi.org/10.3389/fpubh.2021.795284
Bochniarz, K. T., Czerwiński, S. K., Sawicki, A., & Atroszko, P. A. (2022). Attitudes to AI among high school students: Understanding distrust towards humans will not help us understand distrust towards AI. Personality and Individual Differences, 185. https://doi.org/10.1016/j.paid.2021.111299
Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4). https://doi.org/10.1109/TLT.2017.2740172
Bogina, V., Hartman, A., Kuflik, T., & Shulner-Tal, A. (2022). Educating Software and AI Stakeholders About Algorithmic Fairness, Accountability, Transparency and Ethics. International Journal of Artificial Intelligence in Education, 32(3). https://doi.org/10.1007/s40593-021-00248-0
Bozkurt, A., Xiao, F., Lambert, S., Pazurek, A., Crompton, H., Koseoglu, S., Farrow, R., Bond, M., Nerantzi, C., Honeychurch, S., Bali, M., Dron, J., Mir, K., Stewart, B., Stewart, B., Costello, E., Mason, J., Stracke, C., Romero-Hall, E., & Jandric, P. (2023). Speculative Futures on ChatGPT and Generative Artificial Intelligence (AI): A Collective Reflection from the Educational Landscape. Asian Journal of Distance Education, 18, 53–130 [Journal Article, Zenodo]. https://doi.org/10.5281/zenodo.7636568
Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2019). Thematic analysis. Handbook of Research Methods in Health Social Sciences, 843–860. https://doi.org/10.1007/978-981-10-5251-4_103/COVER
Buckingham-Shum, S. J. (2019). Critical data studies, abstraction and learning analytics: Editorial to Selwyn’s LAK keynote and invited commentaries. Journal of Learning Analytics, 6(3). https://doi.org/10.18608/jla.2019.63.2
Casey, K., & Azcona, D. (2017). Utilizing student activity patterns to predict performance. International Journal of Educational Technology in Higher Education, 14(1). https://doi.org/10.1186/s41239-017-0044-3
Chen, M., Siu-Yung, M., Chai, C. S., Zheng, C., & Park, M. Y. (2021). A Pilot Study of Students’ Behavioral Intention to Use AI for Language Learning in Higher Education. Proceedings - 2021 International Symposium on Educational Technology, ISET 2021. https://doi.org/10.1109/ISET52350.2021.00045
Chikobava, M., & Romeike, R. (2021). Towards an Operationalization of AI acceptance among Pre-service Teachers. ACM International Conference Proceeding Series. https://doi.org/10.1145/3481312.3481349
Chocarro, R., Cortiñas, M., & Marcos-Matás, G. (2021). Teachers’ attitudes towards chatbots in education: a technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educational Studies. https://doi.org/10.1080/03055698.2020.1850426
Demir, K., & Güraksın, G. E. (2022). Determining middle school students’ perceptions of the concept of artificial intelligence: A metaphor analysis. Participatory Educational Research, 9(2). https://doi.org/10.17275/per.22.41.9.2
Dreyfus, H. L. (2007). Why Heideggerian AI failed and how fixing it would require making it more Heideggerian. Artificial Intelligence, 171(18). https://doi.org/10.1016/j.artint.2007.10.012
Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K., & Kyngäs, H. (2014). Qualitative Content Analysis: A Focus on Trustworthiness. SAGE Open, 4(1). https://doi.org/10.1177/2158244014522633
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research Evidence on the Use of Learning Analytics: implications for education Policy. https://doi.org/10.2791/955210
Floridi, L. (2023). The Ethics of Artificial Intelligence. Principles, Challenges, and Opportunities. Oxford University Press. https://global.oup.com/academic/product/the-ethics-of-artificial-intelligence-9780198883098?cc = it&lang = en
Freitas, R., & Salgado, L. (2020). Educators in the loop: Using scenario simulation as a tool to understand and investigate predictive models of student dropout risk in distance learning. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12217 LNCS. https://doi.org/10.1007/978-3-030-50334-5_17
Gado, S., Kempen, R., Lingelbach, K., & Bipp, T. (2022). Artificial intelligence in psychology: How can we enable psychology students to accept and use artificial intelligence? Psychology Learning and Teaching, 21(1). https://doi.org/10.1177/14757257211037149
Gallagher, S. (2014). Phenomenology | The Encyclopedia of Human-Computer Interaction, 2nd Ed. https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/phenomenology
Ghotbi, N., Ho, M. T., & Mantello, P. (2022). Attitude of college students towards ethical issues of artificial intelligence in an international university in Japan. AI and Society, 37(1). https://doi.org/10.1007/s00146-021-01168-2
González-Pérez, S., Mateos de Cabo, R., & Sáinz, M. (2020). Girls in STEM: Is It a Female Role-Model Thing? Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.02204
Greenland, S. J., & Moore, C. (2022). Large qualitative sample and thematic analysis to redefine student dropout and retention strategy in open online education. British Journal of Educational Technology, 53(3). https://doi.org/10.1111/bjet.13173
Guggemos, J., Seufert, S., & Sonderegger, S. (2020). Humanoid robots in higher education: Evaluating the acceptance of Pepper in the context of an academic writing course using the UTAUT. British Journal of Educational Technology, 51(5), 1864–1883. https://doi.org/10.1111/bjet.13006
Gutiérrez, F., Seipp, K., Ochoa, X., Chiluiza, K., de Laet, T., & Verbert, K. (2020). LADA: A learning analytics dashboard for academic advising. Computers in Human Behavior, 107. https://doi.org/10.1016/j.chb.2018.12.004
Hayes, S. (2021). Postdigital Positionality. Developing Powerful Inclusive Narratives for Learning, Teaching, Research and Policy in Higher Education. Brill. https://brill.com/view/title/57466?language = en
Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., & Koedinger, K. R. (2022). Ethics of AI in Education: Towards a Community-Wide Framework. International Journal of Artificial Intelligence in Education, 32(3). https://doi.org/10.1007/s40593-021-00239-1
Hu, Y. H., Lo, C. L., & Shih, S. P. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behavior, 36, 469–478. https://doi.org/10.1016/j.chb.2014.04.002
Jalalov, D. (2023, March 9th). L’evoluzione dei chatbot da T9-Era e GPT-1 a ChatGPT. (The evolution of chatbots from T9-Era and GPT-1 to ChatGPT.) [Blog]. Metaverse Post. https://mpost.io/it/l%27evoluzione-dei-chatbot-dall%27era-t9-e-gpt-1-a-chatgpt/
Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10474 LNCS. https://doi.org/10.1007/978-3-319-66610-5_7
Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. Internet and Higher Education, 47. https://doi.org/10.1016/j.iheduc.2020.100758
Kabathova, J., & Drlik, M. (2021). Towards predicting student’s dropout in university courses using different machine learning techniques. Applied Sciences (Switzerland), 11(7). https://doi.org/10.3390/app11073130
Kerr, A., Barry, M., & Kelleher, J. D. (2020). Expectations of artificial intelligence and the performativity of ethics: Implications for communication governance. Big Data and Society, 7(1). https://doi.org/10.1177/2053951720915939
Kim, J., Merrill, K., Xu, K., & Sellnow, D. D. (2020). My Teacher Is a Machine: Understanding Students’ Perceptions of AI Teaching Assistants in Online Education. International Journal of Human-Computer Interaction, 36(20). https://doi.org/10.1080/10447318.2020.1801227
Kim, J. W., Jo, H. I., & Lee, B. G. (2019). The Study on the Factors Influencing on the Behavioral Intention of Chatbot Service for the Financial Sector: Focusing on the UTAUT Model. Journal of Digital Contents Society, 20(1), 41–50. https://doi.org/10.9728/dcs.2019.20.1.41
Krumm, A. E., Waddington, R. J., Teasley, S. D., & Lonn, S. (2014). A learning management system-based early warning system for academic advising in undergraduate engineering. In J. A. Larusson & B. White (Eds.), Learning Analytics: From Research to Practice (pp. 103–119). https://doi.org/10.1007/978-1-4614-3305-7_6
Lund, B., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40. https://doi.org/10.1108/LHTN-01-2023-0009
Liz-Domínguez, M., Caeiro-Rodríguez, M., Llamas-Nistal, M., & Mikic-Fonte, F. A. (2019). Systematic literature review of predictive analysis tools in higher education. Applied Sciences (Switzerland), 9 (24). https://doi.org/10.3390/app9245569
Makridakis, S. (2017). The forthcoming artificial intelligence (AI) revolution: Its impact on society and firms. Futures, 90, 46-60. https://doi.org/10.1016/j.futures.2017.03.006
Ng, D. T. K., Leung, J. K. L., Su, J., Ng, R. C. W., & Chu, S. K. W. (2023). Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educational Technology Research and Development, 71(1), 137–161. https://doi.org/10.1007/s11423-023-10203-6
Ortigosa, A., Carro, R. M., Bravo-Agapito, J., Lizcano, D., Alcolea, J. J., & Blanco, Ó. (2019). From Lab to Production: Lessons Learnt and Real-Life Challenges of an Early Student-Dropout Prevention System. IEEE Transactions on Learning Technologies, 12(2), 264–277. https://doi.org/10.1109/TLT.2019.2911608
OpenAI. (2022, November 30th). Introducing ChatGPT [Blog]. Openai.Com. https://openai.com/blog/chatgpt
Pedró, F., Subosa, M., Rivas, A., & Valverde, P. (2019). Artificial intelligence in education: challenges and opportunities for sustainable development. Working Papers on Education Policy, 7.
Pichai, S. (2023, February 6th). Google AI updates: Bard and new AI features in Search [Blog]. Google.com https://blog.google/technology/ai/bard-google-ai-search-updates/
Pintrich, P. R. (2000). An Achievement Goal Theory Perspective on Issues in Motivation Terminology, Theory, and Research. Contemporary Educational Psychology, 25(1). https://doi.org/10.1006/ceps.1999.1017
Plak, S., Cornelisz, I., Meeter, M., & van Klaveren, C. (2022). Early warning systems for more effective student counselling in higher education: Evidence from a Dutch field experiment. Higher Education Quarterly, 76(1). https://doi.org/10.1111/hequ.12298
Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1). https://doi.org/10.1186/s41039-017-0062-8
Prinsloo, P. (2019). A social cartography of analytics in education as performative politics. British Journal of Educational Technology, 50(6). https://doi.org/10.1111/bjet.12872
Prinsloo, P., Slade, S., & Khalil, M. (2022). The answer is (not only) technological: Considering student data privacy in learning analytics. British Journal of Educational Technology, 53(4), 876–893. https://doi.org/10.1111/BJET.13216
Qin, F., Li, K., & Yan, J. (2020). Understanding user trust in artificial intelligence-based educational systems: Evidence from China. British Journal of Educational Technology, 51(5), 1693–1710. https://doi.org/10.1111/bjet.12994
Raffaghelli, J. E., Loria-Soriano, E., González, M. E. R., Bañeres, D., & Guerrero-Roldán, A. E. (2022). Extracted and Anonymised Qualitative Data on Students’ Acceptance of an Early Warning System [dataset]. Zenodo. https://doi.org/10.5281/ZENODO.6841129
Raffaghelli, J. E., Rodríguez, M. E., Guerrero-Roldán, A.-E., & Bañeres, D. (2022). Applying the UTAUT model to explain the students’ acceptance of an early warning system in Higher Education. Computers & Education, 182, 104468. https://doi.org/10.1016/j.compedu.2022.104468
Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. International Review of Research in Open and Distance Learning, 19(5), 187–202. https://doi.org/10.19173/irrodl.v19i5.3493
Sáinz, M., & Eccles, J. (2012). Self-concept of computer and math ability: Gender implications across time and within ICT studies. Journal of Vocational Behavior, 80(2). https://doi.org/10.1016/j.jvb.2011.08.005
Scherer, R., & Teo, T. (2019). Editorial to the special section—Technology acceptance models: What we know and what we (still) do not know. In British Journal of Educational Technology (Vol. 50, Issue 5, pp. 2387–2393). Blackwell Publishing Ltd. https://doi.org/10.1111/bjet.12866
Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3). https://doi.org/10.18608/jla.2019.63.3
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18(1). https://doi.org/10.1186/s41239-021-00292-9
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Tzimas, D., & Demetriadis, S. (2021). Ethical issues in learning analytics: a review of the field. Educational Technology Research and Development, 69(2). https://doi.org/10.1007/s11423-021-09977-4
Valle, N., Antonenko, P., Dawson, K., & Huggins-Manley, A. C. (2021). Staying on target: A systematic literature review on learner-facing learning analytics dashboards. In British Journal of Educational Technology (Vol. 52, Issue 4). https://doi.org/10.1111/bjet.13089
van Brummelen, J., Tabunshchyk, V., & Heng, T. (2021). Alexa, Can I Program You?”: Student Perceptions of Conversational Artificial Intelligence before and after Programming Alexa. Proceedings of Interaction Design and Children, IDC 2021. https://doi.org/10.1145/3459990.3460730
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly: Management Information Systems, 27(3), 425–478. https://doi.org/10.2307/30036540
Xing, W., Chen, X., Stein, J., & Marcinkowski, M. (2016). Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization. Computers in Human Behavior, 58, 119–129. https://doi.org/10.1016/j.chb.2015.12.007
Xu, J. J., & Babaian, T. (2021). Artificial intelligence in business curriculum: The pedagogy and learning outcomes. International Journal of Management Education, 19(3). https://doi.org/10.1016/j.ijme.2021.100550
Zander, L., Höhne, E., Harms, S., Pfost, M., & Hornsey, M. J. (2020). When Grades Are High but Self-Efficacy Is Low: Unpacking the Confidence Gap Between Girls and Boys in Mathematics. Frontiers in Psychology, 11, 2492. https://doi.org/10.3389/fpsyg.2020.552355
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1–27. https://doi.org/10.1186/s41239-019-0171-0
Zhou, Y., Zhao, J., & Zhang, J. (2020). Prediction of learners’ dropout in E-learning based on the unusual behaviors. Interactive Learning Environments. https://doi.org/10.1080/10494820.2020.1857788
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1). https://doi.org/10.3102/0002831207312909
Additional Files
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2024 M. Elena Rodríguez, Juliana E. Raffaghelli, David Bañeres, Ana Elena Guerrero-Roldán, Francesca Crudele
This work is licensed under a Creative Commons Attribution 4.0 International License.
Formazione & insegnamento is distributed under Attribution 4.0 International (CC BY 4.0).
For further details, please refer to our Repository & Archiving Policy, as well as our Copyright & Licensing Terms.
