SIREF Formazione & insegnamento 1973-4778 2279-7505 1973-4778 PensaMultimedia https://www.pensamultimedia.it/ https://ojs.pensamultimedia.it/index.php/siref/ 10.7346/-fei-XXII-01-24_09 Exploring Higher Education Students' Experience with AI-powered Educational Tools: The Case of an Early Warning System Esplorare l'esperienza degli studenti universitari con strumenti educativi basati sull'IA: Il caso di un sistema di allerta precoce Explorar la experiencia de los estudiantes universitarios con herramientas educativas basadas en IA: El caso de un sistema de alerta temprana Explorando a experiência de estudantes universitários com ferramentas educacionais baseadas em IA: O caso de um sistema de alerta precoce AI-powered educational tools (AIEd) include early warning systems (EWS) to identify at-risk undergraduates, offering personalized assistance. Revealing students' subjective experiences with EWS could contribute to a deeper understanding of what it means to engage with AI in areas of human life, like teaching and learning. Our investigation hence explored students' subjective experiences with EWS, characterizing them according to students’ profiles, self-efficacy, prior experience, and perspective on data ethics. The results show that students, largely senior workers with strong academic self-efficacy, had limited experience with this method and minimal expectations. But, using the EWS inspired meaningful reflections. Nonetheless, a comparison between the Computer Science and Economics disciplines demonstrated stronger trust and expectation regarding the system and AI for the former. The study emphasized the importance of helping students’ additional experiences and comprehension while embracing AI systems in education to ensure the quality, relevance, and fairness of their educational experience overall. Gli strumenti educativi alimentati dall'intelligenza artificiale (AIEd) includono sistemi di allerta precoce (EWS) per identificare gli studenti universitari a rischio, offrendo assistenza personalizzata. Rivelare le esperienze soggettive degli studenti con gli EWS potrebbe contribuire a una comprensione più profonda di cosa significhi interagire con l'IA in aree della vita umana quali l'insegnamento e l'apprendimento. La nostra indagine ha quindi esplorato le esperienze soggettive degli studenti con gli EWS, caratterizzandole secondo i profili degli studenti, l'autoefficacia, l'esperienza pregressa e la prospettiva sull'etica dei dati. I risultati mostrano che gli studenti, per lo più lavoratori senior con forte autoefficacia accademica, avevano esperienze limitate con questo metodo e aspettative minime. Ciononostante, l'utilizzo degli EWS ha ispirato riflessioni significative. Nonostante ciò, un confronto tra le discipline di Informatica ed Economia ha dimostrato una maggiore fiducia e aspettativa riguardo al sistema e all'IA per la prima. Lo studio ha sottolineato l'importanza di aiutare gli studenti a maturare ulteriori esperienze e comprensioni mentre si avvalgono dei sistemi AI nell'educazione per garantire la qualità, la rilevanza e l'equità della loro esperienza educativa complessiva. Las herramientas educativas impulsadas por la inteligencia artificial (AIEd) incluyen sistemas de alerta temprana (EWS) para identificar a los estudiantes universitarios en riesgo, ofreciendo asistencia personalizada. Revelar las experiencias subjetivas de los estudiantes con los EWS podría contribuir a una comprensión más profunda de lo que significa interactuar con la IA en áreas de la vida humana como la enseñanza y el aprendizaje. Nuestra investigación, por lo tanto, ha explorado las experiencias subjetivas de los estudiantes con los EWS, caracterizándolas según los perfiles de los estudiantes, la autoeficacia, la experiencia previa y la perspectiva sobre la ética de los datos. Los resultados muestran que los estudiantes, en su mayoría trabajadores senior con alta autoeficacia académica, tenían experiencias limitadas con este método y expectativas mínimas. Sin embargo, el uso de los EWS ha inspirado reflexiones significativas. A pesar de ello, una comparación entre las disciplinas de Informática y Economía demostró una mayor confianza y expectativa respecto al sistema y la IA para la primera. El estudio ha subrayado la importancia de ayudar a los estudiantes a desarrollar más experiencias y comprensiones mientras utilizan sistemas de IA en la educación para garantizar la calidad, relevancia y equidad de su experiencia educativa en general. As ferramentas educacionais alimentadas por inteligência artificial (AIEd) incluem sistemas de alerta precoce (EWS) para identificar estudantes universitários em risco, oferecendo assistência personalizada. Revelar as experiências subjetivas dos estudantes com os EWS poderia contribuir para uma compreensão mais profunda do que significa interagir com a IA em áreas da vida humana como o ensino e a aprendizagem. Portanto, nossa investigação explorou as experiências subjetivas dos estudantes com os EWS, caracterizando-as de acordo com os perfis dos estudantes, autoeficácia, experiência anterior e perspectiva sobre a ética dos dados. Os resultados mostram que os estudantes, em sua maioria trabalhadores seniores com forte autoeficácia acadêmica, tinham experiência limitada com este método e expectativas mínimas. No entanto, o uso dos EWS inspirou reflexões significativas. Apesar disso, uma comparação entre as disciplinas de Informática e Economia demonstrou maior confiança e expectativa em relação ao sistema e à IA para a primeira. O estudo enfatizou a importância de auxiliar os estudantes a desenvolverem mais experiências e compreensões ao utilizar sistemas de IA na educação para garantir a qualidade, relevância e equidade de sua experiência educacional geral. Student's experience Artificial intelligence Early Warning System Higher Education Thematic Analysis Esperienze degli studenti Intelligenza artificiale Sistema di allerta precoce Alta formazione Analisi tematica Experiencias de los Estudiantes Intelligencia Artificial Sistema de Alerta Temprana Educación Superior Análisis Temático Experiências dos estudantes Inteligência Artificial Sistema de Alerta Precoce Educação Superior Análise Temática 30 4 2024 11 1 2024 23 4 2024 XXII 1 Copyright (c) 2024 M. Elena Rodríguez, Juliana E. Raffaghelli, David Bañeres, Ana Elena Guerrero-Roldán, Francesca Crudele 2024 Author(s) Creative Commons Attribution 4.0 International License https://creativecommons.org/licenses/by/4.0/

Supplementary Materials : Exploring Higher Education Students’ Experience with AI‍-‍powered Educational Tools: The Case of an Early Warning System

Materiali addizionali : Esplorare l’esperienza degli studenti universitari con strumenti educativi basati sull’IA: Il caso di un sistema di allerta precoce

M. Elena Rodríguez

Faculty of Computer Science, Multimedia and Telecommunications; Universitat Oberta de Catalunya, Spain – mrodriguezgo@uoc.edu

https://orcid.org/0000-0002-8698-4615

Juliana E. Raffaghelli

FISPPA - Faculty of Philosophy, Sociology, Education and Applied Psychology; University of Padua, Italy – juliana.raffaghelli@unipd.it

https://orcid.org/0000-0002-8753-6478

David Bañeres

Faculty of Computer Science, Multimedia and Telecommunications; Universitat Oberta de Catalunya, Spain – dbaneres@uoc.edu

https://orcid.org/0000-0002-0380-1319

Ana Elena Guerrero-Roldán

Faculty of Computer Science, Multimedia and Telecommunications; Universitat Oberta de Catalunya, Spain – aguerreror@uoc.edu

https://orcid.org/0000-0001-7073-7233

Francesca Crudele

FISPPA - Faculty of Philosophy, Sociology, Education and Applied Psychology; University of Padua, Italy – francesca.crudele@phd.unipd.it

https://orcid.org/0000-0003-1598-2791

HOW TO CITE

Rodríguez, M. E., Raffaghelli , J. E., Bañeres , D., Guerrero-Roldán, A. E., & Crudele, F. (2024). Supplementary Materials: Exploring Higher Education Students’ Experience with AI-powered Educational Tools: The Case of an Early Warning System. Formazione & insegnamento , 22 (1), S1‍–‍S5. https://ojs.pensamultimedia.it/index.php/siref/article/view/6976

Context

Courses are organised in virtual classrooms that provide access to the syllabus, resources, exercises, and different communication spaces (teacher’s blackboard, forum, and debate) where teachers and students interact. The educational model is student- centered and based on Continuous Assessment Activities (CAAs), where most students have work and family commitments. There are two types of feedback to support students —general and personalised. General feedback is addressed to all students who share a virtual classroom, and the teachers provide it through their blackboard. After each activity is assessed, each student also receives personalised feedback, together with their mark. CAAs are graded with the following qualitative scale: A (very high), B (high), C+ (sufficient), C- (low), and D (very low), where a C+ is the minimum passing grade. Grade N is used when a student does not submit the CAA.

The EWS

Our EWS (called blinded for the students) offers two outputs. First, a dashboard presents descriptive and predictive information. Second, the intervention mechanism sends feedback messages on the teachers’ behalf. Their design is underpinned by Self-Regulated Learning (SRL) theory ( Pintrich , 2000; Zimmerman, 2008), commonly adopted in learning analytics research ( Jivet et al., 2017; Valle et al., 2021). The dashboard mainly aims to enhance students’ metacognitive skills (supporting performance awareness, reflection, and progress monitoring) and behavioural skills (increasing course engagement and participation). Feedback messages try to improve cognitive skills (supporting goal achievement and performance improvement), behavioural skills (promoting help-seeking behaviour), and motivational/affective skills (by increasing motivation). The goals are improving student performance, increasing retention, and promoting fluent student-teacher communication.

Firstly, the student can review their profile information on the dashboard and the timeline of CAAs proposed in the course jointly with the evolution of the risk level. The dashboard is positioned on the ongoing CAA by default (CAA3 in Figure 6). Information about past CAAs can be accessed by clicking the corresponding CAA. Below the timeline, the student can first see a green-amber-red traffic light showing their low, moderate, or high risk of failing the course, respectively. Adjacent to the traffic light, self-and norm-referenced performance frames regarding the last CAA are provided. Following this, the student has a second traffic light that shows their risk of dropping out of the ongoing CAA. Finally, information about their engagement level in the ongoing CAA is provided (access to the teaching plan and communication spaces). Each area also provides textual information to clarify the display. By clicking the question mark icon, the student can obtain contextual help. The mail envelope icon is for addressing questions and concerns. The information provided by the traffic lights is computed by two different predictive models based on machine learning algorithms trained with anonymised data from past students. The failure risk prediction model issues a prediction after each CAA is graded. In contrast, the dropout risk prediction model is computed daily. The full description and evaluation of the predictive models can be found in Authors et al. (2020).

Figure 5 . Student dashboard (risk level for activities).

The last section of the dashboard (see Figure 5) expands the information predicting a risk of failing. For the graded CAAs, the obtained grade is displayed using triangles (C+ for CAA1 and D for CAA2 for the student depicted in Figure 1), jointly with the risk level zones and their precision. Although such information may be helpful, it is not enough. The EWS also provides information about the risk levels associated with the ongoing CAA grades before submission. The objective is to provide early information about the minimum grade to move past the risk situation. This is achieved by simulating the failure risk prediction model with all possible grades of the CAA being analysed. The student shown in Figure 1 needs a minimum grade of C+ in CAA3 to leave the high-risk situation. Finally, the intervention mechanism triggers different message interventions, which may be adapted to suit the student’s risk situation and profile ( Raffaghelli et al., 2022).

Additional tables

Theme Subtheme n.int. % cov . Fr.code % code n.words % words
Tool characteristics Students’ mention of blinded characteristics during the interview Email 16 38.10% 29 42.03% 889 39.56%
Introduction 2 4.76% 2 2.90% 78 3.47%
Lights 5 11.90% 7 10.14% 224 9.97%
Panel 10 23.81% 13 18.84% 368 16.38%
Tool-Error 9 21.43% 18 26.09% 688 30.62%
MTaI - IF and % 16 76.19% 69 17.42% 2,247 18.65%
UX- Colourlight Colourlight-GeneralCom 1 4.55% 1 4.00% 25 7.65%
UX on blinded characteristics: Always-Green 17 77.27% 18 72.00% 113 34.56%
Mostly-Green 0 0.00% 0 0.00% 0 0.00%
Light MixedGreen -Yellow 4 18.18% 6 24.00% 189 57.80%
Mostly-Yellow 0 0.00% 0 0.00% 0 0.00%
Mostly-Red 0 0.00% 0 0.00% 0 0.00%
MTaI - IF and % 17 80.95% 25 6.31% 327 2.71%
UX-Interest High 17 80.95% 36 87.80% 775 85.26%
UX on blinded characteristics: Overall perceived interest in the tool Middle 2 9.52% 3 7.32% 90 9.90%
Low 2 9.52% 2 4.88% 44 4.84%
MTaI - IF and % 17 80.95% 41 10.35% 909 7.55%
UX-Relevance High 14 73.68% 31 81.58% 925 82.08%
UX on blinded characteristics: relevance of blinded for potential future students Middle 2 10.53% 2 5.26% 51 4.53%
Low 3 15.79% 5 13.16% 151 13.40%
MTaI - IF and % 14 66.67% 38 9.60% 1,127 9.36%
UX-Understanding Excellent 11 52.38% 23 51.11% 488 34.39%
User experience on blinded characteristics: Level of understanding of the tool approach and functions Good 3 14.29% 4 8.89% 151 10.64%
Regular 7 33.33% 18 40.00% 780 54.97%
MTaI - IF and % 11 52.38% 45 11.36% 1,419 11.78%

Table 7. Codes coverage for tool characteristics themes.

Theme Subtheme n.int. % cov . Fr.code % code n.words % words
UX-Proposals Design-Viz 4 11.43% 7 10.61% 249 10.52%
Panel-Viz 10 28.57% 24 36.36% 609 25.72%
User experience on blinded characteristics: Proposals for improvement Information-on-Prediction 5 14.29% 9 13.64% 416 17.57%
Provide-Deeper-Insights 12 34.29% 21 31.82% 950 40.12%
Tutorial 4 11.43% 5 7.58% 144 6.08%
MTaI - IF and % 12 57.14% 66 16.67% 2,368 15.59%

Table 8. Codes coverage for UX-proposals theme.

Theme Subtheme n.int. % cov . Fr.code % code n.words % words
Opinion on data capture Open-Cautious 8 33.33% 12 30.00% 521 33.85%
Open-Proactive 13 54.17% 24 60.00% 893 58.02%
Problems connected to data privacy Restricted-Cautious 3 12.50% 4 10.00% 125 8.12%
Restricted-Proactive 0 0.00% 0 0.00% 0 0.00%
MTaI - IF and % 13 61.90% 40 10.10% 1,539 12.78%

Table 9. Codes coverage for data capture theme.

Theme Subtheme n.int. % cov . Fr.code % code n.words % words
AI-Experience AutomatedEdSystem 7 36.84% 10 40.00% 235 29.78%
Overall opinion experience and opinion on AI systems in society and education ImageProcessing 3 15.79% 5 20.00% 252 31.94%
RecommenderSystem 7 36.84% 8 32.00% 211 26.74%
TutorEd 2 10.53% 2 8.00% 91 11.53%
MTaI - IF and % 7 33.33% 25 6.31% 789 6.55%
Expectations Curiosity 3 30.00% 3 25.00% 53 20.95%
Initial expectations relating to the usage of an EWS like blinded HighExpectations 1 10.00% 1 8.33% 38 15.02%
LowExpectations 6 60.00% 8 66.67% 162 64.03%
MTaI - IF and % 6 28.57% 12 3.03% 253 2.10%

Table 10. Codes coverage for AI experience and expectations themes.

Theme Subtheme n.int. % cov . Fr.code % code n.words % words
Self-efficacy Very-High 4 17.39% 4 11.43% 109 10.21%
Students’ perception of ability to organise their academic work and get good results High 11 47.83% 14 40.00% 409 38.30%
NeitherLow-NorHigh 3 13.04% 8 22.86% 330 30.90%
Low 5 21.74% 9 25.71% 220 20.60%
Very-Low 0 0.00% 0 0.00% 0 0.00%
MTaI - IF and % 11 52.38% 35 8.84% 1,068 8.87%

Table 11. Codes coverage for self-efficacy theme.

References RaffaghelliJ. E. RodríguezM. E. Guerrero-RoldánA.-E. BañeresD. Applying the UTAUT model to explain the students’ acceptance of an early warning system in Higher Education Computers & Education 2022 182 10.1016/j.compedu.2022.104468