Conception et évaluation d’un agent conversationnel enrichi par la génération augmentée par récupération : effet sur l’acquisition des connaissances des personnes apprenantes, l’utilisabilité perçue et l’expérience d’interaction

Abstract

Artificial intelligence (AI) has significantly evolved in recent years, particularly with the emergence of large language models (LLMs) such as the generative pre-trained transformer (GPT) family. These models, capable of generating fluent and contextually relevant text, show promising potential for transforming various sectors, including education. However, their application in educational contexts presents certain limitations, notably hallucinations, or the generation of incorrect information, which can negatively impact learners’ learning experience. To mitigate these limitations, Retrieval-Augmented Generation (RAG) has been integrated into the language models. This approach enhances the accuracy, relevance and reliability of generated responses by combining LLM capabilities with information retrieval from a pre-constructed knowledge base enriched with relevant documents. However, the application of RAG-enhanced language models in educational settings, particularly in MOOCs, remains underexplored, especially regarding their impact on knowledge acquisition and learners’ interaction experience. In this study, we designed and developed a GPT-4-powered, RAG-enhanced conversational agent to provide real-time, contextually relevant support to learners in a MOOC. This agent helps learners to clarify complex concepts while guiding them throughout their learning process. Our conversational agent was evaluated with 25 learners enrolled in a MOOC. An analysis of the results revealed that knowledge acquisition was significantly improved in the experimental group compared to the control group. Additionally, the conversational agent received a high score on the System Usability Scale (SUS), indicating a positive perception of its usability. Semi-structured interviews further highlighted a generally favorable interaction experience with the agent. These findings underscore the potential of generative AI-powered conversational agents enriched with RAG to enhance learning in online learning environments, including MOOCs. They also pave the way for future research on the role of such agents as intelligent learning companions, capable of adapting their support to learners’ specific needs.