Innovations in Education, CDS Faculty Deploys AI Tutor
As AI becomes increasingly integrated into higher education, educators continue to face challenges with LLMs such as ChatGPT and other artificial intelligence chatbots. Nearly two years after ChatGPT’s debut, the question persists: How can faculty ensure rich learning when students have access to tools that might bypass the learning process? Kevin Gold, CDS associate professor of the practice, delved into this question.
During spring 2024, Gold enlisted 127 students from his DS 110 “Introduction to Data Science with Python” course to test an LLM-based homework tutor. Building on the Socratic teaching method — a learning approach that promotes student engagement through guided questioning rather than direct answers — Gold developed a GPT-4-powered bot as the core of the AI tool. The bot was given the homework assignments along with solutions and was instructed not to provide answers directly. Gold appropriately named the bot the DS 110 Socratic AI Tutor.
“The goal is to bring students into the interaction, work for the answers using the Socratic method, and not just give away the answer immediately,” he said.
Gold emphasized the importance of crafting an experience that students would choose over the standard ChatGPT. The new system offered three key advantages: First, it utilized the more advanced GPT-4, which was not available for free to students. Second, students were reassured by the fact that the system was provided with solutions, which alleviated concerns about unreliable or “hallucinated” responses. Third, the course-approved interaction was perceived as more legitimate and conducive to learning, unlike the direct use of ChatGPT, which many students felt was less appropriate for their educational needs.
“The DS 110 AI tutor would not give away the answers but instead guided us through the questions to enable us to find the answers on our own,” said Amanda Atlas (CDS’27), one of Gold’s students. “The tutor assisted with questions I needed help understanding, such as how to code and how to approach a problem.”
Throughout the semester, Gold recorded interactions and anonymously logged and coded 17 attributes, including helpfulness and the extent of answer leakage. Surveys were also deployed to better understand students’ perceptions and use cases.
“The tutor was generally perceived as helpful to learning by students and an independent coder and only rarely directly leaked the solution,” Gold said.
The average student rating of helpfulness was 4.0/5, comparable to the helpfulness of 4.2/5 for teaching assistant office hours. Some AI tutor tasks that the majority of students rated “very helpful” included clarifying what questions were asking, answering questions about Python syntax, and describing general programming approaches. Students rated its ability to help debug or provide programming examples as “moderately helpful.”
Gold and Educational Technologist Shuang Geng, an expert in learning analytics with BU Digital Learning & Innovation, conducted an analysis and co-authored a paper on the LLM-based homework tutor. The approach received the “Best-in-Session” award at the 5th International Conference on Artificial Intelligence in Education Technology held in Spain last summer.
“I think there was a general feeling at the conference that this was addressing an important problem,” Gold said. “Students need programming homework in some form to get the practice they need to achieve mastery, and large language models are short-circuiting that. The idea that we could turn AI assistance from a bug in the process to a feature is very attractive.”
The AI Socratic Tutor inspired the early-stage development of a more general, reusable, and retargetable AI Course Assistant spearheaded by Thomas Gardos, associate professor of the practice and director of Master’s in Data Science program at CDS. To that end, Professor Gardos and his students are developing an open-source LLM-based AI course assistant that can target a particular course and be available to students via a chatbot interface. Using advanced Retrieval Augmentation Generation (RAG) methods, the AI assistant uses a Socratic style to guide students’ learning and point them to relevant course content as part of their answer. By mining the chat histories, they are able to give guidance to instructors on where students might be struggling. Beyond deployment in courses, this project is meant as a teaching tool, for students who want to understand more about LLM deployment and also contribute in its development.
- By Maureen McCarthy