TY - GEN
T1 - Outcomes, Perceptions, and Interaction Strategies of Novice Programmers Studying with ChatGPT
AU - Penney, Jacob
AU - Acharya, Pawan
AU - Hilbert, Peter
AU - Parekh, Priyanka
AU - Sarma, Anita
AU - Steinmacher, Igor
AU - Gerosa, Marco Aurelio
N1 - Publisher Copyright:
© 2025 Copyright held by the owner/author(s).
PY - 2025/7/7
Y1 - 2025/7/7
N2 - Large Language Model (LLM) conversational agents are increasingly used in programming education, yet we still lack insight into how novices engage with them for conceptual learning compared with human tutoring. This mixed-methods study compared learning outcomes and interaction strategies of novices using ChatGPT or human tutors. A controlled lab study with 20 students enrolled in introductory programming courses revealed that students employ markedly different interaction strategies with AI versus human tutors: ChatGPT users relied on brief, zero-shot prompts and received lengthy, context-rich responses but showed minimal prompt refinement, while those working with human tutors provided more contextual information and received targeted explanations. Although students distrusted ChatGPT’s accuracy, they paradoxically preferred it for basic conceptual questions due to reduced social anxiety. We offer empirically grounded recommendations for developing AI literacy in computer science education and designing learning-focused conversational agents that balance trust-building with maintaining the social safety that facilitates uninhibited inquiry.
AB - Large Language Model (LLM) conversational agents are increasingly used in programming education, yet we still lack insight into how novices engage with them for conceptual learning compared with human tutoring. This mixed-methods study compared learning outcomes and interaction strategies of novices using ChatGPT or human tutors. A controlled lab study with 20 students enrolled in introductory programming courses revealed that students employ markedly different interaction strategies with AI versus human tutors: ChatGPT users relied on brief, zero-shot prompts and received lengthy, context-rich responses but showed minimal prompt refinement, while those working with human tutors provided more contextual information and received targeted explanations. Although students distrusted ChatGPT’s accuracy, they paradoxically preferred it for basic conceptual questions due to reduced social anxiety. We offer empirically grounded recommendations for developing AI literacy in computer science education and designing learning-focused conversational agents that balance trust-building with maintaining the social safety that facilitates uninhibited inquiry.
KW - AI Literacy
KW - CS1
KW - Computer Science Pedagogy
KW - Conversational Agents
KW - Large Language Models
KW - Software Engineering Education
UR - https://www.scopus.com/pages/publications/105011653367
UR - https://www.scopus.com/inward/citedby.url?scp=105011653367&partnerID=8YFLogxK
U2 - 10.1145/3719160.3736625
DO - 10.1145/3719160.3736625
M3 - Conference contribution
AN - SCOPUS:105011653367
T3 - CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces
BT - CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces
PB - Association for Computing Machinery, Inc
T2 - 7th Conference on Conversational User Interfaces, CUI 2025
Y2 - 8 July 2025 through 10 July 2025
ER -