TY - GEN
T1 - How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
AU - Choudhuri, Rudrajit
AU - Liu, Dylan
AU - Steinmacher, Igor
AU - Gerosa, Marco
AU - Sarma, Anita
N1 - Publisher Copyright:
© 2024 ACM.
PY - 2024
Y1 - 2024
N2 - Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
AB - Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
KW - ChatGPT
KW - Empirical Study
KW - Generative AI
KW - Software Engineering
UR - http://www.scopus.com/inward/record.url?scp=85196840199&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85196840199&partnerID=8YFLogxK
U2 - 10.1145/3597503.3639201
DO - 10.1145/3597503.3639201
M3 - Conference contribution
AN - SCOPUS:85196840199
T3 - Proceedings - International Conference on Software Engineering
SP - 2270
EP - 2282
BT - Proceedings - 2024 ACM/IEEE 44th International Conference on Software Engineering, ICSE 2024
PB - IEEE Computer Society
T2 - 44th ACM/IEEE International Conference on Software Engineering, ICSE 2024
Y2 - 14 April 2024 through 20 April 2024
ER -