How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering

Rudrajit Choudhuri, Dylan Liu, Igor Steinmacher, Marco Gerosa, Anita Sarma

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.

Original languageEnglish (US)
Title of host publicationProceedings - 2024 ACM/IEEE 44th International Conference on Software Engineering, ICSE 2024
PublisherIEEE Computer Society
Pages2270-2282
Number of pages13
ISBN (Electronic)9798400702174
DOIs
StatePublished - 2024
Event44th ACM/IEEE International Conference on Software Engineering, ICSE 2024 - Lisbon, Portugal
Duration: Apr 14 2024Apr 20 2024

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257

Conference

Conference44th ACM/IEEE International Conference on Software Engineering, ICSE 2024
Country/TerritoryPortugal
CityLisbon
Period4/14/244/20/24

Keywords

  • ChatGPT
  • Empirical Study
  • Generative AI
  • Software Engineering

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering'. Together they form a unique fingerprint.

Cite this