TY - GEN
T1 - Unveiling the Potential of a Conversational Agent in Developer Support
T2 - 1st ACM International Conference on AI-Powered Software, AIware 2024, co-located with the ACM International Conference on the Foundations of Software Engineering, FSE 2024
AU - Correia, João
AU - Nicholson, Morgan C.
AU - Coutinho, Daniel
AU - Barbosa, Caio
AU - Castelluccio, Marco
AU - Gerosa, Marco
AU - Garcia, Alessandro
AU - Steinmacher, Igor
N1 - Publisher Copyright:
© 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
PY - 2024/7/10
Y1 - 2024/7/10
N2 - Large language models and other foundation models (FMs) boost productivity by automating code generation, supporting bug fixes, and generating documentation. We propose that FMs can further support Open Source Software (OSS) projects by assisting developers and guiding the community. Currently, core developers and maintainers answer queries about processes, architecture, and source code, but their time is limited, often leading to delays. To address this, we introduce DevMentorAI, a tool that enhances developer-project interactions by leveraging source code and technical documentation. DevMentorAI uses the Retrieval Augmented Generation (RAG) architecture to identify and retrieve relevant content for queries. We evaluated DevMentorAI with a case study on PDF.js project, using real questions from a development chat room and comparing the answers provided by DevMentorAI to those from humans. A Mozilla expert rated the answers, finding DevMentorAI's responses more satisfactory in 8/14 of cases, equally satisfactory in 3/14, and less satisfactory in 3/14. These results demonstrate the potential of using foundation models and the RAG approach to support developers and reduce the burden on core developers.
AB - Large language models and other foundation models (FMs) boost productivity by automating code generation, supporting bug fixes, and generating documentation. We propose that FMs can further support Open Source Software (OSS) projects by assisting developers and guiding the community. Currently, core developers and maintainers answer queries about processes, architecture, and source code, but their time is limited, often leading to delays. To address this, we introduce DevMentorAI, a tool that enhances developer-project interactions by leveraging source code and technical documentation. DevMentorAI uses the Retrieval Augmented Generation (RAG) architecture to identify and retrieve relevant content for queries. We evaluated DevMentorAI with a case study on PDF.js project, using real questions from a development chat room and comparing the answers provided by DevMentorAI to those from humans. A Mozilla expert rated the answers, finding DevMentorAI's responses more satisfactory in 8/14 of cases, equally satisfactory in 3/14, and less satisfactory in 3/14. These results demonstrate the potential of using foundation models and the RAG approach to support developers and reduce the burden on core developers.
KW - Conversational Agents
KW - Developer Assistance
KW - Large Language Models
KW - Software Development
KW - Software Engineering
UR - http://www.scopus.com/inward/record.url?scp=85199904763&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85199904763&partnerID=8YFLogxK
U2 - 10.1145/3664646.3664758
DO - 10.1145/3664646.3664758
M3 - Conference contribution
AN - SCOPUS:85199904763
T3 - AIware 2024 - Proceedings of the 1st ACM International Conference on AI-Powered Software, Co-located with: ESEC/FSE 2024
SP - 10
EP - 18
BT - AIware 2024 - Proceedings of the 1st ACM International Conference on AI-Powered Software, Co-located with
A2 - Adams, Bram
A2 - Zimmermann, Thomas
A2 - Ozkaya, Ipek
A2 - Lin, Dayi
A2 - Zhang, Jie M.
PB - Association for Computing Machinery, Inc
Y2 - 15 July 2024 through 16 July 2024
ER -