TY - GEN
T1 - Towards a Dataset to Assess Conversational Agent's Efficacy in Mentoring Newcomer Developers
AU - Etchie, Misan Paul
AU - Beach, Hunter
AU - Felizardo, Katia
AU - Steinmacher, Igor
N1 - Publisher Copyright:
© 2025 IEEE.
PY - 2025
Y1 - 2025
N2 - Many Open Source Software (OSS) maintainers experience burnout due to the constant need to support and answer questions posed by newcomer developers. This leads to slower response times, and unanswered questions which discourage newcomers and hinder community growth. Research indicates that conversational agents trained on community data can answer common questions, thus reducing maintainers' work-load. However, conversational agents should only be used if they are useful to newcomers. To help prove a conversational agent can be useful, we manually curated a dataset of questions posted to the JabRef issue tracker and Gitter, submitted by newcomers. This dataset of questions will be invaluable for conducting studies using LLMs to support newcomers. This dataset paper details our collection process, describes the data, and offers reflections on future research.
AB - Many Open Source Software (OSS) maintainers experience burnout due to the constant need to support and answer questions posed by newcomer developers. This leads to slower response times, and unanswered questions which discourage newcomers and hinder community growth. Research indicates that conversational agents trained on community data can answer common questions, thus reducing maintainers' work-load. However, conversational agents should only be used if they are useful to newcomers. To help prove a conversational agent can be useful, we manually curated a dataset of questions posted to the JabRef issue tracker and Gitter, submitted by newcomers. This dataset of questions will be invaluable for conducting studies using LLMs to support newcomers. This dataset paper details our collection process, describes the data, and offers reflections on future research.
KW - Dataset
KW - Human-Bot Collaboration
KW - Newcomers
KW - Open Source Software
UR - https://www.scopus.com/pages/publications/105010617080
UR - https://www.scopus.com/inward/citedby.url?scp=105010617080&partnerID=8YFLogxK
U2 - 10.1109/BotSE67031.2025.00018
DO - 10.1109/BotSE67031.2025.00018
M3 - Conference contribution
AN - SCOPUS:105010617080
T3 - Proceedings - 2025 IEEE/ACM International Workshop on Bots in Software Engineering, BotSE 2025
SP - 48
EP - 52
BT - Proceedings - 2025 IEEE/ACM International Workshop on Bots in Software Engineering, BotSE 2025
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 6th IEEE/ACM International Workshop on Bots in Software Engineering, BotSE 2025
Y2 - 27 April 2025
ER -