TY - GEN
T1 - Data extraction for systematic mapping study using a large language model - a proof-of-concept study in software engineering
AU - Felizardo, Katia Romero
AU - Steinmacher, Igor
AU - Lima, Márcia Sampaio
AU - Deizepe, Anderson
AU - Conte, Tayana Uchôa
AU - Barcellos, Monalessa Perini
N1 - Publisher Copyright:
© 2024 ACM.
PY - 2024/10/24
Y1 - 2024/10/24
N2 - Context: Systematic mapping studies (SMS) are adopted in Software Engineering (SE) to select and synthesize relevant literature on a research topic and, thus, support evidence-based decision-making. Performing SMS is effort-demanding and time-consuming. Hence, using tools is beneficial. Large Language Models (LLMs) such as ChatGPT-4.o can potentially accelerate repetitive activities, such as data extraction in SMS, saving time and effort. Goal: We conducted this work to evaluate and provide preliminary evidence on how ChatGPT-4.o can support data extraction in SMS. Method: We performed a proof-of-concept study and assessed the results' accuracy of using ChatGPT 4.0 to extract data in one SMS compared to the results produced manually. Results: The accuracy of ChatGPT-4.o was 87.83%. Conclusions: Our preliminary findings suggest that entirely replacing the manual data extraction with ChatGPT-4.o is not recommended. However, employing ChatGPT for semi-automated data extraction to aid in evidence synthesis in SMS is promising.
AB - Context: Systematic mapping studies (SMS) are adopted in Software Engineering (SE) to select and synthesize relevant literature on a research topic and, thus, support evidence-based decision-making. Performing SMS is effort-demanding and time-consuming. Hence, using tools is beneficial. Large Language Models (LLMs) such as ChatGPT-4.o can potentially accelerate repetitive activities, such as data extraction in SMS, saving time and effort. Goal: We conducted this work to evaluate and provide preliminary evidence on how ChatGPT-4.o can support data extraction in SMS. Method: We performed a proof-of-concept study and assessed the results' accuracy of using ChatGPT 4.0 to extract data in one SMS compared to the results produced manually. Results: The accuracy of ChatGPT-4.o was 87.83%. Conclusions: Our preliminary findings suggest that entirely replacing the manual data extraction with ChatGPT-4.o is not recommended. However, employing ChatGPT for semi-automated data extraction to aid in evidence synthesis in SMS is promising.
KW - ChatGPT
KW - Data Extraction
KW - LLM
KW - Mapping Study
UR - http://www.scopus.com/inward/record.url?scp=85210568147&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85210568147&partnerID=8YFLogxK
U2 - 10.1145/3674805.3690743
DO - 10.1145/3674805.3690743
M3 - Conference contribution
AN - SCOPUS:85210568147
T3 - International Symposium on Empirical Software Engineering and Measurement
SP - 407
EP - 413
BT - Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024
PB - IEEE Computer Society
T2 - 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024
Y2 - 24 October 2024 through 25 October 2024
ER -