What to Expect from Code Review Bots on GitHub?: A Survey with OSS Maintainers

Mairieli Wessel; Alexander Serebrenik; Igor Wiese; Igor Steinmacher; Marco A. Gerosa

doi:10.1145/3422392.3422459

What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers

Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, Marco A. Gerosa

Informatics, Computing, and Cyber Systems, School of

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

23 Scopus citations

Abstract

Software bots are used by Open Source Software (OSS) projects to streamline the code review process. Interfacing between developers and automated services, code review bots report continuous integration failures, code quality checks, and code coverage. However, the impact of such bots on maintenance tasks is still neglected. In this paper, we study how project maintainers experience code review bots. We surveyed 127 maintainers and asked about their expectations and perception of changes incurred by code review bots. Our findings reveal that the most frequent expectations include enhancing the feedback bots provide to developers, reducing the maintenance burden for developers, and enforcing code coverage. While maintainers report that bots satisfied their expectations, they also perceived unexpected effects, such as communication noise and newcomers' dropout. Based on these results, we provide a series of implications for bot developers, as well as insights for future research.

Original language	English (US)
Title of host publication	Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020
Publisher	Association for Computing Machinery
Pages	457-462
Number of pages	6
ISBN (Electronic)	9781450387538
DOIs	https://doi.org/10.1145/3422392.3422459
State	Published - Oct 21 2020
Event	34th Brazilian Symposium on Software Engineering, SBES 2020 - Virtual, Online, Brazil Duration: Oct 21 2020 → Oct 23 2020

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	34th Brazilian Symposium on Software Engineering, SBES 2020
Country/Territory	Brazil
City	Virtual, Online
Period	10/21/20 → 10/23/20

Keywords

code review
open source software
pull-based model
software bots

ASJC Scopus subject areas

Software
Human-Computer Interaction
Computer Vision and Pattern Recognition
Computer Networks and Communications

Access to Document

10.1145/3422392.3422459

Cite this

Wessel, M., Serebrenik, A., Wiese, I., Steinmacher, I., & Gerosa, M. A. (2020). What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. In Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020 (pp. 457-462). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3422392.3422459

What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. / Wessel, Mairieli; Serebrenik, Alexander; Wiese, Igor et al.
Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020. Association for Computing Machinery, 2020. p. 457-462 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Wessel, M, Serebrenik, A, Wiese, I, Steinmacher, I & Gerosa, MA 2020, What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. in Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 457-462, 34th Brazilian Symposium on Software Engineering, SBES 2020, Virtual, Online, Brazil, 10/21/20. https://doi.org/10.1145/3422392.3422459

@inproceedings{81fde2c38eda4a609933dd0d7897b0c6,

title = "What to Expect from Code Review Bots on GitHub?: A Survey with OSS Maintainers",

abstract = "Software bots are used by Open Source Software (OSS) projects to streamline the code review process. Interfacing between developers and automated services, code review bots report continuous integration failures, code quality checks, and code coverage. However, the impact of such bots on maintenance tasks is still neglected. In this paper, we study how project maintainers experience code review bots. We surveyed 127 maintainers and asked about their expectations and perception of changes incurred by code review bots. Our findings reveal that the most frequent expectations include enhancing the feedback bots provide to developers, reducing the maintenance burden for developers, and enforcing code coverage. While maintainers report that bots satisfied their expectations, they also perceived unexpected effects, such as communication noise and newcomers' dropout. Based on these results, we provide a series of implications for bot developers, as well as insights for future research.",

keywords = "code review, open source software, pull-based model, software bots",

author = "Mairieli Wessel and Alexander Serebrenik and Igor Wiese and Igor Steinmacher and Gerosa, {Marco A.}",

note = "Publisher Copyright: {\textcopyright} 2020 ACM.; 34th Brazilian Symposium on Software Engineering, SBES 2020 ; Conference date: 21-10-2020 Through 23-10-2020",

year = "2020",

month = oct,

day = "21",

doi = "10.1145/3422392.3422459",

language = "English (US)",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "457--462",

booktitle = "Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020",

}

TY - GEN

T1 - What to Expect from Code Review Bots on GitHub?

T2 - 34th Brazilian Symposium on Software Engineering, SBES 2020

AU - Wessel, Mairieli

AU - Serebrenik, Alexander

AU - Wiese, Igor

AU - Steinmacher, Igor

AU - Gerosa, Marco A.

PY - 2020/10/21

Y1 - 2020/10/21

N2 - Software bots are used by Open Source Software (OSS) projects to streamline the code review process. Interfacing between developers and automated services, code review bots report continuous integration failures, code quality checks, and code coverage. However, the impact of such bots on maintenance tasks is still neglected. In this paper, we study how project maintainers experience code review bots. We surveyed 127 maintainers and asked about their expectations and perception of changes incurred by code review bots. Our findings reveal that the most frequent expectations include enhancing the feedback bots provide to developers, reducing the maintenance burden for developers, and enforcing code coverage. While maintainers report that bots satisfied their expectations, they also perceived unexpected effects, such as communication noise and newcomers' dropout. Based on these results, we provide a series of implications for bot developers, as well as insights for future research.

AB - Software bots are used by Open Source Software (OSS) projects to streamline the code review process. Interfacing between developers and automated services, code review bots report continuous integration failures, code quality checks, and code coverage. However, the impact of such bots on maintenance tasks is still neglected. In this paper, we study how project maintainers experience code review bots. We surveyed 127 maintainers and asked about their expectations and perception of changes incurred by code review bots. Our findings reveal that the most frequent expectations include enhancing the feedback bots provide to developers, reducing the maintenance burden for developers, and enforcing code coverage. While maintainers report that bots satisfied their expectations, they also perceived unexpected effects, such as communication noise and newcomers' dropout. Based on these results, we provide a series of implications for bot developers, as well as insights for future research.

KW - code review

KW - open source software

KW - pull-based model

KW - software bots

UR - http://www.scopus.com/inward/record.url?scp=85099338284&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85099338284&partnerID=8YFLogxK

U2 - 10.1145/3422392.3422459

DO - 10.1145/3422392.3422459

M3 - Conference contribution

AN - SCOPUS:85099338284

T3 - ACM International Conference Proceeding Series

SP - 457

EP - 462

BT - Proceedings - 34th Brazilian Symposium on Software Engineering, SBES 2020

PB - Association for Computing Machinery

Y2 - 21 October 2020 through 23 October 2020

ER -