Communicating Study Design Trade-offs in Software Engineering

Martin P. Robillard, Deeksha M. Arya, Neil A. Ernst, Jin L.C. Guo, Maxime Lamothe, Mathieu Nassif, Nicole Novielli, Alexander Serebrenik, Igor Steinmacher, Klaas Jan Stol

Research output: Contribution to journalArticlepeer-review


Reflecting on the limitations of a study is a crucial part of the research process. In software engineering studies, this reflection is typically conveyed through discussions of study limitations or threats to validity. In current practice, such discussions seldom provide sufficient insight to understand the rationale for decisions taken before and during the study, and their implications. We revisit the practice of discussing study limitations and threats to validity and identify its weaknesses. We propose to refocus this practice of self-reflection to a discussion centered on the notion of trade-offs. We argue that documenting trade-offs allows researchers to clarify how the benefits of their study design decisions outweigh the costs of possible alternatives. We present guidelines for reporting trade-offs in a way that promotes a fair and dispassionate assessment of researchers' work.

Original languageEnglish (US)
Article number112
JournalACM Transactions on Software Engineering and Methodology
Issue number5
StatePublished - Jun 4 2024


  • Empirical software engineering
  • empirical study design
  • metascience
  • research design trade-offs
  • research validity
  • threats to validity

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'Communicating Study Design Trade-offs in Software Engineering'. Together they form a unique fingerprint.

Cite this