Statistical assumptions in L2 research: A systematic review

Statistical tests carry with them a number of assumptions that must be checked. Failing to do so and to report the results of such preliminary analyses introduce a potential threat to the internal validity of a study and to our ability as consumers to put faith in study findings. This article systematically examines the reporting of checks on assumptions in two major journals of second-language (L2) research (Language Learning and Second Language Research) over a span of six years (2012–2017 inclusive). Using synthetic techniques, each study in the sample (K = 107) was coded for whether or not the assumptions associated with five statistical tests were reported: independent samples t-test, one-way ANOVA, Pearson correlation, multiple regression, and chi-square. In doing so, two coding standards (‘stringent’ and ‘lenient’) were adopted to arrive at a more comprehensive picture of researchers’ practices and to reflect the different types of checks required by quantitative analyses. The lenient standard was coded as being met if one or more of the assumptions for a given analytical technique was reported; the stringent standard, by contrast, was recorded as met if all assumptions were reported and checked. Overall, 17% and 24% of the sample were found to have met the stringent and lenient standards, respectively. The study also addresses differences in the checking and reporting of assumptions across the five statistical procedures we examined. Despite overall weak reporting, we note several improvements taking place over time. Suggestions for improving data analytic and reporting practices in the field are discussed as well as implications for researcher training (see Loewen et al., 2014; Plonsky, 2014).