TY - JOUR
T1 - A Bayesian Approach to Measuring Evidence in L2 Research
T2 - An Empirical Investigation
AU - Norouzian, Reza
AU - Miranda, Michael De
AU - Plonsky, Luke
N1 - Publisher Copyright:
© National Federation of Modern Language Teachers Associations
PY - 2019/3/1
Y1 - 2019/3/1
N2 - Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.
AB - Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.
KW - Bayes factor
KW - Bayesian hypothesis testing
KW - null hypothesis testing
KW - open science framework
KW - p-value
KW - quantitative L2 research
UR - http://www.scopus.com/inward/record.url?scp=85061001909&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85061001909&partnerID=8YFLogxK
U2 - 10.1111/modl.12543
DO - 10.1111/modl.12543
M3 - Article
AN - SCOPUS:85061001909
SN - 0026-7902
VL - 103
SP - 248
EP - 261
JO - Modern Language Journal
JF - Modern Language Journal
IS - 1
ER -