How Many Raters Can Be Enough: G Theory Applied to Assessment and Measurement of L2 Speech Perception

Kevin Hirschi, Okim Kang

Research output: Contribution to journalArticlepeer-review


This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on G studies-a method of breaking down measurement variance-and D studies-a predictive study of the impact on reliability when modifying the number of raters, items, or other facets that assist the field in adopting measurement designs that include comprehensibility, accentedness, and intelligibility. When data from a single audio sample per learner were subjected to D-studies, we find that both semantic differential and rubric scales for comprehensibility were reliable at the.90 level with about 15 trained raters or 50 untrained crowdsourced raters. In order to offer generalizable and dependable evaluations, empirically informed recommendations are given, including considerations for the number of speech samples rated, or the granularity of the scales for various assessment and research purposes.

Original languageEnglish (US)
Pages (from-to)213-230
Number of pages18
JournalLanguage Teaching Research Quarterly
StatePublished - 2023


  • Accentedness
  • Comprehensibility
  • Generalizability
  • Intelligibility
  • L2 Speech
  • Measurement

ASJC Scopus subject areas

  • Education
  • Language and Linguistics
  • Linguistics and Language


Dive into the research topics of 'How Many Raters Can Be Enough: G Theory Applied to Assessment and Measurement of L2 Speech Perception'. Together they form a unique fingerprint.

Cite this