Comparing pivotal and REML-based confidence intervals for heritability

Heritability quantifies the extent to which a physical characteristic is passed from one generation to the next. From a statistical perspective, heritability is the proportion of the phenotypic variance attributable to (additive) genetic effects and is equal to a function of variance components in linear mixed models. Relying on normal distribution assumptions, one can compute exact confidence intervals for heritability using a pivotal quantity procedure. Alternatively, large-sample properties of the restricted maximum likelihood (REML) estimator can be used to construct asymptotic confidence intervals for heritability. Exact and asymptotic intervals are compared to one another in a variety of situations, including a mixed model having correlated loineye muscle area measurements and balanced one-way random effects models having groups of correlated responses. In some cases the two interval methods yield vastly different results and the REML-based confidence interval does not maintain the nominal coverage value even for seemingly large sample sizes. For finite sample size applications, the validity of the REML-based procedure depends on the correlation structure of the data.