Instrument Validity in Manuscripts Published in the Journal of Agricultural Education between 2007 and 2016

Abstract

We examined authors’ treatment of instrument validity in a stratified random sample (n = 94) of quantitative studies published in the Journal of Agricultural Education from 2007 to 2016. A majority (78.7%) of studies reported use of either a researcher-developed instrument (41.5%) or an existing instrument modified by the researchers (37.2%). In 67.0% of articles, authors validated the instrument for the study reported; in 16.0%, the authors claimed instrument validity based on previous studies. Authors made no claim for instrument validity in 11.7% of articles and claimed an unspecified form of validity in another 11.7%. Among the 72 articles where specific validity claims were made, 67 (93.1%) claimed face and content validity, either alone or in combination with other forms of validity. Claims for content, concurrent, and discriminant validity, either alone or in combination with other forms of validity, were made in 11 (15.3%) of the 72 articles. Among 70 articles claiming face, content, or construct validity, 91.4% included a description of the validation panel; panelists were most often described as ‘experts’ (70.3%), although their area(s) of expertise were specified in only 29.7% of articles. We conclude with specific recommendations intended to shift the profession’s subjective norms related to instrument validity.