Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program

Jill F. Kilanowski; S. Dee Jepsen; Rebecca Berenbon

doi:10.5032/jae.v66i4.3277

Authors

Jill F. Kilanowski The Ohio State University https://orcid.org/0000-0003-0455-1275
S. Dee Jepsen The Ohio State University https://orcid.org/0000-0001-7221-5375
Rebecca Berenbon The Ohio State University https://orcid.org/0000-0002-8744-3132

DOI:

https://doi.org/10.5032/jae.v66i4.3277

Keywords:

reliability scores, item test bank, agricultural youth, tractor safety

Abstract

The purpose of the study was to evaluate the reliability of a newly developed test bank of exam questions to support the U.S. Department of Labor tractor and machinery safety certificate’s written exam. This test bank was developed by a panel of eight subject matter experts, from different U.S. geographic regions, in face-to-face meetings for item writing, to assess subject knowledge in middle and high school students aged 13-to-18 years old. The guiding theoretical frameworks were Webb’s Depth of Knowledge and Bloom’s Taxonomy. Via multiple recruitment efforts, high school agricultural educators and community 4-H advisors who taught youth certification programs were recruited to serve as classroom facilitators and replace their current knowledge exam with the study’s WebXam online exam. Students (n = 96) from three states and six classrooms participated in the study, representing four different program formats. Results showed strong reliability (a = 0.93) for the exam, with no significant difference in the likelihood of passing based on age or sex. Seven themes were included in the 70-item test bank, where all questions were answered correctly by at least 30% of respondents, indicating there were no questions too difficult for the test-taker population. The results will serve as a foundation for establishing an online testing platform for educators who offer tractor and machinery safety trainings across the U.S., and in-turn may also create program consistency for the DOL written exam with a national test bank of valid and reliable questions.

Downloads

Download data is not yet available.

References

Agricultural Safety and Health eXtension Community of Practice. (2019). Farm Safety Makes you the Hero. NSTMOP Program Overview. Retrieved May 12, 2025, from https://ag-safety.extension.org/nstmop-program-overview/

Agricultural Safety and Health eXtension Community of Practice. (2025). AgSafety4U Certificate Course. Retrieved May 13, 2025, from https://ag-safety.extension.org/agsafety4u-certificate-course/

Arduini-Van, N. (2020). Educational Psychology. Press Books. https://edpsych.pressbooks.sunycreate.cloud/chapter/teacher-made-assessments/

Assessment Systems Corporation. (1979). Laboratory of Educational Research Test Analysis Package (Lertap5). (Version 11) [Computer software]. https://lertap5.com/HTMLHelp/HTML/index.html?how_to_get_lertap.htm

Berman, A. I., Haertel, E. H., & Pellegrino, J. W. (2020). Comparability of large-scale educational assessments: Issues and recommendations. National Academy of Education. https://naeducation.org/wp-content/uploads/2020/06/Comparability-of-Large-Scale-Educational-Assessments.pdf DOI: https://doi.org/10.31094/2020/1

Bloom, B. S.; Engelhart, M. D.; Furst, E. J.; Hill, W. H.; Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Vol. Handbook I: Cognitive domain. David McKay Company.

Center on Education and Teaching for Employment (CETE). (n.d.). About WebXam. Retrieved January 11, 2025, from https://www.webxam.org/Admin/Docs#:~:text=WebXam%20is%20an%20online%20testing%20fulfillment%20of%20graduation%20requirements

Center on Education and Teaching for Employment (CETE). (2016). A subject-matter expert’s guide. Creating quality multiple-choice items. The Ohio State University, Center on Education and Teaching for Employment.

Center on Education and Teaching for Employment (CETE). (2025). CETE Homepage. Retrieved May 23, 2025, from https://cete.osu.edu

Collaborative Institutional Training Initiative (CITI). (2025). The trusted standard in research, ethics, compliance, and safety training. Retrieved January 13, 2025, from https://about.citiprogram.org DOI: https://doi.org/10.5195/ijms.2025.2730

College of Food, Agricultural, and Environmental Sciences. (2024). Ohio State University Extension is…Retrieved January 12, 2025, from https://extension.osu.edu/about/mission-vision-values/osu-extension.

Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart, & Winston.

Dillman, D. A. (2007). Mail and Internet surveys: The tailored design method-2007 Update with new Internet, visual, and mixed-mode guide. John Wiley & Sons.

Farm Safety SAY Clearinghouse. (2025). Retrieved March 5, 2025, from https://extension.psu.edu/business-and-operations/farm-safety/say-project.

Finn, A. S., Kraft, M. A., West, M. R., Leonard, J. A., Bish, C.E., Martin RE, Sheridan, M. A., Gabrieli, C. F., Gabrieli, J. D. (2014). Cognitive skills, student achievement tests, and schools. Psychological Science, 25(3), 736-44. https://doi.org/ 10.1177/0956797613516008. DOI: https://doi.org/10.1177/0956797613516008

French, B. F.; Breidenbach, D. H., Field, W. E., Tormoehlen, R. (2007). The psychometric properties of the Agricultural Hazardous Occupations Order Certification Training Program written examinations. Journal of Agricultural Education, 48(4), 11-19. https://doi.org/10.5032/jae.2007.04011 DOI: https://doi.org/10.5032/jae.2007.04011

Garvey, P. M., Murphy, D. J., Yoder, A. M., & Hilton, J. W. (2008). National Safe Tractor and Machinery Operation Program: Development and content evaluation. Journal of Agricultural Safety and Health, 14, 333–349. https://doi.org/10.13031/2013.24567 DOI: https://doi.org/10.13031/2013.24567

Gearing Up for Safety Production. (2021). Retrieved March 5, 2025, from https://www.asec.purdue.edu/tractor/index.html

Golden Triangle Curriculum Consortium. Golden Triangle Curriculum Consortium. (n.d.). Retrieved May 22, 2025, from https://www.gtccmt.org

Haladyna, T. (2002). Supporting documentation: Assuring more valid test score interpretations and uses. Lawrence Erlbaum Associates.

Harshman, W. C., Yoder, A. M., Hilton, J. W., & Murphy, D. J. (Eds.). (Revised 3rd ed.). 2020. National Safe Tractor and Machinery Operation Program: Student manual. Penn State Extension, Ohio State Extension, and the National Safety Council.

Hazardous Occupations Safety Training for Agriculture Program (HOSTA). (n.d.). Retrieved Sept 22, 2005, from https://www.nifa.usda.gov/sites/default/files/resources/Hosta_brochure.pdf

Hoover, W. B., French B. F., Field, W. E., Tormoehlen R. L., (2012). Adjusting the passing scores for Gearing Up for Safety: Production agriculture safety training for youth curriculum test Instruments. Journal of Agricultural Education, 53(3), 1-13. https://doi.org/10.5032/jae.2012.03001 DOI: https://doi.org/10.5032/jae.2012.03001

Instructure. (2025). Canvas credential, the all-in-one digital badging for higher education. Retrieved May 23, 2025, from https://www.instructure.com/higher-education/products/canvas/canvas-credentials

Jennings, J. L., & Bearak, J. M. (2014). “Teaching to the test” in the NCLB era: How test predictability affects our understanding of student performance. Educational Researcher, 43(8), 381-389. https://doi.org/10.3102/0013189X14554449 DOI: https://doi.org/10.3102/0013189X14554449

Jensen, J.L., McDaniel, M. A., Kummer, T. A., Godoy, P. D. D. M., St Clair, B. (2020). Testing effect on high-level cognitive skills. CBE - Life Science Education, 19(3), 19:ar39.1-13. https://doi.org/10.1187/cbe.19-10-0193 DOI: https://doi.org/10.1187/cbe.19-10-0193

Kingman, D. M., Yoder, A. M., Hodge, N. S., Ortega, R., & Field, W. E. (2005). Utilizing expert panels in agricultural safety and health research. Journal of Agricultural Safety and Health, 11, 61-74. https://doi.org/10.13031/2013.17897 DOI: https://doi.org/10.13031/2013.17897

Mann, A. J. (2017). Identification of learning outcomes and development of assessment methods for agricultural safety and health content in secondary agricultural education classrooms [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1492733303282755

Mann, A.J., Field, W.E., Tormoehlen, R., & French, B.F. (2016). Development and validation of written exam items for the Agricultural Hazardous Occupations Orders (AgHOs) Certification Training Program. Journal of Agricultural Safety and Health, 22, 121-134. https://doi.org/10.13031/jash.22.11296 DOI: https://doi.org/10.13031/jash.22.11296

Mann, A. J., & Jepsen, S.D. (2017). Using expert panel data to guide youth agricultural safety and health training resources in the US. Safety, 3, 1(4). https://doi.org/10.3390/safety3010004 DOI: https://doi.org/10.3390/safety3010004

Mann, A. J., & Jepsen, S. D. (2019). Hazardous agricultural tasks completed by youth as part of their Supervised Agricultural Experience (SAE): A descriptive study. Journal of Agricultural Safety and Health, 25(3), 107–116. https://doi.org/10.13031/jash.12998 [elibrary.asabe.org] DOI: https://doi.org/10.13031/jash.12998

Mathew, S. N., Field, W. E., & French, B. F. (2011). Content validation using an expert panel: Assessment process for assistive technology adopted by farmers with disabilities. Journal of Agricultural Safety and Health, 17, 227–241. https://doi.org/10.13031/2013.38184 DOI: https://doi.org/10.13031/2013.38184

Meyer, M. A., & Booker, J. M. (2001). Eliciting and analyzing expert judgment: a practical guide. Society for Industrial and Applied Mathematics. https://doi.org/10.1137/1.9780898718485 DOI: https://doi.org/10.1137/1.9780898718485

Murphy, D.M. (2019). National Safe Tractor and Machinery Operation Program. Retrieved May 22, 2025, from https://www.academia.edu/?h=39958622

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). McGraw-Hill.

Ohio State University (OSU) Center on Education and Training for Employment (CETE). (2025). About. Retrieved April 16, 2025, from https://cete.osu.edu/about/

Ortega, R. R., Tormoehlen, R. L., Field, W. E., Balschweid, M., & Machtmes, K. L. (2003). Determining critical subject matter content for a safety certification program for youth employed in agricultural production. Journal of Agricultural Education, 44(4), 67–79. https://doi.org/10.5032/jae.2003.04067 DOI: https://doi.org/10.5032/jae.2003.04067

Safety in Agriculture for Youth (SAY) National Clearinghouse. (2025). Retrieved May 10, 2025, from https://ag-safety.extension.org/say-national-clearinghouse/

Pate, M. L., Lawver, R. G., & Sorensen, T. J. (2016). Safety in high school supervised agricultural experiences: Teachers’ training and students’ injury awareness. Journal of Agricultural Safety and Health, 22(1), 61–74. https://doi.org/10.13031/jash.22.11142 DOI: https://doi.org/10.13031/jash.22.11142

Purdue University. [n.d.] Gearing Up for Safety: Production Agriculture Safety Training. Retrieved May 22, 2025, from https://www.asec.purdue.edu/tractor/index.html

Snyder, S., French, B., Field, W., Tormoehlen, R., & Ess, D. (2013). Identification and validation of agricultural hazardous occupations order certification program instructor criteria and competencies. Journal of Agricultural Education, 54(1), 111–123. https://doi.org/10.5032/jae.2013.01111 [files.eric.ed.gov] DOI: https://doi.org/10.5032/jae.2013.01111

Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education. 48, 1273-1296. https://doi.org/10.1007/s11165-016-9602-2 DOI: https://doi.org/10.1007/s11165-016-9602-2

Tormoehlen, R., Field, W., Fox, R., Personett, C., Vollmer, W., & Ortega, R. (2003). Gearing Up for Safety: Production Agricultural Safety Training for Youth. Purdue University: West Lafayette, IN.

U.S. Department of Labor (DOL). (2024). Code of Federal Regulations, Title 29, Part 570, Subpart E-1—Occupations in agriculture particularly hazardous for the employment of children below the age of 16 years (29 CFR § 570 Subpart E-1). U.S. Government Publishing Office. Retrieved May 12, 2025, from https://www.ecfr.gov/current/title-29/part-570/subpart-E-1

U. S. Department of Labor (DOL). (2016a). Fact Sheet #40: Overview of Youth Employment (Child Labor) Provisions of the Fair Labor Standards Act (FLSA) for Agricultural Occupations, Revised December 2016, Retrieved May 12, 2025, from https://www.dol.gov/agencies/whd/fact-sheets/40-child-labor-farms

U.S. Department of Labor (DOL). (2016b). Child labor requirements in agricultural occupations under the Fair Labor Standards Act. Child Labor Bulletin 102. Publication No. WH-1295. Washington, D.C.: U.S. Department of Labor. Retrieved May 12, 2025, from https://www.dol.gov/sites/dolgov/files/WHD/legacy/files/childlabor102.pdf

Webb, N. (2002). Depth-of-Knowledge Levels for four content areas. Madison: Wisconsin Center for Educational Research. Retrieved June 1, 2025, from https://dpi.wi.gov/sites/default/files/imce/assessment/pdf/All%20content%20areas%20%20DOK%20levels.pdf

Webb, N. L., Alt, M., Ely, R., Vesperman, B. (2005, 24 July). Web alignment tool. Wisconsin Center of Educational Research. University of Wisconsin-Madison. February 2, 2006. Retrieved June 1, 2025, from https://www.scribd.com/document/280830775/Webb-Et-Al-2005-Web-Alignment-Tool-WAT-Training-Manual-2-1-Draft-091205

Wellberg, S. (2023) Teacher-made tests: why they matter and a framework for analysing mathematics exams, Assessment in Education: Principles, Policy & Practice, 30 (1), 53-75, https://doi.org/10.1080/0969594X.2023.2189565 DOI: https://doi.org/10.1080/0969594X.2023.2189565

Yoder, A. M. (n.d.). AgSafety4U 2.0: Agricultural safety and health online training program. Agricultural Safety and Health eXtension Community of Practice. Retrieved from https://ag-safety.extension.org/agsafety4u-certificate-course/