ESTABLISHING A VALID AND RELIABLE COMPETENCY ASSESSMENT FOR GYMNASTICS COACH CERTIFICATION

Abstract

Coaches in Canada are trained through the National Coaching Certification Program (NCCP) administered by The Coaching Association of Canada. The NCCP is the national standard and recently shifted its educational emphasis from the transfer of knowledge to the development of coaching competency. As a result, coaches are required to demonstrate competency in a specified sport and level prior to being awarded a coaching certification. The purpose of this study was to establish validity and reliability of an NCCP coaching competency assessment. Gymnastics Canada agreed to have their Community Sport coaching competency instrument tested for content validity, face validity, inter-rater reliability and intra-rater reliability. Their original assessment instrument was revised during content validity testing based on recommendations from five coaching context experts. Removing six items, adding one item, and revising four items for clarity modified the original 48-item instrument. An assessment of overall competency was added to the instrument to rate the coach as Beginner, Competent, Proficient, or Expert. Three certified coaches confirmed the face validity of the modified instrument. Reliability tests were conducted on the ratings provided by ten experts who observed a coach’s lesson on video. The result was a moderate level of inter-rater reliability, displayed by an Agreement Coefficient (AC1) of 0.43 and a Percent of Agreement (PA) of 72%. Nine of the ten raters assessed the coach’s performance as Competent, Proficient or Expert, while one of the ten rated the coach as Beginner. After a repeat observation of the same coaching performance, the intra-rater reliability of five raters resulted in agreement levels of Moderate (AC1 = 0.45; PA = 67%), Substantial (AC1 = 0.75, 0.77; PA = 81%, 89%), and Almost Perfect (AC1 = 0.87, 0.82; PA = 93%, 90%). The modified instrument has content and face validity. However, its usefulness is impacted by variability in rater preferences. The inter-rater reliability results attest to concerns about the consistency in assessment of a coaching certificate for Community Sport Artistic Gymnastics coaches. Suggestions to improve the reliability of the instrument include training raters to classify the coach’s competency on a standardized scale of sport-specific expertise.