Statistical Procedures for Determining the Validity and Reliability of a Multidimensional Survey Instrument: The School Success Profile

Gary Bowen
School of Social Work
The University of North Carolina at Chapel Hill
301 Pittsboro Street
Chapel Hill NC 27599-3550
USA
Phone: 919-962-6542
FAX: 919-962-3689
Email: glbowen@email.unc.edu
 
Roderick Rose
School of Social Work
The University of North Carolina at Chapel Hill
301 Pittsboro Street
Chapel Hill NC 27599-3550
USA
Phone: 919-962-8826
Email: rarose@email.unc.edu
 
Natasha Bowen
School of Social Work
The University of North Carolina at Chapel Hill
301 Pittsboro Street
Chapel Hill NC 27599-3550
USA
Phone: 919-843-2434
FAX: 919-843-8715
Email: nbowen@email.unc.edu
 
Objective: Valid and reliable assessment of students is a starting point for evidence-based practice in schools. This presentation reviews both the steps and the findings of a comprehensive psychometric evaluation of the School Success Profile (SSP). The SSP is a 220 question self-report survey that examines middle and high school students' beliefs about their neighborhood, schools, families, peer groups, and themselves. The SSP, which is widely used for student-level intervention planning by school systems across the United States, results in one-page summary profile for each student respondent that includes information on 22 dimensions related to the student's social environment and individual adaptation. Information about the validity and reliability of the SSP is critical for school professionals who depend on quality assessment data for informing individual and group interventions.

Methods: Data were collected between 2001 and 2003, and included SSP results for16,636 middle and high school students from nearly 400 schools in multiple states. Each scale dimension (14 social environment dimensions, 8 individual adaptation dimensions) was examined for normality, internal consistency, standard deviation of error, factor structure, measurement invariance (sex, race/ethnicity, grade level), construct validity, and either predictive validity (social environment dimensions) or clinical validity (individual adaptation dimensions).

Results: Results support the psychometric integrity of the SSP as an assessment tool across race, gender and grade level. The factor and construct validity analyses support the composition of the scales from the SSP items. Social environment dimensions predict individual adaptation dimensions, which subsequently demonstrate clinical validity across socioeconomic status.

Discussion: Psychometric data is lacking on many comprehensive youth surveys used in schools and communities. The analysis reported provides guidelines for evaluating multidimensional survey instruments, and demonstrates the quality of the SSP as a practice tool for schools and communities that want to improve adolescent outcomes.