Boys and Girls Clubs of the Peninsula (BGCP) partners with families, schools, and school districts to provide students with expanded learning opportunities through after-school, summer, and school-day programming.
With a research grant from Fund for Shared Insight, BGCP contracted to work with Hamai Consulting on a multi-year study to explore the relationship between student feedback and school success outcomes and to examine the role that student engagement and program quality might have in that relationship. After the work started, Hamai Consulting decided, along with BGCP, to expand on the original study to explore if and how the experiences, feedback, and outcomes differ for students for whom English is a second language (referred to as English Language Learners, ELL) and students for whom English is their first language (non-ELL).
Overall, our results show that student feedback is a valuable tool for direct-service organizations aiming to improve a program’s impact. The research also demonstrates that feedback predicts student outcomes, but not always in the ways we expect. Feedback, along with program attendance and the assessment of program quality, each contribute to student outcomes in unique and sometimes unanticipated ways.
How does BGCP gather feedback?
BGCP regularly collects student feedback data using print and online surveys distributed to all 3rd- through 12th–grade students twice a year, once in the fall and once in the spring. The surveys include 25 questions about perceptions of overall satisfaction, safety, belonging, engagement, and positive adult relationships.
How does BGCP assess program quality?
Once a year at each program site, BGCP uses the David P. Weikart Center’s Youth Program Quality Assessment (PQA) model to collect program observations and scoring that culminates in a plan to address areas for improvement. The PQA assesses four domains that constitute high-quality youth development programs:
- Safe, supportive environment: Students are physically and emotionally safe.
- Interaction: Students get to know each other and build relationships with caring adults.
- Engagement: Students enjoy activities and build their habits, mindsets, and skills.
- Effectiveness: Students experience a well-managed environment, adequate preparation for learning experiences that make real-life connections, and high-quality activities.
What did our research find?
Student feedback does not always positively predict student outcomes.
More positive student feedback predicts:
- lower academic performance for elementary school students
- higher academic performance for middle and high school students
More positive student feedback predicts lower:
- school attendance rate for elementary school students
- social and emotional skills for middle school students
- school attendance rates and social and emotional skills for high school students
The research also found that student feedback indirectly contributes to student outcomes through its relationship with program attendance. More positive student feedback predicts higher BGCP program attendance, which then predicts academic performance and school attendance for elementary and middle school students.
We also found that program quality contributes to student outcomes in different ways, above and beyond the contribution of student feedback. For elementary and middle school students, the assessment of quality indirectly predicts outcomes – specifically, quality predicts BGCP program attendance and program attendance predicts academic performance and school attendance. For high school students, program quality directly predicts school attendance.
Generally, the results were the same for students whether they identified as ELLs or non-ELLs. With an additional Shared Insight grant to examine racial equity issues within the original feedback research, we undertook a case study that further explored the experiences of BGCP English Language Learners. Many ELL students spoke about how mentors and other staff members encouraged them to develop a growth mindset within an environment that made it safe for them to learn and try new things, such as new activities and meeting new people.
What are some lessons from the research and next steps?
Now, with a greater understanding of the relationship between student feedback and program attendance and a closer look at the mixed and unexpected relationships between student feedback and student outcomes, BGCP has been inspired to revisit how it measures its programs and collects feedback:
- BGCP plans to continue to collect data longitudinally for students participating in the program to later revisit program impact. Because of the constraints of collecting data from the school district and Covid, the data set for this study included only two years of data. BGCP would not necessarily expect to see academic impacts from the program after one year of participation.
- Accounting for whether students choose to participate in the program may impact study outcomes. High school students choose to go off-campus to the BGCP Club to participate in the program, while elementary and middle school students stay at their school and likely have less choice than high school students about whether they stay to participate or go home after school. The active choice that high school students make to participate may explain why their feedback, assessment of program quality, and program attendance more strongly predict their outcomes.
- Measuring and accounting for school and home experiences may help better clarify and reveal program impacts. This may be particularly relevant for middle school students who are selected to participate in BGCP because they are facing challenges at school.
- Because of participation in this study, BGCP reevaluated its survey questions. After a review, the organization determined that some questions were not appropriate for elementary school students and it adjusted the survey for its youngest students.
- BGCP has started to use non-academic measures to gauge success of its programs, which support students academically but focus on other supports and goals. BGCP provides students with a safe, supportive, and stable environment where students can develop a growth mindset, confidence, and social skills. To measure these non-academic factors, BGCP now asks questions about socio-emotional skills, the growth mindset, and student engagement on its student surveys and the Holistic Student Assessment (HSA).
Overall, this research study has shown that feedback is a valuable tool to measure program outcomes and has assisted BGCP in improving its methods for measuring program impact.
About the author: