As our learning and evaluation partner ORS Impact releases its latest look at the value of perceptual feedback data through the experiences of Fund for Shared Insight’s six feedback research grantees, we took the opportunity to revisit those organizations.
While the grantee teams designed research projects with different methodologies given their varied programs, and despite setbacks that came with the COVID-19 pandemic, taken together, the research projects delivered distinct evidence that establishes gathering high-quality perceptual feedback as a best practice.
Here, we review this body of work and provide updates, highlighting the ways in which feedback as an ideal and a practice has become increasingly embedded into the work, values, and goals of these six organizations.
A memo from Shared Insight’s learning and evaluation partner, ORS Impact.
Boys & Girls Club of the Peninsula (BGCP)
Boys & Girls Club of the Peninsula (BGCP) is an after school provider of social, emotional, and academic learning. Partnering with Hamai Consulting, the BGCP feedback research team explored the relationship between student perceptions of BGCP’s programs and school success, and also examined how the experiences of English Language Learners (ELL) may differ from their native English-speaking peers.
Results of the research show that student perceptions as measured on student surveys are predictive of academic performance, even if not always in expected ways. It also found indirect impacts of student feedback, such as how it predicts program attendance.
Where are they now?
The feedback research grant provided BGCP the opportunity to gather feedback from ELLs and underscored the importance of nurturing partnerships between the staff and the students, leading especially to adults supporting youth in finding their voice. For additional learning, BGCP is diving into more research, working with Qualtrics and Education Northwest to update the student surveys to include questions related to new programs and initiatives, and to conduct a longitudinal study on the impacts of student perception on school success. The original surveys were administered during Covid-19 lockdowns, leading to a shorter data collection window and smaller than expected sample sizes. Now the largest after school provider in their county, BGCP expects their new research will gather more data, advance new questions and findings, and help them better understand the relationship between feedback and student outcomes across their program sites.
Center for Employment Opportunities (CEO)
Center for Employment Opportunities (CEO) offers employment support services to individuals returning from incarceration. Partnering with MDRC, CEO’s research team examined the feedback program CEO had in place since 2015, called Constituent Voice (CV), looking at how participants responded, how useful the information was, which participant characteristics were associated with response rates, and how responses were associated with outcomes.
The research found that client’s responsiveness to text messages from the CV program was associated with positive client outcomes, such as increased likelihood of gaining permanent employment. Overall, CEO found that feedback data are especially valuable for identifying clients who could benefit from more tailored, specific types of supports.
Where are they now?
In response to what was learned from the feedback research effort, CEO made some immediate changes, including providing job coaches with smartphones so clients can contact them directly; lining up orientation class schedules with clients’ availability; and improving descriptions of jobs so clients are better prepared for work.
CEO is working to address constructive client feedback around the desire for more work, higher pay, additional support for basic needs such as food and housing costs, and more program options, such as advanced training opportunities. They are also considering ways to redesign intake and assessment processes to gather more comprehensive information about clients’ strengths, establishing a baseline understanding of which aspects of the program facilitate internal growth, contribute to mindset shifts, and increase feelings of self-efficacy. By collecting that additional client information, including about their goals and expectations, as they onboard, CEO aims to create a more tailored and effective experience for individuals as they move through the program, thereby improving their outcomes.
Nurse-Family Partnership (NFP)
Nurse-Family Partnership (NFP) is a community-health program that connects first-time moms with a nurse. In two phases of research over three years, NFP’s National Service Office (NSO) gathered non-anonymous client feedback to assess the relationship between client satisfaction and program retention and outcomes. The NSO also examined whether clients’ satisfaction with the NFP program is impacted by their reported experiences around social isolation and discrimination and by whether they are matched with nurses with the same racial or ethnic identity.
Key findings, such as that clients who shared a similar race or ethnicity with their nurse reported greater satisfaction with the program and, in some cases, longer retention, led to changes at NFP, including recruiting and hiring more nurses of color. Improvements to NFP’s programming included adding more in-person nurse visits, classes, and opportunities for nurses and clients to interact and for moms and families to connect with each other through social media and at social events.
Where are they now?
Through the feedback research opportunity, the NSO expanded and standardized the infrastructure and systems for collecting client feedback across its network and many sites, allowing it to more immediately and consistently identify areas of improvement and respond to feedback. The study also evolved NSO’s focus on equity, providing a vehicle to better understand clients’ experience with discrimination outside of NFP. Feedback will continue to feature prominently in the NSO’s work, as they plan for widening the survey base to include local partners and families so that their feedback can lead to a broader response around community needs.
Pace Center for Girls (Pace)
Pace Center for Girls (Pace) is on a mission “to find the great in every girl” by providing year-round academic services, counseling, life-skills training, career preparation, and other supports to girls and young women in middle school and high school. In collaboration with Covian Consulting and Arc of Evidence, Pace tested the hypothesis that positive feedback from participants in its programs is linked to program engagement and positive outcomes..
The research findings suggest that the most engaged girls tend to provide feedback and this activity helps further improve their program engagement and outcomes. The findings also show that the provision of multiple venues for feedback (e.g., surveys, focus groups, informal conversations) supports individualized services that contribute to positive outcomes.
Where are they now?
Pace leaders say feedback has become a foundation of the group’s learning agenda, contributing to quality improvement efforts, innovation, and social impact. Pace’s participant feedback helps link the day-to-day work of team members with its mission by bringing participants’ voices to the table when discussing issues affecting program implementation. Pace girls continue to participate in program evaluation activities by vetting questions included in surveys and focus groups, supporting data collection activities, and discussing findings with team members as well as community partners. Pace is also committed to sharing the value of feedback with others and attended the Building Evidence to Advance Equity convening sponsored by the Annie E. Casey Foundation. The book, Next Generation Evidence, published by Brookings Institution Press, highlighted Pace’s work using feedback to support participatory approaches.
YouthTruth
YouthTruth is a national survey project that examines both student and stakeholder feedback to guide educational and philanthropic decision making. Conducted in partnership with SRI Education, YouthTruth’s research examined the extent to which various student experiences measured by the organization’s student perception surveys predict academic and behavioral outcomes at the school level.
Overall, the study found that each of the six YouthTruth student experience scales (academic challenge, culture, engagement, relationships, belonging and peer collaboration, and college and career readiness) demonstrated evidence of concurrent and predictive validity with at least one outcome for one or more school levels. Scales for academic challenge, culture, and belonging and peer collaboration were associated with key academic and behavioral outcomes across all school levels (elementary, middle, and high school).
Where are they now?
YouthTruth has continued to leverage the feedback research in four primary ways:
- In conversations, presentations, and proposals to school systems or other prospective partners or funders: YouthTruth describes the research base, highlighting the concurrent and predictive validity study and summarizing the findings.
- At state and local conferences: YouthTruth presents the feedback research to lend additional credibility to the strength of its survey instruments and the relevance of this type of perception data to the student outcomes audiences care about.
- As part of follow-up advisory supports for schools: YouthTruth uses the research to help partners more deeply understand their student feedback and position them to act on the data.
- In research citations: In addition to its work with specific schools and districts, YouthTruth uses its national dataset — citing its own research — to examine aspects of the student experience, such as mental health. YouthTruth’s research is also regularly cited by other feedback research that uses YouthTruth data.
REDF
REDF invests in and incubates social enterprises that provide jobs, training, and support to people breaking through barriers to employment. Partnered with Research Triangle Institute (RTI), REDF conducted a perceptual feedback study to examine the relationship between workers’ perceptions of their employment social enterprise (ESE) and later employment outcomes.
Key findings include that employee satisfaction with the program and the staff predicts stronger positive outcomes, such as earning higher wages and stable employment after leaving the program.
Where are they now?
As an intermediary organization that provides technical assistance to organizations, one opportunity that REDF saw to continue to promote the practice of high-quality feedback was to provide support to a cohort of grantees to participate in the capacity-building program Listen4Good. And as REDF continues to consider how to incorporate listening best practices into their own offerings, its manager of learning and evidence, Yon Jimenez-Marcuso, has kept actively engaged in the feedback research process as a Feedback Labs’ Feedback Champion.
About the author:
