The Pew Fund for Health and Human Services launched the Evaluation Capacity Building Initiative (ECBI) in early 2018 to strengthen our grantees’ abilities to monitor and evaluate their programs and use data to improve services for some of Philadelphia’s most vulnerable residents.
For more than a year, Pew partnered with outside experts to help 30 of our grantees explore how to collect more useful data, encourage data-driven organizational cultures, and build stronger programs. With the first cohort complete, here are three examples of the changes already underway for some of the participating organizations.
Supportive Older Women’s Network seeks better data and to build a culture of learning
Supportive Older Women’s Network (SOWN) works to prevent and reduce isolation and depression in older adults. Before participating in the ECBI, SOWN struggled to use data on an ongoing basis and measure the impact of its services. The organization primarily relied on paper surveys conducted once a year. The resulting data proved useful for reporting to funders because it showed whether participants’ symptoms had improved over the year. But the information didn’t help staff better understand and meet the needs of their clients on a regular basis. For example, understanding the underlying issues related to their clients’ depression would have helped SOWN staff better identify ways to help, and to provide that help when it was most needed.
Through participation in the ECBI, the organization adopted a better tool to measure and understand depression and isolation among its elderly clients. This, in turn, helped case workers better support their clients’ needs and enabled the organization to measure and understand the impact of its services. SOWN staff also created a new position, an outcomes specialist, who is responsible for training staff on how to collect and use data, refining collection and analytics tools, and drawing insights from data.
The lessons learned about evaluation in only 15 months are already making an impact on the culture of the organization.
“The [data-informed] mindset is infused in every meeting. We constantly ask which data will help us understand our program’s successes and challenges, and how can our data be used to strengthen our messaging to clients, the community, and funders,’’ said Merle Drake, SOWN’s executive director. “We are not at the endpoint by any means, but a cultural shift is occurring. Instead of conversations about compliance, we are thinking about how to maximize impacts.”
With refined goals, Ed Snider Youth Hockey Foundation is strengthening programs to achieve them
The Ed Snider Youth Hockey Foundation uses the sport to educate and empower youth to succeed in school and life. One reason the organization participated in Pew’s ECBI was to learn to concisely describe what it wants program participants to accomplish and the key strategies needed to achieve desired outcomes. Snider Hockey created its own version of what is called a “theory of change”—identifying who can benefit from the program, how to provide that benefit, and how to measure success.
Through this process, the nonprofit worked to clarify its purpose statement and identify needed improvements. Staff found new ways to capture long-term participant success in areas such as academic achievement, physical and mental health, and employment upon graduation. Moreover, the theory-of-change approach proved to be a valuable tool for determining whether new opportunities will advance their goals; if they won’t, the organization does not adopt them.
Engaging Snider Hockey’s key stakeholder groups throughout the ECBI process proved integral to its success. Members of the board provided insights and support as the organization worked to develop new goals. Staff members were important contributors too, offering valuable “front-line” perspectives. By incorporating the voices and feedback of different stakeholders, everyone across the organization made a greater commitment to collecting and using data.
“We are committed to using what we’ve learned and built to make a stronger organization,” said Jan Koziara, Snider Hockey’s executive vice president. “Our newfound curiosity, as well as the defined program and associated indicators and evaluation tools, have allowed us to learn from students and families in ways that we didn’t think possible … and to adapt our work to respond to their needs.”
For example, the organization has improved how it integrates feedback from program participants and their parents or guardians into its practices. Before the ECBI, staff didn’t formally seek valuable feedback from either.
Now, to understand why some youth attend a program on the first day and don’t return, staff members follow up with families with a text, asking whether participants enjoyed the program and whether they will be coming back. If the parent responds “no” to either question, staff follow up to get a better sense of why and to look for trends in responses that could help adjust programming. And to better measure program impact, the organization has implemented a “Think Beyond the Rink” annual survey to assess hockey skills and other core outcome areas such as academic achievement and life skills.
Lutheran Settlement House developed a learning culture and focused resources on monitoring and evaluation
Lutheran Settlement House is a multiservice agency that empowers individuals and families to achieve and maintain self-sufficiency through a range of social, educational, and advocacy supports. Its leaders are working to build an organizational culture dedicated to using data and learning from it. The nonprofit always had a commitment to evaluation but struggled to get staff excited about collecting data—which can often feel like an “add on” to already difficult jobs. Staff also wanted to find ways to better use that information for decision-making. Multiple data systems, competing funder requirements, and leadership transitions created additional obstacles.
Still, Lutheran leaders persevered to shift to a culture that values, learns from, and uses data.
What Does a Learning Culture Look Like?
How organizations can shift from compliance to curiosity
A learning culture is created when a nonprofit values reflection—examining data that show both successes and failures—and establishes regular practices to use data to make improvements. In contrast to a compliance culture, in which an organization collects or examines data only to respond to questions from external sources like government regulators or funders, a learning culture rewards curiosity, encouraging staff and leaders to always ask “why”: why something worked or why it didn’t, and what can be done to improve how the organization works in the future.
Source: The Urban Institute, https://www.urban.org/research/publication/strategies-cultivating-organizational-learning-culture
Since completing the ECBI curriculum, Lutheran Settlement House has dedicated new resources to monitoring and evaluation, including development of an agency-wide technology system that helps decision-makers evaluate impact beyond individual programs.
The organization established a committee comprising staff from program, fundraising, and executive leadership so that information from the new system is used broadly to make both program and operations decisions. Lutheran also plans to create user-friendly reports so that relevant data can be discussed in weekly staff meetings to better understand the needs of clients and connect them to the right services.
“For the first time, there’s an excitement in the air at Lutheran Settlement House (about data), an eagerness to learn more about our work, our clients, and our impact—and to use data to inform strategic changes,” said Erica Zaveloff, the organization’s director of development.
These are just a few examples of how a shared dedication to learning and improving can and will strengthen the individual and collective impact of our grantees and their programs on Philadelphia’s most vulnerable populations.
Kristin Romens is the project director of the Pew Fund for Health and Human Services in Philadelphia; Meridith Polin leads the fund’s Evaluation Capacity Building Initiative.