In December 2016, we released a final report highlighting the accomplishments of the Pathways Fund, a five-year fund with the goal of strengthening the bridge between education and workforce development in order to foster opportunity and access for low-income youth. In the coming weeks, we will provide an in-depth look into the report, the success stories of the Pathways Fund and the lessons we’ve learned.
Today we focus on increasing the evidence base of a cohort through rigorous evaluation studies.
As part of the Social Innovation Fund, the Pathways Fund included a mandate to engage in rigorous evaluation studies in line with the federal government’s commitment to promote evidence-based programming. This aspect of the fund was built into New Profit’s capacity building approach with the organizations and was integral to the learning community. SIF funding enabled New Profit to invest heavily in evaluation: Pathways grants covered 25% to over 90% of each organization’s study, and collectively funded almost $3.5 million of evaluation work in the college access/success field. Each of the Pathways organizations committed to undertaking rigorous evaluation studies to increase their evidence base and to build out their measurement capabilities.
Once complete, these evaluation studies will have:
- Increased level of evidence for Pathways organizations - Each organization will have increased their level of evidence through a third-party evaluation.
- Increased evaluation capacity for Pathways organizations - All organizations increased their internal monitoring and evaluation capacity; four of the six Pathways organizations created internal senior evaluation positions as a result of their participation in the Pathways Fund.
- Increased volume and quality of evidence in the college access/success field - Through the evaluation studies and supplemental research being conducted across the network, the Pathways Fund will add significantly to the volume of evidence regarding levers for impact in the college access/success field. Three of the largest programs in the field are represented in this cohort.
Through these evaluation studies, New Profit has learned that procuring an evaluator, designing the study, managing inevitable changes in timeline, scope, design or personnel, and other challenges mean evaluation studies need careful planning and active management to be successful:
Study timelines are long. For the Pathways organizations, they were almost certain to be longer than the five years allotted within the SIF grant, especially if the organization was not already primed to launch a rigorous study. Organizations that need to bolster their monitoring efforts or conduct formative research should plan for longer timelines, as do those that have long intervention or follow periods before effects can be determined.
Intermediaries and grantee organizations should engage in open and honest communication about the level of effort and expertise required to successfully manage a rigorous outcome study, and whether the organization is prepared to engage in this significant undertaking. Successful management of outcome studies not only requires upfront investment in procuring an evaluator and designing the study but ongoing management of the usual changes in the course of evaluations (timeline changes, design/scope revisions, recruiting efforts, personnel changes). If intermediaries select organizations with weak or modest monitoring and evaluation capacity, they should be prepared to balance that with strengths/capacity that they or other partners can provide to the grantees.
Raising funding for evaluations can pose a challenge for organizations. Without prior experience in evaluation, a common challenge that organizations face is inadequately budgeting and/or fundraising for evaluation. Organizations that overestimate their capacity to raise evaluation dollars can be left struggling to complete their contracted work.
Due to a push in recent years from funders, policymakers, and other stakeholders for the social sector to measure and show its impact, an increasing number of nonprofits are engaging in formal evaluation studies as a communications and fundraising tool to showcase program effects. While these are valuable, organizations struggle with how to also use their study results as a tool for internal learning and program improvement, versus a thumbs-up or -down assessment. Because there are often funding implications for organizations whose outcomes fall short of expectations, engaging in a rigorous evaluation can seem like a high-risk opportunity; organizations may be hesitant to conduct outcome studies unless they are already confident that their programs are producing positive outcomes. This cycle undermines the intent behind conducting evaluations, which is to further research and continuous learning. Funders have an important role to play in creating a safe environment that encourages organizations to engage in measurement and evaluation activities, and to partner with organizations to turn their evaluation findings into real program improvements.
Special thanks to the primary funders of the Pathways Fund:
Corporation for National and Community Service | Blue Ridge Foundation New York | Carnegie Corporation New York | The Kresge Foundation | Oak Foundation | Robin Hood Foundation | State Street Foundation | JPMorgan Chase Foundation | Open Society Foundation | SeaChange Capital Partners