A Common Language for Evidence-Based Programming

As state and local budgets continue to face constraints, policymakers are looking to maximize the return on every public dollar. One way government leaders are strategically targeting their investments is through evidence- based policymaking.

Over the past few years, as interest in using evidence to inform budget and policy decisions has steadily increased, several states have created formal definitions for levels of evidence that focus on the rigor and quantity of the research. In 2015, the Pew-MacArthur Results First Initiative reviewed these definitions, identified the best existing examples, and profiled them in a fact sheet. While such definitions provide a necessary framework for understanding the different types of evidence, they do not gauge the direction of a program’s impact.

Instead, Results First staff reviewed the rating criteria used by the clearinghouses in the Results First Clearinghouse Database. Based on those criteria, the staff created definitions that classify evidence-based programs into discrete categories, according to the program’s impact (positive impact, no impact, and negative impact). Since they are aligned with the Clearinghouse Database, the definitions may be particularly helpful to jurisdictions that use it as a resource to understand the effectiveness of evidence-based programs.

Definitions of evidence-based categories

“Evidence-based” programs, practices, and policies are those that have demonstrated measurable impact on a desired outcome, gauged by rigorous evaluations that incorporate valid and reliable comparison group designs. The evidence of impact can point in different directions:

• Evidence of positive impact. The program, practice, or policy has shown a positive impact on at least one outcome.

  • For example, a prison-based program demonstrated a decrease in recidivism, thus representing a positive impact for that outcome.

  • In the Clearinghouse Database, these programs are categorized as highest-rated and second-highest rated.

• Evidence of no impact. The program, practice, or policy has shown no impact on at least one outcome.

  • For example, a school-based program demonstrated neither a positive nor a negative impact on high school graduation.

  • In the Clearinghouse Database, these programs would be categorized as no evidence of effects.

• Evidence of negative impact. The program, practice, or policy has shown a negative impact on at least one outcome.

  • For example, a substance use prevention program demonstrated an increase in smoking, thus representing a negative impact for that outcome.

  • In the Clearinghouse Database, these programs would be categorized as negative effects.

These definitions can provide decision-makers and program providers with a shared understanding of terms, enabling a more robust conversation about evidence-based programming. In turn, this common language can help state and local government leaders identify and invest in programs that are most likely to benefit their constituents.

Pew-MacArthur Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation, works with states to implement an innovative benefit-cost analysis approach that helps them invest in policies and programs that are proved to work.