High-speed internet is a staple of everyday life, but many Americans don’t have access to it because no network reaches them—or they can’t afford to pay.
Sharing charts, maps, and more to show who Americans are, how policy affects the everyday, and how we can use data to make a difference. Nonpartisan forever.
Colorado Dives Into Evidence-Based Policymaking
Focused, multipronged investment yields benefits for the state
Since 2011, Colorado has implemented several data-driven initiatives that are delivering positive results for Coloradans and changing the way its government leaders think about evidence. One key project includes a partnership with the Pew-MacArthur Results First Initiative to expand the use of evidence in policy and budget decision-making. Managed by a team housed in the Office of State Planning and Budgeting, the Results First approach to evidence-based policymaking—along with its staff and tools—has enabled state leaders to promote a culture in their government that supports data-driven decision-making.
Through this approach, the team has helped Colorado to:
Build knowledge of what works and what doesn’t to improve outcomes for residents.
Promote investment in effective programming.
Improve the way evidence informs the budget process.
This brief provides an overview of Colorado’s efforts to build a culture of evidence-based policymaking, highlights the role of Results First—including successes and lessons learned—in doing this work, and details innovative ways the state continues to advance the use of evidence.
A culture of evidence-based policymaking
Colorado implemented the Lean performance improvement program, which trains the state’s workforce to apply its model, in 2011. This sequence of nine process improvement activities aims to continuously increase the efficiency of state programs, eliminate waste, and generate returns on government expenditures, such as costs saved or avoided, or time saved.1 Lean has increased the effectiveness of government service. For example, it has reduced by half the time it takes to process the state’s mental health hospital admissions as well as the Department of Education’s teacher licenses.2
Building on this success, Colorado passed the State Measurement for Accountable, Responsive, and Transparent (SMART) Government Act in 2013 to further advance efficiency and transparency across departments. The legislation requires departments to develop measurable goals—and plans that outline strategies to achieve them.3 To inform the public of its progress, the state launched an online dashboard that captures the key goals and outcome measures across five priority areas: health, economic and infrastructure development, environment and energy, workforce development and education, and quality government services.4 The dashboard enables policymakers and residents to see how state leaders are using taxpayer dollars to achieve these goals, providing fiscal transparency.
In 2014, state officials saw an opportunity to build on this growing portfolio of data-driven initiatives with the Results First approach, which uses rigorous research and cost-benefit analyses to help government leaders identify and invest in cost-effective policies and programs. Leaders of the executive and legislative branches committed to partnering with Results First and approved funding for two full-time staff members in the Office of State Planning and Budgeting (OSPB) to help manage the project.
"We have been able to integrate several initiatives that help us
improve service to constituents, make better decisions, and grow
employees’ skill sets. It has been all the more gratifying to have
continued and growing bipartisan support from the Colorado
General Assembly and effective partnerships across government
for all these evidence-based policymaking efforts."
Henry Sobanet, former directorColorado Governor’s Office of State Planning and Budgeting
While the Results First project was underway, OSPB also recognized the need to build capacity to engage in related elements of evidence-based policymaking. The office formed a research and evidence-based policy initiatives team to advance these efforts through several projects, including the state’s work with Results First, an implementation and evaluation grant program, a Pay for Success initiative to support effective youth programs, an evidence-informed budget request process, and other data-driven initiatives.
Collectively, these projects are promoting the use of data and research across state agencies and stakeholders. “Culture change has been one of our biggest successes. We’ve been able to go from mere discussions about data to promoting full assessments of programs to understand what works,” said Ann Renaud Avila, former director of research and evidence-based policy initiatives.
How Results First works in Colorado
The Results First approach and tools enable Colorado leaders and stakeholders to embrace evidence-based policymaking in several ways.
Build evidence of what works and what doesn’t to improve outcomes for residents
As in many states, Colorado policymakers had limited knowledge about taxpayer-funded programs, including information about their effectiveness. The program inventories and cost-benefit analyses that are produced with Results First are helping the state fill this critical information gap by providing a common method to compare programs based on the evidence supporting their effectiveness and predicted return on investment (ROI).
"There was a need to put structure around the problem of not
knowing what’s really going on, and Results First is one approach
to help us make progress based on what works."
Colorado Representative Bob Rankin (R)
With support from Results First staff, the research and evidence-based policy initiatives team inventories state programs in various social policy areas (for example, behavioral health, child welfare, criminal justice). These inventories collect information such as program description, location, cost, and number of clients served, and assesses it against the national evidence base. Next, using the Results First cost-benefit model, the team calculates the monetary returns that the state can potentially gain from investing in evidence-based programs, then publishes the results of these analyses, organized by policy area. To date, the team has produced reports for adult criminal justice; behavioral health; child welfare; juvenile justice; prevention programs; and health-related programs including chronic conditions, system efficiencies (such as reducing hospital readmissions and unnecessary emergency department visits), and child and maternal health.5
Across all policy areas examined, the reports show that Colorado funds many effective, evidence-based programs that are expected to generate positive returns. For example, the behavioral health analysis revealed that 79 percent of the interventions inventoried are effective, evidence-based, or promising interventions, meaning they will improve behavioral health outcomes for Coloradans if properly implemented.6 The remaining 21 percent need additional evaluation to determine efficacy, so the team is pursuing opportunities to evaluate these and other human services programs that lack information on effectiveness.
Most evidence-based programs that the team has analyzed demonstrated a positive ROI. For example, of the eight juvenile justice interventions studied, all but one are anticipated to achieve a positive ROI ranging from $1.20 to $11 per $1 invested due to reduced reoffending. In addition, all four of the child welfare programs analyzed are expected to produce returns as high as $6 per $1 invested from reduced child-neglect and maltreatment rates.7 The team presents cost-benefit findings in an apples-to-apples comparison format so policymakers can easily see which programs are likely to generate a return on taxpayer dollars and which probably won’t (see Table 1). Policymakers and state entities—including OSPB—have used this information to inform funding and programmatic decisions, as described below.
How Colorado Conducts Its Cost-Benefit Analyses
The behavioral health cost-benefit analysis that Colorado produced with the support of Results First provides an apples-to-apples comparison of similar programs within this policy area, although the programs listed may serve different populations, have different goals, and be delivered in different settings.
Program costs, taxpayer benefits, and nontaxpayer benefits are rounded to the nearest dollar. Taxpayer benefits represent the costs of government resources avoided, such as medical care and treatment, and lost taxes from wages. Nontaxpayer benefits represent the avoided costs to both society and individual program participants for medical care and treatment, increased insurance premiums, loss of employment or reduced wages, and crime victimization.
The analysis also assumes that programs are implemented in accordance with their original design, which is necessary to achieve intended outcomes and generate anticipated returns.
Don Gralls
Getty Images
These analyses also show that some interventions are likely to produce a negative ROI, generating a total benefit of less than $1 for every tax dollar invested. For these programs, the research and evidence-based policy initiatives team carefully considers the ROI alongside other important factors when presenting this information in its reports and to policymakers. These factors include:
How long a program has been operating. A newly implemented program might have higher upfront costs that could lower over time.
How the program compares with alternatives.
Whether it is the most effective and appropriate intervention for addressing a population’s needs.
Whether it has benefits that cannot be monetized, such as reducing homelessness. A relevant program’sROI could potentially be higher if these benefits were assigned a dollar value.
For instance, Table 1 shows that the Mobile Crisis Response intervention will probably generate only 26 cents for every tax dollar invested. The team’s cost-benefit report explains that this program had recently been implemented and that providers of this service are required to offset costs with other revenue sources, such as private insurance. This means that the costs captured for the analysis may later be paid for by nontaxpayer funds, which could increase the program’s ROI.8 The team regularly emphasizes such factors in its reports and communications to policymakers.
Promote investments in effective programming
Colorado leaders are using the Results First program inventories and cost-benefit analyses to inform decisionmaking. For instance, the state’s Office of Community Corrections (OCC) found through such an analysis and its own data on adult criminal justice programs that several of its interventions were not generating favorable outcomes or returns.
“The results did not come back in as favorable a way as we wanted, but this created opportunity and space to innovate and implement new cost-beneficial programs,” said Glenn Tapia, former OCC program director.
Based on the information and previous recommendations from the Colorado Commission on Criminal and Juvenile Justice, the OCC redirected funding toward a new pilot intervention—cognitive behavioral therapy— that is projected to generate a positive return. In addition, the Legislature granted new funding to implement an innovative program evaluation tool to assess adherence to evidence-based practices by community corrections programs.
OSPB used Results First analyses to inform the programs selected to receive funding through the state’s new Pay for Success initiative.9 After assessing the level of evidence supporting each proposal, OSPB requested approximately $6 million over four fiscal years (2018-22) to fund three programs. To support this request, the research and evidence-based policy initiatives team used the cost-benefit model to calculate the value of key outcomes of the selected programs—such as on-time graduation and avoided child abuse and neglect— to demonstrate the potential monetary impact of achieving these results.10 For example, to help legislators understand the value of Colorado youth graduating on time, the team calculated the lifetime projected benefits— costs avoided from graduating on time due to reduced crime and health care, and increased earnings—to both taxpayers and nontaxpayers, which is approximately $536,000 per youth.11 The Legislature ultimately approved funding.
"Both sides of the aisle have a goal for more efficient government.
The Results First approach aims to show what works, what
doesn’t, and how we can make the most of limited resources.
Embedding this approach over the long term will help our state
make the most of taxpayer dollars."
Colorado Representative Chris Hansen (D)
Improve the way evidence informs the budget process
To ensure that new investments continue to support efficient and effective services, OSPB has developed a process to regularly integrate data and research into the state’s budget. In 2016, OSPB began to include evidence requirements in its budget instructions that agencies must satisfy when they submit requests for funding to implement new or expand existing programs. Requests must include information on the program’s expected effects on outcomes and anticipated ROI—which the team will help identify—and, if the program is untested, a plan to evaluate it.12
Along with OSPB analysts, the team reviews requests and provides research and funding recommendations as needed. For instance, a Department of Human Services request to expand investments in a respite care program revealed limited research on the program’s effectiveness. The team recommended that the state fund the program on the condition that the department perform an outcome evaluation to assess effectiveness.13
The Pew Charitable Trusts
Bureau of Land Management
Lessons learned from applying the Results First approach
Colorado’s research and evidence-based policy initiatives team attributes the state’s successes in using Results First to several key lessons learned, including:
Early and ongoing leadership buy-in. Support from state leaders has been essential to driving the use of evidence to create effective change. The team noted that engaging leaders and stakeholders early and often helps generate and sustain buy-in. For example, legislative and executive leaders demonstrated their commitment to this effort when they approved funding for two full-time staff members to work on evidence-based policymaking initiatives. With support from Results First staff, the team continues to sustain the support of state leaders through legislative and agency employees training on the Results First approach.
Understanding that ROI tells only part of the story. The cost-benefit model is only one tool. Sometimes the research is limited and does not capture all of the outcomes a program is capable of achieving. Moreover, its value cannot be measured by just cost-effectiveness. Context and factors such as overall effectiveness (regardless of cost) and the availability of alternatives are important to consider when identifying programs that can best meet the needs of Coloradans. The team emphasizes these factors in its reports and shares them with stakeholders when conducting outreach and meetings.
More evaluation is needed to understand the effectiveness of Colorado’s programs. Not every program in Colorado can be assessed using the Results First model, nor do all programs meet the team’s “evidence-based” standard. The team’s goal is to meet programs where they are and help move them toward rigorous evaluation to build evidence of their effectiveness. In fiscal 2018, the Legislature appropriated $500,000 to OSPB to support program operations, including evaluation.14 OSPB asked departments to apply for funding through a letter of interest. Departments responded with nearly $2 million in requested assistance, indicating that programs need these supports.15
Also in fiscal 2018, the state received $4.5 million in philanthropic funding to establish the Colorado Evaluation and Action Lab, a public-private partnership housed at the University of Denver. The lab, the first of its kind in the state, will partner with interested state agencies to identify ways to improve the services that address the governor’s priorities16—recidivism reduction, affordable housing, youth prevention and early intervention— including conducting rigorous evaluations of programs to determine their effectiveness.17
Effective program implementation is key to achieving results. State leaders recognize that to obtain the best results from evidence-based programs, they must ensure that they are properly implemented and that relevant data are collected to evaluate their efficacy. One way the state has supported these efforts is by creating the Evidence-Based Practices Implementation for Capacity (EPIC) Resource Center, which partners with the state’s criminal justice agencies to create sustainable, structural implementation components to support their practices.18 Since its inception, EPIC has worked with more than 100 government entities across the state and has seen substantial gains in its development of effective implementation strategies.
OSPB is also promoting effective implementation by using some of the $500,000 appropriated to support programs for these activities and by updating the fiscal 2019-20 budget instructions to require agency requests for funding to start a new or expand an existing program to include an implementation plan.19
Using data and evidence as learning opportunities. The program inventories and cost-benefit analyses serve as resources to agency and legislative leaders who want to learn more about the programs the state funds and operates, and to inform future budget and program decisions. But they are not meant to dictate what should or should not be funded or to call out poor investments. The team is working to ensure that these resources are also shared and discussed with program administrators to help inform strategic planning within departments—such as through the state’s Performance Management Academy, which works with program leaders across state departments to promote the importance of evidence.
Furthermore, representatives from the Governor’s Office, state agencies, and evidence-focused nonprofit and private organizations formed the Evidence-Based Policymaking Collaborative in 2017 to create a space for dialogue on evidence-based policymaking, provide resources and trainings on this approach, and identify ways to address gaps and barriers to doing this work.
As the research and evidence-based policy initiative team has learned, multiple partners are important to this effort: state leaders who help champion and promote this method of policymaking; legislative and government officials who approve funding to support the work; agency staff and program providers who facilitate the implementation, monitoring, and evaluation of the state’s programs; and stakeholders who can lend support. The team continues to connect with these partners to learn from them and create a common language for this work.
Conclusion
While many states have begun to use evidence-based policymaking in a number of ways, Colorado stands out as a leader. The state has unequivocally embraced tools and systems such as the Results First approach to improve how government operates and to inform its budget and policy decisions. These efforts—which require investments of time, employees, and funding—are building a culture in which government staff, policymakers, and local stakeholders work together to expand the use of evidence and make data-driven decisions.
Pay for Success is a funding model in which a government entity enters into an agreement with a third party to fund programs designed to improve outcomes. Traditionally, the government entity agrees to reimburse the third party, plus pay an ROI, on the condition that an independent evaluation shows that outcomes are achieved.
Colorado Governor’s Office of State Planning and Budgeting, “Schedule 13: BA-01(GOV) OSPB Youth Pay for Success Initiative.”
Colorado Governor’s Office of State Planning and Budgeting, “Funding Request for FY 2018-19 Budget Cycle: R-11 Continuation of Respite Care Task Force Recommendations,” Oct. 20, 2017, https://drive.google.com/open?id=0B6eSaRI8GDVNQTAtRU1OOTRFbEU.