A Guide to Evidence-Based Budget Development

Editor's note: This brief is one in a series about the five key components of evidence-based policymaking as identified in “ Evidence-Based Policymaking: A Guide for Effective Government,” a 2014 report by the Pew-MacArthur Results First Initiative. The other components are program assessment, implementation oversight, outcome monitoring, and targeted evaluation.

Overview

For many years, policymakers and citizens have expressed concerns about the effectiveness of public programs. While efforts such as performance-based and zero-based budgeting have been helpful, these initiatives have often had only a limited impact on the way funding decisions have been made. Nonetheless, the importance of using taxpayer resources wisely has never been greater, particularly given the ongoing budget stresses facing many state and local governments.

The past decade has seen a dramatic increase in the availability of research that has rigorously tested the effectiveness of many social programs. Despite increased data on “what works,” many governments continue to allocate their resources based on anecdotes and inertia, funding the same programs year after year with limited knowledge of the outcomes they achieve. A better method, which some governments are beginning to implement, incorporates this information into an evidence-based approach to budgeting.

This issue brief, part of a series on evidence-based policymaking, highlights three key steps that governments can take to more effectively use data to inform the critical budget decisions they make each year:

1. Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.
2. Require agencies to justify requests for new funding with rigorous research on program effectiveness.
3. Embed evidence requirements into agency contracts and grants to ensure that research guides program activities.

The challenges to creating a better budget process

The traditional budget process often frustrates policymakers. It typically focuses attention on incremental changes to existing spending, such as increasing or decreasing funding for all agencies by 2 percent, while the impact of the programs administered by these agencies receives little scrutiny. Policymakers interested in using evidence to inform their budget decisions face two main challenges.

Existing budget processes reinforce the status quo.

Most state and local governments use an incremental budgeting process, meaning that the activities funded in next year’s budget will be very similar to the current year’s, with a few exceptions for new initiatives or priorities. Budget rules and processes tend to reinforce this dynamic by requiring that agencies justify the amount of funding requested, rather than the results their program can be expected to achieve. Few governments ask agencies to support funding requests by providing evidence of program effectiveness. When they do, it is typically for new programs or additional funding only, so a large majority of programs are not subject to these requirements.

Governments lack key information about the programs they fund.

Most governments do not gather important facts about the programs that are included in the continuation base budget. For example, most lack a comprehensive list of these activities, and few compile comprehensive data on their cost and performance. In the absence of information about the outcomes delivered by individual programs, it is understandable that policymakers resort to across-the-board cuts when required to make budget reductions, as the dearth of data hinders their ability to target cuts to underperforming programs.

Three steps to creating an evidence-based budget

Policymakers in both the executive and legislative branches can make more informed decisions by incorporating rigorous evidence into the budget process. Key steps include developing a complete inventory of currently funded programs, requiring agencies to justify budget requests with rigorous research, and mandating that agencies incorporate evidence requirements in the contracts and grants they administer. Collectively, these actions can help ensure that funding is directed toward programs that are most likely to achieve desired outcomes.

Step 1: Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.

The first step in developing an evidence-based budgeting process is to create a comprehensive list of funded programs and assess the level of evidence that exists about the effectiveness of each intervention. This enables policymakers to understand what programs are currently operating in the jurisdiction and what level of confidence they can have that investments in these interventions are likely to produce desired results.

Inventory current programs. Budget documents typically provide limited, if any, information about the individual programs that are being delivered by agencies and providers. An inventory enables policymakers to look inside the base budget and examine the specific interventions that are being funded. Governments typically assign responsibility for this inventory process to a central unit that can oversee agency data collection efforts. If possible, inventories should include a description of each program and its goals, services being provided, target population and number of clients served, and per-client and total cost. The inventory creates a baseline of information that can then be used to match programs to the evidence base.

Assess the likelihood that programs are successful. In a perfect world, governments would conduct periodic evaluations of all programs that they operate to provide comprehensive information about the effectiveness of each activity. However, such evaluations are expensive and time-consuming. Governments typically can conduct only a few such studies each year. Fortunately, the effectiveness of many interventions has been rigorously studied across the country in recent years. This information about what works can be used to assess whether local programs that are based on the same design are likely to generate positive outcomes. For example, a program that has been evaluated in multiple jurisdictions and consistently found to be successful is highly likely to produce good results if implemented locally with fidelity and under similar conditions.

Staff can begin this process by examining national research clearinghouses, which review and aggregate rigorous studies and rate programs by their level of evidence of effectiveness. This information can be found through the clearinghouses’ online portals (see “Using Research Clearinghouses to Match Local Programs to the Evidence Base”). Staff can then compare the programs their jurisdiction is operating (contained in the inventory) to those rated by the clearinghouses—considering factors such as target population and delivery setting—to determine if their local programs are evidence-based. Additionally, some governments rank programs by definitions of evidence that they have established through rule or law,¹ which can be helpful when classifying programs that are rated differently by clearinghouses.

Once completed, this evidence assessment enables leaders to categorize each program from their inventory by the level of available evidence about its effectiveness, identifying which ones are highly likely to work, which have been shown to be ineffective, and which need more research. This information can help governments as they analyze current spending and prioritize requests for new funding.

For example, in 2012, the Wyoming Department of Health contracted with the Wyoming Survey & Analysis Center (WYSAC) at the University of Wyoming to create an inventory of substance abuse prevention strategies focusing on changing aspects of the environment that contribute to the use of alcohol, tobacco, and other drugs. The inventory ranks each strategy according to the evidence supporting its effectiveness and highlights whether the strategy was implemented in the state.³ The report’s purpose was to provide information to community prevention professionals charged with implementing programs aimed at reducing alcohol, tobacco, and other drug use and abuse within their communities. “The county-level prevention workers are highly skilled but don’t have the time or resources to wade through all of the research on what’s effective and what isn’t,” said Humphrey Costello, an assistant research scientist at WYSAC and one of the authors of the report. “The goal with this report was to provide information on the evidence of program effectiveness in a straightforward way.”

The Wyoming program inventory has become a valuable tool for community coalitions engaged in making decisions about which strategies to implement to address critical needs in their communities. “The program catalogue gives us a framework for how decisions can be made,” says Brittany Ritter, a community prevention professional from Lincoln County and a member of the Prevention Management Organization of Wyoming. “We want to know that we are implementing the strategies that the research shows to work, and [the program inventory document] helps us do that.”

The Mississippi and Washington legislatures have passed laws requiring agencies to inventory their current programs and categorize each by defined standards of evidence. For example, in 2014, Mississippi mandated that the state’s corrections, health, education, and transportation agencies report program inventories, which staff then categorized into four groupings—evidence-based, research-based, promising practices, and lacking proof of effectiveness.⁴ Legislators plan to use these results to inform future budget decisions and reinvigorate the state’s performance budget system.

This approach has also been used effectively by local governments. Santa Barbara County in California surveyed its jail- and community-based service providers to develop a comprehensive list of the criminal justice treatment programs being delivered and to determine the extent to which evidence-based practices were being used. This analysis revealed that many of the interventions were not closely aligned with evidence-based practices, prompting the county to offer training and technical assistance to providers to help them improve their current activities or replace them with alternative evidence-based programs.⁵

Analyze whether programs’ benefits outweigh their costs. Cost-benefit analysis, which compares a program’s expense to the value of its expected benefits, can enable policymakers to assess whether programs achieve returns on investments that justify continued funding. While some cost-benefit analyses examine a single program, a growing number of governments are using the approach to compare the relative return on investment of portfolios of alternative interventions and target funding to those that are predicted to achieve the best outcomes. Several states, including Colorado, Iowa, Massachusetts, New Mexico, New York, Rhode Island, and Washington, have adopted this practice to analyze their investments in human services and rank programs by their predicted return on investment.⁶

The county-level prevention workers are highly skilled but don’t have the time or resources to wade through all of the research on what’s effective and what isn’t. The goal with this [program inventory] report was to provide information on the evidence of program effectiveness in a straightforward way.Humphrey Costello, assistant research scientist, Wyoming Survey & Analysis Center

For example, in 2014, Massachusetts completed an analysis of the predicted returns on investment and reductions in recidivism achieved by its corrections, probation, and parole programs. After examining this report, policymakers approved a competitive grant program to expand evidence- and research-based programming to reduce recidivism.⁷ In 2015, Colorado’s Office of State Planning and Budgeting released a report analyzing the benefits and costs of programs in adult criminal justice, juvenile justice, and child welfare. It plans to use these results to inform funding and policy decisions in the upcoming budget cycle.⁸

Use outcome data to determine whether programs are achieving expected results. Even the best evidence-based programs, if implemented poorly by agencies, will fail to deliver good results. Accordingly, governments should establish performance goals that measure the outcomes of all major programs and report this information during the budget process. For those that have been rigorously tested, outcome measures can gauge whether programs are achieving the results predicted by the research. For programs that lack rigorous research on their effectiveness, outcome data are still important for identifying trends, such as whether performance is getting better or worse. These trends can lead to questions during budget deliberations about whether alternative approaches should be considered.

Governments should make certain that measures are agreed upon by members of both the legislative and executive branches and that data are verified or audited to ensure accuracy and reliability. Maryland, for example, provides detailed requirements to agencies for developing and reporting outcome measures as part of the governor’s budget request, including developing controls and procedures to verify that data are accurate and reliable.⁹ New Mexico requires both the legislative and executive branches to agree on which outcome measures will be reported, as well as any changes to the measures, and has protocols for ensuring that data are verified for accuracy.¹⁰

State and local governments should also set performance targets as a way of determining whether programs are exceeding or falling behind in meeting their goals. Where feasible, research on expected results when a program is properly implemented should inform these targets. For example, an evidence-based home visiting program for new mothers—predicted to reduce the frequency by which children must be removed from the home by 10 percent—should be compared against baseline rates to determine its success rate. When research is not available, governments may utilize other benchmarks that allow for meaningful comparison, such as the performance in similar states or jurisdictions, or national standards. Virginia, for example, compares the performance of many state programs with the national average, the top-performing state in the nation, and three states similar to itself.¹¹

Step 2: Require agencies to justify requests for new funding with rigorous research on program effectiveness.

In addition to creating an evidence baseline on existing programs, policymakers can require requests for new or additional funding to be justified with rigorous evidence. Governments can also compel agencies to give priority to evidence-based programs when implementing their budget allocations.

Require agencies to incorporate evidence into budget requests. State and local governments can instruct agencies to justify requests for new or increased funding with rigorous evidence showing that the additional resources would produce positive results. Governments may choose to gradually phase in these requirements over time so that they apply to currently funded programs as well as to new programs. Two states are becoming exemplars in this approach.

In 2015, the Mississippi Legislature began requiring agencies seeking funding for new programs through the appropriations process to answer a series of questions to ensure that proposed initiatives adhered to key elements of high-quality design. For example, agencies must report whether a requested program has “an evidence-base, research base, promising practice or best practice type model”; must describe the monitoring system that will ensure implementation fidelity; and must explain how they will measure results.¹² During the fiscal year 2016 Regular Session, the newly created Performance Based Budgeting Committee of the House of Representatives expanded these requirements to include some existing programs operated by certain agencies. In some cases, vendors who contract with the state are also subject to this review.

We turned down a lot of requests this year because agencies didn’t have the data to support their requests.Kristen Cox, executive director, Governor’s Office of Management and Budget in Utah

As part of the annual appropriations process, state agencies in Mississippi submit their requests for funding to the Legislative Budget Office. During the 2016 Regular Session, staff from the Joint Committee on Performance Evaluation and Expenditure Review (PEER) reviewed agency requests for new program funding using the elements of quality design to determine whether they met the evidence standards. Preference was given to evidence-based programs that have been proved effective in addressing a particular problem.􀀀However, given the limited number of submissions for new funding that met the evidence standard, the Legislature provided limited funding for some non-evidence-based programs with the caveat that rigorous research on their effectiveness would be conducted during fiscal 2017. To facilitate this, PEER is developing standards for what constitutes a rigorous research evaluation.

“Developing skill for evidence evaluation—among state agencies and any other applicants for public funds—is an important next step in the process,” said Kirby Arinder, a PEER staff member. “As those parties become better at reviewing evidence in a disciplined fashion, they can help filter out all but the most evidentially supported, effective uses of public dollars.”

Utah has taken a similar approach by embedding evidence requirements into its state budget instructions. As part of Governor Gary Herbert’s effort to improve government performance by at least 25 percent, the Office of Management and Budget has required agencies to provide evidence of program effectiveness and performance. This may include recent performance data compared with industry standards or other relevant benchmarks, or more rigorous forms of evidence including program evaluations. Agencies must propose a business case for why they need new funding and describe whether the activity is an evidenced-based practice or otherwise supported by research, data, evaluation, or professional standards.¹³

“Agencies are coming from different starting points and have different experiences and levels of confidence in using data,” said Kristen Cox, executive director of the Governor’s Office of Management and Budget (GOMB) in Utah. “GOMB uses a consistent measurement approach that shows the relationships between cost, volume, and quality measures that provide more immediate feedback than some that take years to determine outcomes.”¹⁴

Gov. Herbert’s 2016-17 budget identified 35 programs that would receive one-time funding but would be required to provide additional evidence of effectiveness to receive continued support beyond 2017.¹⁵ “We turned down a lot of requests this year because agencies didn’t have the data to support their requests,” Cox said.¹⁶

Require agencies to prioritize funding to evidence-based programs. Policymakers can also use legislation or administrative policy to require agencies to give preference to evidence-based programs when implementing their funding allocations. These mandates are typically phased in over time, requiring agencies to increase the percentage of funding on evidence-based programs up to a certain threshold.

In 2003, Oregon passed S.B. 267, which required five state agencies to gradually increase the amount of funding allocated to evidence-based programs for drug and alcohol treatment, mental health treatment, adult recidivism prevention, and juvenile crime prevention. Every two years, the agencies were required to report the percentage of their total budgets allocated to evidence-based programs. Beginning in the 2005-07 biennium, the five agencies had to demonstrate that at least 25 percent of their funds were used to support evidence-based programs. That increased to 50 percent by the end of 2007-09, and 75 percent by the end of 2009-11. By 2012, all agencies reported that they had achieved these targets.²⁰

“I don’t think we would have gotten the engagement from the state agencies [to target funding to evidence-based programs] without the law,” said Paul Egbert, operations manager with the Oregon Criminal Justice Commission.²¹ “The legislation itself, judging on its own merits, drove practices that were evidence-based and that the state sorely needed.”

I don’t think we would have gotten engagement from the state agencies [to target funding to evidence-based programs] without the law. The legislation itself, judging on its own merits, drove practices that were evidence-based and that the state sorely needed.Paul Egbert, operations manager, Oregon Criminal Justice Commission

In Tennessee, the state passed law § 37-5-121 in 2007, mandating that by 2012-13 the Department of Children’s Services fund only evidence-based programs that have been proved to reduce juvenile delinquency.²² “There was a sincere desire to be good conservators of state dollars, providing the very best services that had research behind them,” said Debbie Miller, deputy commissioner for the Office of Juvenile Justice.²³ The state created definitions of evidence and partnered with the Peabody Research Institute at Vanderbilt University to evaluate the extent to which providers were implementing programs with fidelity to evidence-based principles.

Agency officials in Tennessee credit the law’s focus on program implementation—ensuring that programs are carried out according to research-based methods—for having a positive impact. “You can set up contracts with providers and say we want you to provide evidence-based services, but you must follow through and make sure those programs are being implemented with fidelity, and the law requires that,” said Elvie Newcomb, special projects manager with the Office of Juvenile Justice.²⁴

Finally, some states have created similar requirements through agency policies or directives, rather than through legislation. In New Mexico, the Corrections Department created an administrative policy in 2015 to prioritize funding toward evidence-based programs. The policy, which is intended to provide guidance and set expectations for the delivery of services across the state, requires the department to spend at least 70 percent of funds on programs with rigorous evidence of effectiveness.²⁵

Step 3: Embed evidence requirements into agency contracts and grants to ensure that research guides program activities.

Many decisions on how governments allocate funding to individual programs occur outside the formal budget process. Agencies often have significant flexibility in spending funds within their legislative budget allocation, and they frequently contract with private and nonprofit organizations to deliver services. State and local governments can use these grant and contracting processes to support evidence-based policymaking principles by requiring agencies to prioritize funding to programs that have strong proof supporting their effectiveness and to restrict funding of programs lacking that data.

Give preference in competitive grant programs to proposals that demonstrate evidence of effectiveness. State or local governments can refine contracting processes to incentivize providers to use research and evidence-based programs. Administrators can work with providers to educate them about these new requirements, and identify and resolve potential administrative barriers to using evidence-based programs. Administrators can also consider building in requirements for overseeing program implementation, outcome monitoring, and evaluation.

In 2014, New York’s Division of Criminal Justice Services (DCJS) awarded $5.1 million through a competitive request for proposal to support targeted, evidence-based alternatives to incarceration for offenders at a high risk of recidivism.²⁶ The grant program meant to generate long-term state and local budget savings by reducing prison and jail populations, reducing recidivism, and avoiding future victimizations. To be considered for funding, grantees were required to identify the specific evidence-based model that they would implement and agree to ongoing oversight and evaluation of outcomes. Since 2014, the competitive grant program has continued to grow, and the DCJS has also expanded provider requirements to increase program alignment with proven, evidence-based practices and has increased the agency’s capacity to perform research that gauges program effectiveness.

“The beauty of this approach is that it is very open and transparent,” said DCJS Executive Deputy Commissioner Michael C. Green. “Everything that we fund must meet a high standard, and providers receive the tools they need to meet that standard. There are no surprises.”²⁷

Build evidence requirements into provider guidelines. Agencies can also incorporate policies requiring evidence-based programs into the performance standards they issue for program providers. These standards are included in service contracts and typically address a wide array of requirements, including defining and describing acceptable levels of services. The provider guidelines therefore provide an opportunity to disseminate expectations for the delivery of evidence-based services. Several states, including Hawaii and New York, have embedded requirements to use evidence-based programs or practices in their provider guidelines.

In 2012, Hawaii’s Department of Health created a set of performance standards for child and adolescent mental health services, which prioritizes the use of evidence-based practices for organizations that contract with the department.²⁸ The standards define evidence-based practices according to four tiers that providers can use to inform the types of services they offer, though providers are urged to give higher priority to more reliable or stronger forms of evidence in making treatment decisions. Contracted providers are required to comply with the guidelines as well as any additional requirements identified in their contracts, unless granted a written waiver from the Child and Adolescent Mental Health Division (CAMHD). “[Over the years] we’ve modified the performance standards to reflect the need to use evidence-based practices,” said Stan Michels, CAMHD administrator. “The standards help provide consistency in the quality of services provided across each island.”²⁹

In New York, the Office of Alcoholism and Substance Abuse Services (OASAS) has developed provider guidelines for organizations that contract with the state to supply prevention services, using the standards as a baseline to monitor program performance and contract compliance.³⁰ A key component of OASAS’ Prevention Framework is the utilization of evidence-based programs and strategies. The guidelines reinforce this by requiring providers to spend an increasing percentage of their grant funding on these proven strategies. (The minimum was 35 percent by 2011 and increases by 5 percent each year to 70 percent by 2018.) “The standard allows for a percentage of non-evidence-based programs as well, which gives agencies the flexibility to be creative in developing services that are responsive to specific community needs,” said Mary Ann DiChristopher, acting associate commissioner for OASAS.³¹

OASAS has also created a registry of evidence-based programs and strategies that is available to providers to offer guidance for selecting the appropriate option. An advisory panel of prevention scientists regularly reviews research on effective practices, including those submitted by providers to identify new programs or strategies that meet the criteria of the registry as evidence-based. “The science has demonstrated that the use of evidence-based programs and strategies has led to positive outcomes for youth and families … and New York wanted to be a leader in this area,” OASAS Commissioner Arlene González-Sánchez said.³²

Conclusion

Policymakers are increasingly interested in using data to guide important decisions on how to allocate scarce taxpayer dollars to achieve better outcomes. While previous efforts to reform the budget process were met with only limited success, the growing availability of rigorous evidence about what works holds the promise of enabling governments to design more effective budget processes and make more informed investment choices. To do this successfully, governments will need to develop a comprehensive baseline of information on the effectiveness of currently funded programs, re-engineer the budget development process to encourage agencies to use rigorous evidence, and provide guidance to agencies on how to incorporate evidence requirements into the contracts and grants they administer.

This brief is one in a series about the five key components of evidence-based policymaking, as identified in Evidence Based Policymaking: A Guide for Effective Government. The other components are program assessment, implementation oversight, outcome monitoring, and targeted evaluation.

Endnotes

1. Many states have established formal definitions for levels of evidence, which provide a common language for discussions about programs’ demonstrated effectiveness. These definitions acknowledge that available data on programs’ effectiveness often vary by both the rigor of the underlying research and the number of studies that test outcomes.

   The strongest evidence typically comes from multiple rigorous evaluations, such as randomized controlled trials and evaluations that incorporate strong comparison group designs to moderate evidence.

   • Programs considered “promising” generally should have some research demonstrating effectiveness, such as a single randomized controlled trial or evaluation with a comparison group design.
   • Other programs may have a strong logic model or theory of change but lack rigorous study. In general, interventions with stronger evidence are more likely to produce positive outcomes when delivered at scale.

   Policymakers can refer to these definitions to create a shared understanding of evidence across agencies and branches of government and, over time, increase its use in the budget and policymaking processes.

2. The Pew Charitable Trusts, “Results First Clearinghouse Database” (June 2015), http://www.pewtrusts.org/en/research-and-analysis/issue-briefs/2014/09/results-first-clearinghouse-database.
3. Wyoming Survey & Analysis Center, “Catalog of Environmental Prevention Strategies” (October 2012), http://www.wishschools.org/resources/Catalog of Environmental Prevention Strategies_Final4 Wyoming.pdf.
4. Mississippi Legislature, 2014 Regular Session, “House Bill 677” (2014), http://billstatus.ls.state.ms.us/2014/pdf/history/HB/HB0677.xml.
5. The Pew Charitable Trusts, “Results First at the Local Level: Evidence-Based Policymaking in Four California Counties” (September 2015), http://www.pewtrusts.org/~/media/assets/2015/09/ results_first_ca_case_study_web.pdf.
6. The Pew Charitable Trusts, “Evidence-Based Policymaking: A Guide for Effective Government” (November 2014), http://www.pewtrusts.org/~/media/assets/2014/11/ evidencebasedpolicymakingaguideforeffectivegovernment.pdf.
7. The Pew Charitable Trusts, “Massachusetts’ Evidence-Based Approach to Reducing Recidivism” (December 2014), http://www.pewtrusts.org/~/media/assets/2014/12/ resultsfirstmassachusetts2014casestudy.pdf.
8. See Colorado Office of State Planning & Budgeting, “Colorado Results First: Results Guide” (2015), https://drive.google.com/file/d/0BxVE8p2dzVf3OFRaNjZQVWVQTjA/view.
9. Maryland Department of Budget and Management, “FY 2017 Operating Budget Submission Requirements” (June 2015), http://dbm.maryland.gov/budget/Documents/operbudget/2017-instructions/sect4-03.pdf.
10. New Mexico Department of Finance and Administration, “Article 3A: Accountability in Government” (2004), http://nmdfa.state.nm.us/uploads/files/Budget Division/AGA.pdf.
11. Council on Virginia’s Future, “The Virginia Report: Staying Ahead in Virginia” (2015), http://future.virginia.gov/publications/docs/VaReport/2015VirginiaReport-Web.pdf; Council on Virginia’s Future, “Virginia Performs,” http://vaperforms.virginia.gov/.
12. See Mississippi Joint Legislative Budget Committee, “Budget Instructions/Forms,” memorandum regarding instructions for fiscal year 2017 budget requests (June 4, 2015), http://www.lbo.ms.gov/pdfs/obrsforms/2017_budget_instructions.pdf.
13. See Utah Governor’s Office of Management and Budget, “FY 16/FY 17 Model Business Case,” form, http://gomb.utah.gov/wp-content/uploads/sites/7/2015/09/FY-16_FY-17-Model-Business-Case.pdf.
14. Kristen Cox (executive director, Utah Governor’s Office of Management and Budget), interview by Pew-MacArthur Results First initiative, Dec. 15, 2015.
15. See Utah Governor Gary Herbert, Budget Recommendations: Fiscal Year 2017— Fiscal Year 2016 Supplementals, Table 10, http://gomb.utah.gov/wp-content/uploads/sites/7/2015/12/FY-2017-Governors-Budget-Recommendations.pdf.
16. Cox, interview.
17. Route Fifty, “2016 Top Management Challenges for State & Local Government” (2016), http://www.routefifty.com/feature/managementchallenges-state-local-government/.
18. See New Mexico Legislative Finance Committee, “Guidelines for FY17: LFC Appropriation Recommendations,” http://www.nmlegis.gov/lcs/lfc/lfcdocs/FY17 Budget Guidlines.pdf.
19. The Pew Charitable Trusts, “New Mexico’s Evidence-Based Approach to Better Governance” (August 2014), http://www.pewtrusts.org/~/media/assets/2014/08/ nm_results_first_brief_web.pdf.
20. See Oregon Criminal Justice Commission, “Agency Report on SB 267,” http://www.oregon.gov/oha/amh/ebp/reports/crim-justice.pdf; Oregon Youth Authority, “Interim Judiciary Committee Report on SB 267” (September 2012), http://www.oregon.gov/oya/docs/ Evidence-based Statute Report (SB 267) - September 2012.pdf; Oregon Department of Corrections, “2010 ODOC Report on Senate Bill 267 Compliance” (2010), http://www.oregon.gov/doc/CC/docs/pdf/2010_agency_report_on_senate_ bill267_compliance.pdf; Oregon Department of Human Services, Addictions and Mental Health Division, “Interim Judiciary Committee Report on Evidence-Based Practices, ORS 182.525” (September 2010), http://www.oregon.gov/oha/amh/ebp/2010 Addictions and Mental Health Division Report.pdf; and Oregon Youth Development Council, “Interim Judiciary Committee Progress Report: SB 267 (ORS 182.525)” (September 2012), http://www.ode.state.or.us/wma/ydd/1a1a2012sb267progressreportydc.pdf.
21. Paul Egbert and Kelly Officer (Oregon Criminal Justice Commission), interview by Pew-MacArthur Results First Initiative, Dec. 16, 2015.
22. Debbie Miller and Elvie Newcomb (Tennessee Department of Children’s Services), interview by Pew-MacArthur Results First initiative, Jan. 20, 2016.
23. Ibid.
24. Ibid.
25. See New Mexico Corrections Department, “Policy CD-010100” (September 2015), http://cd.nm.gov/policies/docs/CD-100000.pdf.
26. The Pew Charitable Trusts, “New York’s Investment in Evidence-Based Policymaking” (January 2016), http://www.pewtrusts.org/~/media/assets/2015/11/rf_nys_case_study.pdf.
27. Pew-MacArthur Results First Initiative, panel interview, April 28, 2015.
28. Hawaii Department of Health, “Child and Adolescent Mental Health Performance Standards” (July 2012), http://health.hawaii.gov/camhd/files/2013/07/Orange-Book.pdf.
29. Stan Michaels (Hawaii Department of Health), interview by Pew-MacArthur Results First initiative, Dec. 15, 2015.
30. New York State Office of Alcoholism and Substance Abuse Services, “2014 Prevention Guidelines” (2014), http://www.oasas.ny.gov/prevention/documents/2014PreventionGuidelines.pdf.
31. Mary Ann DiChristopher, Arlene González-Sánchez and Patricia Zuber-Wilson (New York State Office of Alcoholism and Substance Abuse Services), interview by Pew-MacArthur Results First initiative, Jan. 11, 2016.
32. Ibid.