Minnesota has expanded its internal capacity to evaluate programs and guide their improvement to ensure better public outcomes with the creation of the state’s Impact Evaluation Unit. A part of the Minnesota Department of Management and Budget (MMB), the unit launched in 2019 as an innovative initiative by legislators who wanted to evaluate more state-funded programs to learn what works—and to bolster the state’s response to the opioid crisis.
The evaluation team is now leading studies of taxpayer-funded substance use disorder interventions and human service programs to better understand their effectiveness and potential for improvement. Other states can learn from Minnesota’s investment in research capacity to make evidence-informed budget plans and strengthen services for residents.
The Minnesota Legislature in 2019 directed MMB to evaluate grant-funded human service programs to determine their effectiveness. The department then brought on researchers to do the studies and establish the new unit.
“It’s awesome to have an in-house research resource,” said Neerja Singh, behavioral health clinical director at the Minnesota Department of Human Services. Singh wants to see the impact of new programs. “We have good interagency collaboration. It’s becoming the norm to bring in MMB to plan for evaluation whenever we start a new program.”
The legislature also directed MMB to evaluate grant-funded services offered by the Opiate Epidemic Response Advisory Council, which oversees state efforts to address the opioid crisis and the handling of opioid use disorders (OUDs). With about $5 million in annual revenue, the council provides grants to state and county agencies and private providers to deliver OUD prevention, treatment, and recovery services.
The Impact Evaluation Unit focuses on evaluating council-funded programs that have not been classified as “proven effective.”
“The unit has shown rigorous work,’’ said state Senator Julie Rosen (R), sponsor of the legislation that created the unit. “They give information that helps to make our investments accountable. We want to know: Will the program be worth the investment?”
MMB hired six full-time staffers who specialize in scientific research methods and statistical data analysis, and who have evaluation expertise, to conduct the studies. The researchers have taken on a range of topics related to human services. For example, Weston Merrick, the unit’s senior manager, said one upcoming project will look at the impact of changing practices in driver license suspension for failing to pay child support. Another will analyze efforts to connect people leaving incarceration with Medicaid and other benefits for which they may be eligible.
In August 2021, the unit completed its first evaluation of an opioid-focused initiative, Project ECHO (Extension for Community Health Outcomes), a telementoring and education program developed at the University of New Mexico Health Sciences Center. Project ECHO connects primary care providers to specialists elsewhere to offer training and support for addiction treatment, including best practices for providing medications such as buprenorphine for OUD. The researchers found that providers in Project ECHO were more likely than similar providers to prescribe buprenorphine, so the project was considered successful.
“What was very exciting with MMB was using their vast data resources to really tie in what the ECHOs are doing to tangible changes in practice,” said Dr. Gavin Bart, director of addiction medicine and ECHO team physician with Hennepin Healthcare, the health care system that operates the program. “It’s having a downstream effect on patients, which is the ultimate goal. And we would not have been able to know that without collaborating with MMB.”
Findings from the Project ECHO evaluation helped expand the program’s reach to more counties, including underserved locations. This was important because opioids are the leading cause of drug-involved death, according to data from the Minnesota Department of Health. Moreover, historical data has shown that Native Americans in the state are up to seven times more likely to die from a drug overdose than White people, and African Americans are twice as likely.
“We know it helps because we know more prescriptions are now given to underserved patients seeking care,” said Dr. Brian Grahan, who directs the opioid-focused Project ECHO.
Last fall, Hennepin Healthcare collaborated with the Native American Community Clinic to host the Midwest Tribal ECHO to help train health care teams to provide more culturally appropriate care for Native Americans struggling with OUD.
Conducting an impact study, such as the one for Project ECHO, involves listening to program and agency needs, forming feasible research questions with program stakeholders, and setting evaluation goals guided by the agency or program’s preferences. It also requires managing program directors’ expectations by advising which programs are good candidates to be evaluated.
“We have to say we can’t promise an evaluation is possible, but we can help them see what we can study,’’ Merrick said. “Most program managers want to learn what’s working and what isn’t. And it can be hard for them to turn down projects because an impact evaluation isn’t feasible.”
When an impact evaluation is not possible, Merrick said the unit can provide other technical assistance, such as helping to identify outcome measures from administrative data or referring program directors to other researchers who can do descriptive evaluations.
Once programs are selected for evaluation, the unit reports its study plans to stakeholders and other researchers—before collecting data or reporting findings—by publicizing its study proposals through a pre-analysis registration. “Our evaluation policy and preregistration requirements help legislators see that we’re nonpartisan,” explained Merrick. “And we don’t show reports that lean intentionally in favor of an agency or provider. Pre-registration is a way to uphold the principles [in our evaluation policy].”
Unit staffers communicate regularly with state partners, sharing project updates and findings with commissioners, legislators, and agency leaders. The team gives analyzed data sets back to the agencies and provides them with suggestions for how to use the data in planning, monitoring, and further evaluation of programs and services. After the unit finishes its studies, team members work with stakeholders to find ways to implement findings to improve program performance or administration.
The unit can manage four to five projects at a time; studies are complex and can take 12 to 14 months to complete. Established interagency data sharing agreements help the unit keep down overhead charges that can substantially increase costs in contracted studies. As a part of these agreements, unit staff can work with agencies to access, prepare, and analyze data while following security and privacy requirements.
As leaders in other states look to invest pandemic-related funding from the American Rescue Plan Act and other sources, they should consider strengthening their evaluation capacity so they better understand which programs work and for whom.
Steve Lize works on The Pew Charitable Trusts’ Results First initiative.