Data and Research Offer a Path Forward for Courts in the Pandemic’s Wake

Courtesy of Renee Danser

The COVID-19 pandemic forced state courts throughout the country to move processes online at an unprecedented scale. But even before 2020, many civil courts had already begun adopting new technologies such as online dispute resolution (ODR), which offers the potential to settle some legal matters electronically. But despite the eagerness on the part of many court systems to adopt ODR and other online procedures, such as virtual hearings, the evidence base necessary to understand their potential effects on core functions is incomplete.

Renee Danser, associate director of research and strategic partnerships at the Access to Justice Lab at Harvard Law School, works to incorporate rigorous research into efforts to improve access to the justice system, including deployment of ODR in the courts. She began her career as a manager in Pennsylvania’s trial courts. Later, she served as deputy director of the Self-Represented Litigation Network, in which lawyers, judges, and other professionals focus on creating solutions that can help lay people navigate civil court. She joined the Access to Justice Lab in July 2018.

This interview was edited for length and clarity.

Q: Why should courts evaluate their ODR systems?

A: This technology is being used to facilitate the core functions and duties of the court system—making sure that people know about lawsuits they are involved in and that each party to a lawsuit has the opportunity to tell their story. Courts need to use scientifically sound methods to understand whether ODR is achieving these objectives, improving processes, and providing a positive experience for litigants.

Q: One way to do that is through a randomized controlled trial, correct?

A: That’s right.

Q: How does that work?

A: In a randomized controlled trial, or RCT, researchers select two groups seeking the same service or engaging with the same system, for instance, two sets of defendants in debt collection lawsuits. Then one randomly selected group is exposed to the intervention or program—ODR or other virtual court operations, in this case—and the other is not, which allows us to know for certain whether any differences between the groups are a result of the intervention or program. 

Q: What about for courts that haven’t yet implemented ODR?

A: They’re perfect for what we call baseline testing. That starts with gathering data from a court such as case processing times, wait times, et cetera. Then, the court can compare those baseline conditions with conditions after it implements ODR.

Q: How long should an RCT take to determine whether ODR or another new court technology is working?

A: If a study is conducted to compare ODR with the traditional court process by randomly assigning litigants in a particular type of dispute to either process, the higher the case volume, the sooner you can tell the difference. As such, if there aren’t a lot of cases moving through the system, you might need a longer period to collect enough data from each group to identify a meaningful difference between their experiences. In that kind of a lengthier study, it’s possible to build in checkpoints to identify preliminary findings or conclusions. And while those may change as more data is gathered, they can still yield valuable information.

Q: How do these evaluations account for potential future developments of ODR technology and thinking?

A: ODR is rapidly proliferating in state courts, and both the technology and its integration with court processes are likely to evolve as its use increases. The knowledge we glean from evaluations can contribute to these changes and promote improvements.

Q: How can evaluations benefit courts that have been using ODR for some time?

A: The results of these evaluations will help courts identify common strengths and challenges of the technology and then take the lead in discussions with vendors, by speaking from an understanding of what court users need and are able to use effectively.

Q: How will ODR change in-person processes in courts? How do evaluators control for those changes?

A: I think we all anticipate that long after the pandemic restrictions lessen, there will be sweeping changes in courts’ in-person processes as a result of ODR implementation. But we won’t actually know how those processes will change until more courts implement these technologies and we see how users adapt. 

Q: Could randomized controlled trials help us better understand the impact of other technologies that courts used during the pandemic as well?

A: I caution courts on comparing pre- and post-pandemic data, because the pandemic is the largest confounding factor we’ve seen in our time. Are annual case counts down because we suspended trials for nearly a year or because courts have provided an online space for negotiation and resolution pre-filing? Or is it because self-represented litigants don’t know how to or can’t file their pleadings online? The only way to truly know if the pandemic response increases access to the civil legal system is to make the time and space to evaluate that response rigorously before moving on to the next transformative change.