How Courts Embraced Technology, Met the Pandemic Challenge, and Revolutionized Their Operations

Overview

The outbreak of COVID-19 in early 2020 forced public services to shift to online operations in a matter of weeks. For the nation’s courts, that meant reimagining how to administer justice. Media coverage has focused mainly on the effects of the digital transformation in criminal courts, but a rapid deployment of new technology also took place in the civil legal system.

This adoption of digital tools in the civil courts has significant real-world implications. Unlike their criminal counterparts, civil courts do not guarantee a right to counsel, meaning they do not provide attorneys for those who cannot afford them. This leaves roughly 30 million Americans each year to navigate potentially life-altering legal problems, such as eviction, debt collection, and child support cases, on their own. For these litigants who are responsible for a variety of complex tasks—including finding the appropriate court to hear their case, filing motions, arguing before a judge, and interpreting laws—technology holds the promise of a more accessible system with better outcomes.

Even before the pandemic, national judicial groups such as the Conference of Chief Justices (CCJ) and the Conference of State Court Administrators (COSCA) had called on courts to use technology to improve the experience of litigants, especially people who do not have attorneys. And just months after the pandemic began, states throughout the country moved to adopt a range of technological tools to keep their court systems available to the public, quickly shifting from requiring people to submit paper documents and appear in person before judges to widespread use of electronic filing (e-filing) systems, virtual hearing platforms, and other tools.

To begin to assess whether, and to what extent, the rapid improvements in court technology undertaken in 2020 and 2021 made the civil legal system easier to navigate, The Pew Charitable Trusts examined pandemic-related emergency orders issued by the supreme courts of all 50 states and Washington, D.C. The researchers supplemented that review with an analysis of court approaches to virtual hearings, e-filing, and digital notarization, with a focus on how these tools affected litigants in three of the most common types of civil cases: debt claims, evictions, and child support. The key findings of this research are:

• Civil courts’ adoption of technology was unprecedented in pace and scale. Despite having almost no history of using remote civil court proceedings, beginning in March 2020 every state and D.C. initiated online hearings at record rates to resolve many types of cases.¹ For example, the Texas court system, which had never held a civil hearing via video before the pandemic, conducted 1.1 million remote proceedings across its civil and criminal divisions between March 2020 and February 2021. Similarly, Michigan courts held more than 35,000 video hearings totaling nearly 200,000 hours between April 1 and June 1, 2020, compared with no such hearings during the same two months in 2019.

  Courts moved other routine functions online as well. Before the pandemic, 37 states and D.C. allowed people without lawyers to electronically file court documents in at least some civil cases. But since March 2020, 10 more states have created similar processes, making e-filing available to more litigants in more jurisdictions and types of cases. In addition, after 11 states and D.C. made pandemic-driven changes to their policies on electronic notarization (e-notarization), 42 states and D.C. either allowed it or had waived notarization requirements altogether as of fall 2020.

• Courts leveraged technology not only to stay open, but also to improve participation rates and help users resolve disputes more efficiently. Arizona civil courts, for example, saw an 8% drop year-over-year in June 2020 in the rate of default, or automatic, judgment—which results when defendants fail to appear in court—indicating an increase in participation.² Although national and other state data is limited, court officials across the country, including judges, administrators, and attorneys, report increases in civil court appearance rates.³

• The accelerated adoption of technology disproportionately benefited people and businesses with legal representation—and in some instances, made the civil legal system more difficult to navigate for those without. Although all states and D.C. took steps to allow court business to continue during pandemic lockdowns, those options were not always available in all localities, for all types of cases, or for people without attorneys.⁴ Litigants with lawyers, on the other hand, found that technological improvements made it easier for them to file cases in bulk: For example, after courts briefly closed, national debt collectors who file suits in states across the U.S. quickly ramped up their filings, using online tools to initiate thousands of lawsuits each month.

  By contrast, litigants without legal representation, especially those with other accessibility needs, faced significant disadvantages, even when systems were technically open to them. For instance, users without high-speed internet service or computers faced significant hurdles when trying to access courts using the newly available tools. And although technology holds promise to improve the legal system for people with disabilities and limited English proficiency, courts—like various other government services—have struggled to ensure that their technology is accessible to all users.⁵ Of nearly 10,000 state and local pandemic-related orders reviewed for this study, none specifically addressed technology accommodations for people with disabilities and limited English proficiency.

Court officials have made clear that improvements in technology must benefit all parties. CCJ and COSCA approved a resolution in July 2020 recommending that their members “ensure principles of due process, procedural fairness, transparency, and equal access are satisfied when adopting new technologies.”⁶

Based on research and in consultation with CCJ, COSCA, and other experts, Pew has identified three key steps courts could take to realize the full potential of improvements in technology-driven tools:

1. Combine technological tools with process improvements to better facilitate resolution of legal problems.
2. Before adopting new tools, test them with and incorporate feedback from intended users.
3. Collect and analyze data to help guide decisions on the use and performance of the tools.

The monumental efforts made by state courts in 2020 and 2021 represent an important step toward modernization. This report examines courts’ transformation during the pandemic and assesses the extent to which it has made the civil legal system more open, with operations and procedures that are clear and understandable; equitable, so that all users can assert their rights and resolve disputes even without legal representation; and efficient, to ensure that people’s interactions with courts ensure due process and feel easy and timely. And finally, this report explores additional steps court systems could take to build upon their progress.

Methods

This study employed a two-pronged approach to data collection and analysis of state civil court responses to the coronavirus pandemic. To understand how rapid adoption of online processes affected the ways litigants could interact with the civil legal system, Pew researchers examined pandemic-related emergency orders issued by the supreme courts of all 50 states and D.C. between March 1 and Aug. 1, 2020. That five-month period featured the greatest amount of decision making related to court operations, technology adoption, and the suspension and resumption of various types of cases, of any span since the onset of the pandemic.

The analysis focused on technologies adopted to address court processes that occur across case types, including e-filing, virtual hearings, and e-notarization, as well as the management of specific types of cases—eviction, debt collection, and child support modifications—that fill civil dockets and acutely affect economic outcomes for individuals and families. Which technological tools were examined reflects the importance of two functions— court appearances and document submission—to litigants’ efforts to advance their cases.

Further, the research included a review of about 70 academic and “gray literature” sources (i.e., studies that have not been peer reviewed). About half of those related to how technology adoption affected the experiences of litigants in the three types of cases, including advantages and barriers to online court processes. The other half helped to place pandemic-related adoption of virtual hearings and e-filing within the broader historical context of courts’ use of technology.

Pew researchers also examined data from the U.S. Census Bureau and the Federal Communications Commission (FCC) on broadband internet and related technologies necessary for accessing online court services as well as from a Wesleyan University database of state and local emergency court orders to identify how often those orders referenced accessibility for people with disabilities and limited English proficiency. Please see the separate methodological appendix for more details.

Courts adopted technology at unprecedented speed and scale

In a typical court case, the first step in resolving a legal problem has been filing paperwork with the court clerk to initiate a lawsuit. The opposing sides then appear in court to learn the status of the case, report on whether they have been able to reach a settlement, and determine the steps needed before trial. The process also typically involves submission of evidence, including materials that need to be signed and witnessed by a third party, as well as status reports on negotiations, examination of evidence, and other tasks. And if the dispute is not resolved before the trial date, the parties then appear before a judge.

Even long before the pandemic, court officials recognized that technology would need to become a permanent feature of the legal system. In 2006, CCJ and COSCA called for courts to use technology to improve affordability, efficiency, and access.⁷ Other judicial bodies, as well as individual judges, have made similar pronouncements and recommendations over the past 20 years.⁸

However, that guidance had not delivered the sort of sweeping change that could benefit a variety of users. During the first two decades of the 21st century, some courts had been slowly moving their processes online. Their efforts focused mainly on two sets of functions: the completion of discrete tasks, including filing and notarizing documents; and the hearing of disputes by a judge. (See Figure 1.)

Figure 1

Digital Tools Can Help Courts Streamline Processes, Litigants Prepare for and Resolve Cases

Steps of a civil case and the technologies that support them

(c) 2021 The Pew Charitable Trusts

Further, the extent to which court systems were already online before the pandemic struck—and the types of technologies they were using—varied widely from one state to the next and between cities and counties within the same state.

However, as COVID-19 swept across the country, courthouses shut their doors, and state court systems moved swiftly to digitize their processes. Beginning in March 2020, all 50 states and D.C. adopted statewide or local rules to govern digital operations, shifting civil court business online in two areas: moving from in-person to virtual hearings and digitizing practical tasks—such as preparing and tendering court documents—that litigants must complete before a hearing. In particular, e-filing tools allow litigants to submit documents online, and e-notarization systems facilitate electronic verification of documents.

For evictions, one of the most common types of civil case, no jurisdiction in the country had consistently used virtual meeting technology for these proceedings before the pandemic, but by November 2020, 82% of all state courts were permitting or encouraging remote hearings, with 15% mandating them.¹² (See “Evictions Proceeded During the Pandemic.”)

And similar shifts took place across civil court dockets, as states quickly moved to use virtual meeting technology. For instance, neither Michigan nor Texas had conducted a single video hearing for a civil court case before the pandemic, but between April 1 and June 1, 2020, they conducted more than 35,000 and 122,000 video hearings, respectively.¹³

Further, before the pandemic, many states had some procedures for the electronic submission and verification of documents, but the COVID-19 lockdowns forced the adoption of additional tools and systems to allow business to continue. And the changes reflect court officials’ ability to put user needs before their own preferences and traditions, namely, complex paper-based and in-person functions.

As of 2019, 37 states and D.C. allowed litigants without lawyers to use e-filing to upload complaints, responses, and other documents directly to court systems, and 34 states had authorized e-notarization for official documents, such as written testimony and statements. (See Figure 2.)

As a result of the pandemic, 10 states created new paths for people without lawyers to file papers electronically using dedicated software or other mechanisms, such as email, because either they previously had no e-filing system or their existing tools were accessible only by attorneys. And beginning in March 2020, seven states began allowing electronically notarized documents for the first time.¹⁴ (See Figure 3.) For instance, Alabama courts had long allowed electronic signatures but did not accept electronically, remote, or virtually notarized documents before April 2020; in New Jersey, a 2020 law allowed for temporary use of e-notarization.¹⁵

Additionally, seven states and D.C. responded to the paperwork challenge by identifying alternatives to notarization.¹⁶ Ohio, for example, waived notarization requirements during the public health emergency, and South Carolina now allows court users to submit affidavits, which previously had to be notarized, with simple written certification from the filer that the affidavits’ statements are true.¹⁷ Such solutions reflect the courts’ commitment to examining operations with users’ experiences in mind and devising practical solutions to improve processes, especially for people without lawyers, rather than engaging in a blanket digitizing of all court tasks.

Court officials demonstrated a commitment to a more open, equitable, and efficient civil legal system

These changes are impressive not only because they show the ingenuity of courts in the face of an emergency and allowed court operations to continue during the pandemic, but also because they upended long-standing court norms to better serve court users. And as courts deployed online tools, court officials set out goals for ensuring that those technologies were implemented in ways that addressed inequities in civil legal proceedings.

In July 2020, CCJ and COSCA adopted a resolution declaring that “state courts must ensure that all parties to a dispute—regardless of race, ethnicity, gender, English proficiency, disability, socioeconomic status, or whether they have professional legal representation—have the opportunity to meaningfully participate in court processes and be heard by a neutral third party who will render a speedy and fair decision.”¹⁸

CCJ and COSCA also jointly released the following six guiding principles to help courts build on the technological advances made during the coronavirus pandemic:¹⁹

1. Ensure principles of due process, procedural fairness, transparency, and equal access are satisfied when adopting new technologies.
2. Focus on the user experience.
3. Prioritize court-user driven technology.
4. Embrace flexibility and willingness to adapt.
5. Adopt remote-first (or at least remote-friendly) planning, where practicable, to move court processes forward.
6. Take an open, data-driven, and transparent approach to implementing and maintaining court processes and supporting technologies.

States are also working to create technology guidance. In September 2020, the Texas Judicial Council adopted a statewide framework for implementing online dispute resolution to which all county and local courts must adhere.²⁰ The document gives straightforward guidance on how cases that cannot be resolved online should proceed to court, including procedural requirements to ensure that all parties have an opportunity to be heard and to present their cases before a judge.

In April 2020, the Michigan Virtual Courtroom Task Force released the Michigan Trial Courts Virtual Courtroom Standards and Guidelines to ensure that virtual courtrooms operate efficiently and with transparency,²¹ and published a comprehensive toolkit to help courts in the state comply with the new guidance. The guidelines are based on an assessment of best practices from courts across the country and the state and cover every step in the virtual hearing process, from notification and attorney-client communications to technical standards and press access.

And in June 2020, New York created the Commission to Reimagine the Future of New York’s Courts, a group of judges, lawyers, academics, and technology experts that is studying how courts operated during the pandemic. In April 2021, the group issued technology recommendations to “improve the efficiency and quality of justice services during the ongoing health crisis and beyond.”²²

Technology increased participation in civil courts

Early data indicates that court technology is beginning to deliver on its potential. During 2020, judges and other state court officials reported increases in case participation rates, which they have attributed to the move to remote proceedings.²³ Although recent data on participation in the civil system is limited, experts have noted an overall uptick across court settings. Before the pandemic, civil courts were plagued by a critical challenge to their integrity: low participation, particularly among defendants. From 2010 to 2019, more than 70% of respondents in debt collection suits across multiple jurisdictions failed to appear in court or respond to summonses, resulting in a default judgment for the plaintiff.²⁴ According to Michigan Chief Justice Bridget Mary McCormack, that rate of participation has “literally flipped. The number of people who now show up is as high as the number of people who didn’t show up in physical courtrooms.”²⁵

Case participation is typically measured in two ways: by the number of people filing or initiating lawsuits and by the count of defendants in cases filed against them. Data reviewed by Pew researchers suggests that by the second metric, online proceedings may have driven an increase in participation.²⁶ In June 2020, for example, Arizona civil courts saw an 8% decline in the rate of default judgment resulting from litigants’ failure to appear, compared with June 2019, indicating an increase in participation.²⁷ And the state found similar results at the local level. In Arizona’s largest county, Maricopa, the failure-to-appear rate for eviction cases decreased from nearly 40% in 2019 to approximately 13% in February 2021.²⁸

That finding is consistent with data from other court settings, which shows that failure-to-appear rates dropped dramatically in several states at the start of the pandemic. For instance in criminal courts, New Jersey reported that the no-show rate fell from 20% in the first week of March 2020 to 0.3% the week of March 16, when the state began using virtual hearings, and Michigan reported that its rate dropped from 10.7% in April 2019 to 0.5% in April 2020.²⁹ Similarly, court observers in Texas report that with the switch to video hearings, parent participation in child welfare cases increased in May and June 2020 compared with in-person hearings before the pandemic.³⁰ These state court reports of improved participation rates are consistent with national survey data in which judges cited increased participation as the leading improvement to come from the move to virtual proceedings.³¹

The boost in court appearances that followed the shift to virtual hearings is consistent with pre-pandemic assertions that reducing the day-to-day costs of coming to court—such as transportation, child care, lost wages, and travel time—would increase people’s ability to meaningfully engage in court cases.³² In addition, technology can be used to help people show up to court if tools are made available in multiple languages and are designed to serve people with a range of abilities. And although recent indications are promising, courts need more data to analyze and confirm the trend toward greater participation.

Further, the most active court users—attorneys—have reported a range of benefits associated with the move to online processes. According to one survey from Texas, most judges, prosecutors, and defense attorneys said that remote proceedings saved time and improved efficiency.³³ And in interviews, attorneys in Florida, Missouri, Montana, and Texas reported that not having to travel to and wait at court enabled them to serve more clients than before the pandemic.³⁴

Further, examples from across the country indicate that technology, when implemented thoughtfully, can effectively help people navigate the civil court system, even when they are not represented by an attorney. For instance, Suffolk Law School in Massachusetts, in collaboration with courts in three states, developed Court Forms Online, a website that improves on typical e-filing tools by offering a more user-friendly interface that guides litigants through various court processes. The site walks users through the steps for obtaining a domestic violence restraining order, applying for eviction protection under the Centers for Disease Control and Prevention (CDC) moratorium, and even handling certain appellate matters.³⁵ In one example, a woman was able to use forms provided through the website to electronically file a motion to the state’s Appeals Court and obtain a stay of her improper eviction just as the constable was beginning to move her out of her home.³⁶

In recognition of technology’s potential to make it easier for people to participate in court processes, more court officials plan to embrace virtual services. In 2021, CCJ and COSCA passed a resolution promoting the continued use of remote hearings; and in a June 2021 survey of 240 magistrates, trial judges, and appellate justices from across the country, a majority said they expect remote proceedings to become a permanent fixture of state courts.³⁷

Technology often made the civil legal system harder to navigate for people without lawyers

Although people using the civil legal system, regardless of whether they had legal representation, benefited from courts’ rapid adoption of technology, the advantages were disproportionately enjoyed by parties with lawyers.

This gap between the promise of technology to make courts more equitable for individuals without attorneys and the reality of its implementation is consistent with previous analyses of pioneering court systems that adopted new technologies around the turn of the century. In 2010, the NCSC examined seven states—Iowa, Michigan, Minnesota, New Hampshire, Oregon, Utah, and Vermont—at the forefront of court “re-engineering,” a restructuring of services that included the expanded use of technology.³⁸ But most of the solutions that the center observed were either exclusively for lawyers—such as e-filing systems accessible only by attorneys—or required too much legal expertise to be helpful to people using the courts without professional assistance.

Court processes are not fully open, transparent

Court administrators moved quickly to respond to the pandemic and communicate with the public about changes to court operations. But that rapid action also created some confusion for court users. Information shared on public websites and directly with litigants about online processes did not always fully explain key details, such as how and where documents should be submitted or which types of cases would be served by virtual hearings.

And in those instances, court users sometimes did not know where to turn for help and clarification.³⁹ As more operations moved online during 2020, courts worked to untangle complicated processes and used tools such as legal information portals, virtual help desks, and kiosks in public libraries to provide more usable and accessible public information, but these efforts have also been inconsistent.⁴⁰

Equity gaps

During the pandemic, technology has continued to disproportionately benefit parties with counsel and high-volume users of the court system, such as certain debt collectors, creating challenges to court officials’ goal of ensuring equitable processes. Even before the pandemic, debt collection lawsuits—the most common type of civil case—presented a challenge to the integrity of the courts. A 2020 Pew analysis found that in the several states where data was available, less than 10% of consumers had a lawyer and more than 70% of debt suits ended in default judgment for the collector.⁴¹

However, since the pandemic began, these cases have shown the inequitable availability of electronic court processes.⁴²

Large debt collectors, operating with significant professional legal assistance, leveraged new court technology to their advantage. A review of records from county and state court websites by ProPublica, an independent investigative news organization, found that some major collectors were able to accelerate their filings during the pandemic by using electronic systems to initiate lawsuits in bulk.⁴³ Texas court data likewise demonstrates that debt collectors were able to continue to bring lawsuits at the same rate in fiscal year 2020 as in the previous year.⁴⁴ And according to researchers in California, when courthouses in that state closed in April 2020, debt collectors were able to file as many suits against consumers as they had in April 2019, thanks to electronic filing.⁴⁵

However, electronic filing was not equally available to all: In eight states, people without lawyers had almost no way to file court documents in debt claims against them, leaving most debt defendants in those states unable to participate in court proceedings so that the judges could hear all the facts and render verdicts accordingly.⁴⁶

This research also found similar access and equity problems in eviction cases. Technology would ideally both allow plaintiffs to quickly file cases and give defendants a clear and easy way to respond. Instead, in nine states, people without lawyers had almost no avenue for filing paperwork in eviction cases. Such rules unintentionally advantaged landlords, who have representation in an estimated 90% of eviction cases, compared with 10% for tenants.⁴⁷ (See “Eviction Cases Proceeded During the Pandemic.”)

Efficiency gaps

A lack of consistent rules and offerings of online tools has also limited the potential efficiency that people could gain from their use. For instance, parents who have child support obligations but experience job losses or wage cuts are required to seek a modification of their payments to reflect their change in circumstance. Online tools could offer these people a faster, easier way to request a change and save them the cost of a trip to the courthouse. But many courts did not include online filing for parents in their 2020 technology innovations. Of the 43 states plus D.C. in which courts normally handle child support modifications, 33 and D.C. issued pandemic-related orders or set up formal procedures to allow individuals without lawyers to submit modification requests electronically. The remaining 10 states effectively rendered parents without counsel unable to modify their payments in a timely fashion while courthouses were closed. (See Figure 4.) Parents who fail to modify and subsequently miss payments are subject to enforcement actions, such as garnishment of wages and even jail time.

Even before the pandemic, 1 in 3 U.S. households faced a housing, family, or debt issue serious enough to result in an interaction with the civil legal system.⁴⁸ The sort of planning that identifies and supports the needs of users involved in such high-volume, high-need cases in civil court may not have been possible leading up to and during the first months of the COVID-19 outbreak, but now that the foundational work of moving processes online is done, court officials have an opportunity to improve and enhance those systems to better serve all litigants.

Courts’ technology choices hindered participation for some people without lawyers

During the pandemic, courts—like schools, government agencies, and some businesses—discovered that shifting processes from in-person to online does not necessarily make them easier to navigate. For people without the tools needed to use court technology, such as high-speed internet and a sufficiently powerful computer, the move toward modernization failed to improve their interactions with the civil legal system and may even have made them more difficult. And although technology can be used to make the courts more accessible to people with disabilities and limited English proficiency,⁵⁴ that promise remains largely unrealized. In practice, the new technologies often limited, rather than expanded, the ways in which these groups could interact with the civil system.

Despite their many documented benefits, the digital tools that courts implemented during 2020 often widened the chasm between people with and without attorneys. This was especially true of users with additional access needs.

Internet and computer access and experience

Access to internet service is the baseline requirement for web-based court technologies. Yet, despite the steady growth of internet use over the past two decades, as of 2018, 42 million U.S. adults lacked reliable broadband connectivity, including disproportionately low rates of access for certain populations and locations.⁵⁵ For instance, U.S. Census Bureau research showed that 36.4% of Black households and 30.3% of Hispanic households had neither a computer nor broadband subscription, compared with 21.2% of White and 11.9% of Asian households.⁵⁶

Further, families with incomes below $25,000 were less likely than those with higher incomes to have even minimal internet connectivity, and tribal and rural regions lagged far behind urban areas in terms of internet access.⁵⁷ According to a 2018 FCC report, slightly more than 77% of rural populations had access to an internet connection that met the agency’s benchmark for reliable connectivity, compared with 98.5% of urban populations. Tribal populations fared even worse, at 72.3%.⁵⁸

In addition, many court users access the internet only via smartphone. Approximately a quarter of Hispanic adults identify as “smartphone only” internet users.⁵⁹ Notably, location is not a driver of disparities in mobile internet access, with figures close to 100% no matter where one lives: 99.4% for rural populations and 97.5% for tribal populations, provided they have a mobile phone. But mobile access has limitations, particularly related to streaming live content—such as meetings via Zoom, WebEx, or similar platforms—which are basic requirements for participation in virtual hearings.

Although the courts cannot expand people’s access to broadband internet or computers, they can—and many do—recognize these roadblocks and adjust their processes to account for these challenges. For instance, 28 states and D.C. installed drop boxes outside courthouses to help litigants submit court documents during the pandemic.⁶⁰ And in some states, courts have permitted litigants for whom video technology is not an option to participate in hearings via telephone.

Even when users have sufficient tools to access a court’s online services, e-filing or participating in a video hearing requires a level of digital experience that many people lack. A 2019 Pew Research Center report found that most U.S. adults could answer fewer than half the questions correctly on a digital knowledge quiz.⁶¹ Younger adults and those with bachelor’s degrees were more likely to know the answers to questions about internet privacy measures, such as two-factor authentication, which many users must navigate to take advantage of online court processes.

Access for users with disabilities and limited English proficiency

State courts, like other public institutions, have specific obligations under the Americans with Disabilities Act related to access for people with disabilities, and federal law also requires courts to provide language assistance for those with limited English proficiency.⁶² But according to the National Center for Access to Justice’s 2021 Justice Index, which scores states from 0 to 100 on their adoption of specific policies related to disability accessibility and language access, including court access for people without lawyers, 44 states scored below 50 for accessibility, and 31 scored below 50 for language access.⁶³

Research indicates that, during the height of the pandemic, people relying on court documents for information related to obtaining an interpreter, ensuring reasonable accommodations for a disability, or generally accessing the courts during courthouse closures would have found little.⁶⁴ In a review of nearly 10,000 court documents from all 50 states and D.C., between February and October 2020, researchers from Wesleyan University found that only 253 documents mentioned language access and just 154 contained information for people with disabilities. In total, less than 3% of the documents referenced access for people with limited English proficiency, less than 1.5% mentioned the needs of people with disabilities, and none specifically addressed technology accommodations for these populations.

Now that courts have taken the step of adopting technology, they have an opportunity to use it to address longstanding inequities for these populations. By making sure their technology is accessible and multilingual, and offering a range of high- and lower-tech tools and resources to meet the diverse needs of their users, courts can ensure that technology improves the experiences of all litigants.

Recommendations

CCJ’s and COSCA’s adoption of technology principles is an important first step toward ensuring that measures taken to modernize the civil legal system benefit all users. However, now that state courts have practical, firsthand experience with legal technologies, court officials realize the urgent need to apply such guidance. To that end, and drawing on work with state and local court systems across the country, Pew has identified three important steps court officials should take to make their processes more open, equitable, and efficient:

1. Combine technological tools with process improvements.

Technology, if layered on top of complex court processes, will only reinforce the status quo: complicated, attorney-centered procedures that are difficult for people without lawyers to navigate. Court officials must examine the processes that litigants have to complete during various types of cases to identify opportunities to simplify forms and procedures.

One example of such an effort is that several states reviewed notarization policies, which in many instances led to the elimination of traditional verification requirements, such as in-person document review by a certified notary public. Another is how Hawaii leveraged its online dispute resolution (ODR) project to re-examine and revise its small claims process. ODR was originally developed as a dispute resolution mechanism in the e-commerce sector, and courts around the country began adopting it in 2014 to allow litigants to negotiate and resolve disputes among themselves outside of court business hours. Hawaii took the opportunity presented by the new system to add an early review step in which judges ensure that the collector-plaintiffs own the debts they are attempting to recoup before the case moves forward.⁶⁵ The state also developed and embedded in its ODR platform a user-friendly fee waiver application and review function so that litigants without lawyers must navigate only a single online platform.⁶⁶

2. Test new tools with intended users and incorporate their feedback.

Without rigorous testing of technology platforms, courts may find themselves locked into expensive systems that do little to simplify the legal process for their users. Testing not only helps courts refine and improve upon these tools, but it also gives them an opportunity to proactively engage with end users to make sure that the technology products are functional and meet their needs.

Court ODR pilot projects undertaken before the pandemic demonstrate that it is indeed possible to incorporate user feedback in the deployment of technology. For example, in 2019, the Utah courts engaged an external researcher to conduct a usability study of its ODR platform for small claims cases, which included testing by end users. The research uncovered various issues with the platform’s accessibility and functionality, and the court was able to make targeted improvements.⁶⁷

To help more courts undertake similar efforts, CCJ recommends models of participatory design, including convening stakeholders to establish shared goals, seeking design and implementation guidance from community organizations and key user groups, and incorporating user feedback mechanisms and usability testing in planning.⁶⁸

3. Collect and analyze data to help guide technology decisions.

Most states do not share information with the public in an easy-to-understand format. For example, Texas is the only state that collects and makes publicly available information on debt claims lawsuits, including outcomes, across all courts.⁶⁹

Some courts have begun to share their data with users and the media, for public information purposes, and with external evaluators to enable monitoring of their technology innovations. For example, courts in Florida, Michigan, and Texas have engaged third-party researchers to study their dispute resolution platforms. To support such analyses, NCSC developed a framework for evaluating ODR and made it available to courts across the country. And analysts at Indiana University are partnering with their state’s courts to examine the impact of online hearings on litigants without lawyers.⁷⁰ These state efforts will help courts better understand the effects of their online processes, leverage the benefits, and mitigate any harms.

CCJ and COSCA have promulgated a set of data elements that courts should collect and report on—as well as explanations of what those data elements can reveal about court processes pre- and post-pandemic—that states can use to create guidelines for the collection and reporting of court data.⁷¹ Having operated under pandemic protocols for more than a year, civil courts should engage researchers and other experts to help in developing metrics to measure modernization efforts, collecting data, identifying pandemic-era successes and areas for improvement, and fine-tuning technologies and systems. Such a thorough examination will allow courts to implement data-informed process improvements that enable them to better help people without attorneys navigate and resolve legal issues.

As courts collect and analyze the data on technological solutions, they should consider the following questions:

• What data must be tracked to answer key questions?
• How can litigants, attorneys, court staff, and other stakeholders be engaged in the process of improving the court experience?
• What can the civil court system learn from the experiences of other courts that are implementing and testing similar changes?

An analysis of efforts thus far not only will help courts operate more efficiently but also will help them improve the civil legal system on a broad scale.⁷²

Conclusion

Technology has the potential to substantially improve the civil legal system. Digital tools helped courts remain operational during the public health emergency and are poised to become permanent fixtures of the legal system. By studying how technology worked well—or did not—during the COVID-19 pandemic, courts can better understand their effects on litigants, especially those without lawyers, and undertake improvements to help Americans settle disputes and avoid life-altering consequences.

As courts work to assess and improve these tools, they will need to incorporate feedback from court users, test multiple technology products, collect and analyze use and performance data, combine technology with other process improvements, and implement the principles and safeguards that court officials already have identified as critical to ensuring effective use of technology. With these steps and proven tools, states can modernize the civil courts and make them more open, equitable, and efficient than ever before.

Endnotes

1. E. Scigliano, “Zoom Court Is Changing How Justice Is Served,” The Atlantic, April 13, 2021, https://www.theatlantic.com/magazine/archive/2021/05/can-justice-be-served-on-zoom/618392.
2. Conference of State Court Administrators, National Association for Court Management, and National Center for State Courts’ Joint Technology Committee, “Judicial Perspectives on ODR and Other Virtual Court Processes” (2020), https://www.ncsc.org/__data/assets/pdf_file/0023/34871/2020-05-18-Judicial-Perspectives.pdf; National Center for State Courts, “Will Remote Hearings Improve Appearance Rates?” May 13, 2020, https://www.ncsc.org/newsroom/at-the-center/2020/may-13.
3. See, for example: G. Jurva, “The Impacts of the Pandemic on State & Local Courts Study 2021: Remote Hearings, Legal Technology, Case Backlogs, and Access to Justice” (Thomson Reuters 2021), https://legal.thomsonreuters.com/en/insights/reports/impacts-of-the-pandemic-on-state-local-courts?form=thankyou&gatedContent=%252Fcontent%252Fewp-marketing-websites%252Flegal%252Fgl%252Fen%252Finsights%252Freports%252Fimpacts-of-the-pandemic-on-state-local-courts; Butler Snow, “Texas Appellate Law Podcast,” podcast audio, July 15, 2021, https://www.butlersnow.com/2021/07/disruption-and-increasing-access-to-justice-chief-justice-bridget-mccormack/; National Center for State Courts, “Study of Virtual Child Welfare Hearings: Impressions from Judicial Interviews” (2021), https://www.ncsc.org/__data/assets/pdf_file/0018/65520/Study-of-Virtual-Child-Welfare-Hearings-Judicial-Interviews-Brief.pdf.
4. Connecticut did not have any Supreme Court pandemic emergency orders, but remote hearings did take place on a jurisdiction-by-jurisdiction basis, according to state contacts.
5. American Bar Association, “Report on the Future of Legal Services in the United States” (2016), https://www.americanbar.org/content/dam/aba/images/abanews/2016FLSReport_FNL_WEB.pdf; E.C. Baig, “How a Wave of New Tech Products Are Making Life Easier for People With Disabilities,” USA Today, Sept. 10, 2018, https://www.usatoday.com/story/tech/columnist/baig/2018/09/10/technology-improves-people-disabilities-firms-respond-moral-legal-demands/835232002/; D.S. Raja, “Bridging the Disability Divide Through Digital Technologies” (World Bank Group, 2016), http://pubdocs.worldbank.org/en/123481461249337484/WDR16-BP-Bridging-the-Disability-Divide-through-Digital-Technology-RAJA.pdf; J. Rubin-Wills, “Language Access Advocacy After Sandoval: A Case Study of Administrative Enforcement Outside the Shadow of Judicial Review,” N.Y.U. Review of Law & Social Change 36, no. 3 (2012): 465-511, https://socialchangenyu.com/review/language-access-advocacy-after-sandoval-a-case-study-of-administrative-enforcement-outside-the-shadow-of-judicial-review/.
6. Conference of Chief Justices and Conference of State Court Administrators, “Resolution 2: In Support of the Guiding Principles for Post-Pandemic Court Technology” (2021), https://ccj.ncsc.org/__data/assets/pdf_file/0019/51193/Resolution-2-In-Support-of-the-Guiding-Principles-for-Post-Pandemic-Court-Technology-.pdf.
7. Conference of Chief Justices and Conference of State Court Administrators, “Resolution 13: The Emergence of E-Everything” (2006), https://ccj.ncsc.org/__data/assets/pdf_file/0011/23420/01182006-the-emergence-of-e-everything.pdf.
8. T.M. Clarke, “Reengineering: The Importance of Establishing Principles,” in Future Trends in State Courts (2010): 31-32, https://cdm16501.contentdm.oclc.org/digital/collection/ctadmin/id/1627. In his 2019 State of the Judiciary address, then-Iowa Chief Justice Mark Cady highlighted the promise of a range of technological approaches, noting, “Our future can no longer be about taking small steps or standing still. We need to think big and take big steps. Every day we must seek to achieve what can be imagined.” J.Q. Lynch, “Iowa Chief Justice Urges Lawmakers to ‘Think Big,’’’ Sioux City Journal, Jan. 16, 2019, https://siouxcityjournal.com/news/state-and-regional/govt-and-politics/iowa-chief-justice-urges-lawmakers-to-think-big/article_da82d5c2-99e7-5258-a24c-dc54c944258c.html. See also, for example: M.E. Recktenwald, chief justice, “State of the Judiciary”(Honolulu, Jan. 24, 2019), https://www.courts.state.hi.us/wp-content/uploads/2019/01/State-of-the-Judiciary_FINAL_01-24-19.pdf; H.D. Melton, chief justice, “2020 State of the Judiciary Address”(Atlanta, Feb. 26, 2020), https://www.gasupreme.us/wp-content/uploads/2020/02/Judi20.pdf; J.D. Kemp, chief justice, “State of the Judiciary”(Little Rock, Arkansas, June 14, 2019), https://www.arcourts.gov/courts/supreme-court/state-judiciary; M.G. Heavican, chief justice, “State of the Judiciary” (Omaha, Nebraska, Jan. 17, 2019), https://supremecourt.nebraska.gov/sites/default/files/State-of-Judiciary-2019.pdf; M.E. Barbera, chief judge, Maryland Court of Appeals, “2019 State of the Judiciary Address to the General Assembly of Maryland”(Annapolis, Maryland, Feb. 6, 2019), https://mdcourts.gov/sites/default/files/import/coappeals/speeches/soj2019.pdf; J. Williams, trial court administrator, Massachusetts Trial Court, “2019 State of the Judiciary” (Boston, Oct. 30, 2019), https://www.mass.gov/service-details/2019-state-of-the-judiciary-addresses.
9. National Center for State Courts, “Landscape of Civil Litigation in State Courts” (2015), https://www.ncsc.org/__data/assets/pdf_file/0020/13376/civiljusticereport-2015.pdf.
10. R. Zorza, “Some First Thoughts on Court Simplification: The Key to Civil Access and Justice Transformation,” Drake Law Review 61 (2013): 845-81, http://www.zorza.net/Simple.pdf.
11. National Center for State Courts, “Justice for All State Planning Documents: Lessons From the Field” (2018), https://www.ncsc.org/__data/assets/pdf_file/0016/26305/jfa-lessons-learned-final-2018.pdf; National Center for State Courts, “Guiding Principles for Post-Pandemic Court Technology” (2020), https://www.ncsc.org/__data/assets/pdf_file/0014/42332/Guiding-Principles-for-Court-Technology.pdf.
12. This analysis is based on emergency orders from all 50 states and the District of Columbia. The list of websites used is available upon request. See also E. Benfer, R. Koehler, and A. Alexander, “COVID-19 Eviction Moratoria & Housing Policy: Federal, State, Commonwealth, and Territory,” Boston University, COVID-19 U.S. State Policies Database, Aug. 23, 2021, https://statepolicies.com/policy-by-topic/economic-precarity/housing/.
13. Ibid.
14. DocVerify, “What States Allow Electronic Notary?” accessed Aug. 3, 2021, https://www.docverify.com/Products/E-Notaries/What-States-Allow-Electronic-Notary.
15. E. Rickard and Q. Naqui, “Response to Pandemic Pushes State Court Modernization Forward,” The Pew Charitable Trusts, July 21, 2020, https://www.pewtrusts.org/en/research-and-analysis/articles/2020/07/21/response-to-pandemic-pushes-state-court-modernization-forward.
16. Pew analysis of state supreme court COVID-19 pandemic emergency orders issued from March 1 through Aug. 1, 2020. Three states— Missouri, New Hampshire, and South Carolina—adopted both e-notarization and alternatives to notarization.
17. Rickard and Naqui, “Response to Pandemic”; Supreme Court of Ohio, “Temporary Notarization Forms,” accessed Sept. 20, 2021, https://www.supremecourt.ohio.gov/LegalResources/Rules/notarization/.
18. National Center for State Courts, “Guiding Principles for Post-Pandemic Court Technology.”
19. Ibid.
20. Texas Judicial Council, “Civil Justice Committee Report and Recommendations” (2020), https://www.txcourts.gov/media/1449780/civil-justice-committee-2020_0923_final.pdf.
21. Michigan State Court Administrative Office, Michigan Trial Courts Virtual Courtroom Standards and Guidelines (2020), https://www.courts.michigan.gov/4a1e83/siteassets/court-administration/standardsguidelines/operations/vcr_stds.pdf.
22. J. DiFiore, chief judge of the Court of Appeals and chief judge of the State of New York, “The State of Our Judiciary 2021,” March 2, 2021, https://www.nycourts.gov/whatsnew/pdf/21_SOJ-Speech.pdf; Commission to Reimagine the Future of New York’s Courts, “Report and Recommendations of the Future Trials Working Group” (2021), https://www.nycourts.gov/whatsnew/pdf/future-trials-working-grp-april2021.pdf.
23. Jurva, “The Impacts of the COVID-19 Pandemic on State & Local Courts.”
24. The Pew Charitable Trusts, “How Debt Collectors Are Transforming the Business of State Courts” (2020), https://www.pewtrusts.org/en/research-and-analysis/reports/2020/05/how-debt-collectors-are-transforming-the-business-of-state-courts.
25. Butler Snow,“Texas Appellate Law Podcast.”
26. National Center for State Courts, “Will Remote Hearings Improve Appearance Rates?”; Scigliano, “Zoom Court.”
27. Conference of State Court Administrators, National Association for Court Management, and National Center for State Courts’ Joint Technology Committee, “Judicial Perspectives on ODR and Other Virtual Court Processes.”
28. COVID-19 Continuity of Court Operations During a Public Health Emergency Workgroup, “Post-Pandemic Recommendations” (Arizona Supreme Court, 2021), https://www.americanbar.org/content/dam/aba/administrative/judicial/2021-az-post-pandemic-rec.pdf.
29. National Center for State Courts, “Will Remote Hearings Improve Appearance Rates?”
30. E.G. Thornburg, “Observing Online Courts: Lessons From the Pandemic,” Family Law Quarterly 54, no. 3 (2021), https://www.americanbar.org/groups/family_law/publications/family-law-quarterly/volume-54/issue-3/observing-online-courts-lessons-the-pandemic/.
31. Jurva, “The Impacts of the COVID-19 Pandemic on State & Local Courts.”
32. J. Greacen, “Remote Appearances of Parties, Attorneys and Witnesses: A Review of Current Court Rules and Practices” (Self-Represented Litigation Network, 2017), https://www.srln.org/node/1269/report-remote-appearances-parties-attorneys-and-witness-review-currentcourt-rules-practices.
33. J.I. Turner, “Remote Criminal Justice,” Texas Tech Law Review 53 (2021): 197-271, https://papers.ssrn.com/a=3699045.
34. A.L. Bannon and D. Keith, “Remote Court: Principles for Virtual Proceedings During the COVID-19 Pandemic and Beyond,” Northwestern University Law Review 115, no. 6 (2021): 1875-920, https://scholarlycommons.law.northwestern.edu/nulr/vol115/iss6/7.
35. Court Forms Online, “About Us: The Document Assembly Line Project,” accessed Aug. 20, 2021, https://courtformsonline.org/about.
36. D. Colarusso, director, Suffolk Law School Legal Innovation and Technology Lab, email to The Pew Charitable Trusts, July 23, 2021.
37. Jurva, “The Impacts of the COVID-19 Pandemic on State & Local Courts.” See also: A. Reed and M. Alder, “Zoom Courts Will Stick Around as Virus Forces Seismic Change,” Bloomberg Law, July 30, 2020, https://news.bloomberglaw.com/us-law-week/zoom-courts-will-stick-around-as-virus-forces-seismic-change.
38. D.J. Hall, “Reengineering Lessons From the Field,” Future Trends in State Courts (2010): 36-41, https://cdm16501.contentdm.oclc.org/digital/collection/ctadmin/id/1625.
39. See, for example: Z. Tillman, “Going to Court Without a Lawyer Was Always Hard. The Pandemic Has Made It Much Harder,” Buzzfeed, May 15, 2020, https://www.buzzfeednews.com/article/zoetillman/coronavirus-courts-closed-civil-court-legal-help#news-content.
40. Commonwealth of Massachusetts, “Remote/Virtual Court Services,” accessed Sept. 20, 2021, https://www.mass.gov/info-details/remotevirtual-court-services; E. Rickard and D. White, “Illinois, Michigan, and Ohio Residents Seek Legal Information Amid the COVID-19Pandemic,” The Pew Charitable Trusts, May 19, 2020, https://www.pewtrusts.org/en/research-and-analysis/articles/2020/05/19/illinois-michigan-and-ohio-residents-seek-legal-information-amid-the-covid-19-pandemic.
41. The Pew Charitable Trusts, “How Debt Collectors Are Transforming the Business of State Courts.”
42. bid.
43. P. Kiel and J. Ernsthausen, “Capital One and Other Debt Collectors Are Still Coming for Millions of Americans,” ProPublica, June 8, 2020, https://www.propublica.org/article/capital-one-and-other-debt-collectors-are-still-coming-for-millions-of-americans.
44. State of Texas, Office of Court Administration, “Annual Statistical Report for the Texas Judiciary: Fiscal Year 2020” (2020), https://www.txcourts.gov/media/1451853/fy-20-annual-statistical-report_final_mar10_2021.pdf.
45. C.J. Raba, clinical instructor, University of California, Irvine (presentation, Paths to Justice, Sept. 14, 2021). See also A. DiPierro, “Debt Collection Lawsuits Compound COVID Woes for Some Bay Area Residents,” Bay City News Foundation, Aug. 20, 2021, https://localnewsmatters.org/2021/08/20/debt-collection-lawsuits-compound-covid-woes-for-some-bay-area-residents/.
46. See: A. Warren, S. McKernan, and B. Braga, “Before COVID-19, 68 Million U.S. Adults Had Debt in Collections. What Policies Could Help?” Urban Wire (blog), Urban Institute, April 17, 2020, https://www.urban.org/urban-wire/covid-19-68-million-us-adults-had-debtcollections-what-policies-could-help.
47. H. Schultheis and C. Rooney, “A Right to Counsel Is a Right to a Fighting Chance,” Center for American Progress, Oct. 2, 2019, https://www.americanprogress.org/issues/poverty/reports/2019/10/02/475263/right-counsel-right-fighting-chance.
48. E. Rickard, “Many U.S. Families Faced Civil Legal Issues in 2018,” The Pew Charitable Trusts, Nov. 19, 2019, https://www.pewtrusts.org/en/research-and-analysis/articles/2019/11/19/many-us-families-faced-civil-legal-issues-in-2018.
49. Alabama Association of Realtors v. Department of Health and Human Services et al., 21-5093, 594 (Supreme Court of the United States, Aug. 26, 2021). Congress extended the CDC-issued moratorium in December 2020, and the Biden administration further extended it in January, March, June, and August 2021. See: National Housing Law Project and National Low Income Housing Coalition, “Federal Moratorium on Evictions for Nonpayment of Rent” (2021), https://nlihc.org/sites/default/files/Overview-of-National-Eviction-Moratorium.pdf; L. Wamsley, “Here’s What You Need to Know About the New Eviction Ban,” National Public Radio, Aug. 6, 2021, https://www.npr.org/2021/08/06/1025212834/cdc-new-eviction-ban-moratorium-renters-landlords.
50. Pew analysis of state supreme court COVID-19 pandemic emergency orders issued from March 1 through Aug. 1, 2020.
51. E.A. Benfer et al., “State and Federal Eviction Moratoria and Rental Housing Stabilization Measures During the COVID-19 Pandemic” (forthcoming, 2021).
52. J. Reichman, “First Results of the CDC Eviction Moratorium in Houston,” January Advisors (blog), Feb. 8, 2021, https://www.januaryadvisors.com/cdc-eviction-moratorium-houston/.
53. P. Hepburn et al., “U.S. Eviction Filing Patterns in 2020,” Socius 7 (2021): 1-18, https://doi.org/10.1177/23780231211009983.
54. B.R. Hough, “Let’s Not Make It Worse: Issues to Consider in Adopting New Technology,” in “Using Technology to Enhance Access to Justice,” Harvard Journal of Law and Technology 26, no. 1 (2012): 262, https://jolt.law.harvard.edu/articles/pdf/v26/26HarvJLTech241.pdf. The American Bar Association issued standards on language access: Standard 4.3 and accompanying commentary highlight the changing nature of technology that could benefit litigants with limited English proficiency. See: American Bar Association, “Standards for Language Access in Courts” (2012), https://www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_standards_for_language_access_proposal.authcheckdam.pdf. Further, the National Center for State Courts documented some specific uses of technology for language access, including video remote interpreting and remote translation of written documents. See: National Center for State Courts, “Call to Action: Achieving Civil Justice for All” (2016), https://www.ncsc.org/__data/assets/pdf_file/0021/25581/ncsc-cji-report-web.pdf.
55. N. McCarthy, “Report: 42 Million Americans Do Not Have Access to Broadband,” Forbes, Feb. 7, 2020, https://www.forbes.com/sites/niallmccarthy/2020/02/07/report-42-million-americans-do-not-have-access-to-broadband-infographic/?sh=751bda10b12a; the Pew Research Center, “Internet/Broadband Fact Sheet” (2021), https://www.pewresearch.org/internet/fact-sheet/internet-broadband/.
56. U.S. Census Bureau, “The Digital Divide: Percentage of Households by Broadband Internet Subscription, Computer Type, Race and Hispanic Origin,” Sept. 11, 2017, https://www.census.gov/library/visualizations/2017/comm/internet.html.
57. National Telecommunications and Information Administration, “NTIA Data Reveal Shifts in Technology Use, Persistent Digital Divide” (2020), https://www.ntia.doc.gov/blog/2020/ntia-data-reveal-shifts-technology-use-persistent-digital-divide#main-content.
58. Federal Communications Commission, “2020 Broadband Deployment Report” (2020), https://www.fcc.gov/reports-research/reports/broadband-progress-reports/2020-broadband-deployment-report.
59. S. Atske and A. Perrin, “Home Broadband Adoption, Computer Ownership Vary by Race, Ethnicity in the U.S.,” The Pew Research Center, July 16, 2021, https://www.pewresearch.org/fact-tank/2021/07/16/home-broadband-adoption-computer-ownership-vary-by-race-ethnicity-in-the-u-s.
60. This analysis is based on emergency orders from all 50 states and Washington, D.C. See the separate methodological appendix, available at https://www.pewtrusts.org/-/media/assets/2021/11/clsm-court-tech-methodological-appendix.pdf.
61. E.A. Vogels and M. Anderson, “Americans and Digital Knowledge,” the Pew Research Center, Oct. 9, 2019, https://www.pewresearch.org/internet/2019/10/09/americans-and-digital-knowledge.
62. T.E. Perez, assistant attorney general, Civil Rights Division, U.S. Department of Justice, letter to chief justice/state court administrator, Aug. 16, 2010, https://www.lep.gov/final_courts_ltr_081610.pdf.
63. National Center for Access to Justice, “About the Justice Index,” accessed Aug. 20, 2021, https://ncaj.org/state-rankings/2020/disability-access/about-justice-index.
64. National Science Foundation, “Award Abstract # 2028981—Rapid: Procedural Changes in State Courts During COVID-19,” accessed Aug. 20, 2021, https://www.nsf.gov/awardsearch/showAward?AWD_ID=2028981&HistoricalAwards=false.
65. Ibid.
66. M. Acosta and A. Min, “Tiny Chat 64: Hawaii ODR,” National Center for State Courts, September 2021, https://www.ncsc.org/newsroom/public-health-emergency/tiny-chats.
67. D. Hirsch, principal court management consultant, National Center for State Courts, email to The Pew Charitable Trusts, Oct. 13, 2020. See also: National Center for State Courts, “Eight Lessons to Consider for ODR Implementation” (2020), https://www.ncsc.org/__data/assets/pdf_file/0020/58016/8-Lessons.pdf.
68. S. Butler et al., “The Utah Online Dispute Resolution Platform: A Usability Evaluation and Report” (working paper, University of Arizona, 2020), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3696105.
69. M. Hagan, “Participatory Design for Innovation in Access to Justice,” Daedalus 148, no. 1 (2019): 120-27, https://doi.org/10.1162/DAED_a_00544; National Center for State Courts, “Guiding Principles for Post-Pandemic Court Technology.”
70. The Pew Charitable Trusts, “How Debt Collectors Are Transforming the Business of State Courts.”
71. A.L. Miller, P. Hannaford-Agor, and K. Genthon, “An Evaluation and Performance Measurement Framework for Online Dispute Resolution: Assessing Improvements in Access to Justice” (National Center for State Courts, 2021), https://www.ncsc.org/__data/assets/pdf_file/0022/65641/ODR-Evaluation-Performance-Measure-Framework.pdf.
72. National Center for State Courts, “Open Data Principles to Promote Court Technology Post-Pandemic: Key Data Elements to Collect and Report” (2020), https://www.ncsc.org/__data/assets/pdf_file/0028/59671/Open-Data-Principles-Corrected-2.pdf.
73. J. Karp, “Trial Alternatives Getting Fresh Look With COVID-19 Backlog,” Law360, Feb. 4, 2021, https://www.law360.com/articles/1351450/trial-alternatives-getting-fresh-look-with-covid-19-backlog; R. Lewis, “Justice Delayed: Courts Overwhelmed by Pandemic Backlog,” CalMatters, Jan. 19, 2021, https://calmatters.org/justice/2021/01/justice-courts-overwhelmed-pandemic.