In this column, Jim Landers, associate professor of clinical public affairs and Enarson fellow, John Glenn College of Public Affairs, The Ohio State University, describes two studies, how their methodologies resulted in contradictory findings, and the lessons they offer. For example, he observes that using different data, methods, and time periods can yield varied findings and conclusions, and he emphasizes the need to avoid overstating conclusions.
This piece was originally published by The Pew Charitable Trusts in a March 2021 newsletter distributed to tax incentive evaluators.
###
Reflecting on Evaluations of the Enterprise Zone in Louisville, Kentucky. How Can These Evaluations Inform Our Own Research?
Jim Landers
Associate professor of clinical public affairs, Enarson fellow
John Glenn College of Public Affairs
Over the past six years, participants at the annual National Conference of State Legislatures/Pew incentive evaluators roundtables have discussed evaluation methods and practices to assess how effective incentive programs are in generating additional economic activity. For example, do these initiatives spur new business locations, existing business expansions, or increased investment and employment? At last year’s roundtable, Dr. Tim Bartik of the W.E. Upjohn Institute for Employment Research continued this conversation by explaining the challenges of conducting rigorous causal analysis of incentive programs. I highlighted Bartik’s presentation in my December 2020 newsletter column. A recent debate about contradictory evaluations of the Louisville, Kentucky, Enterprise Zone (EZ) further highlights some of these challenges.
Last summer, Economic Development Quarterly published letters to the editor in its Forum section debating the methods and findings of two studies evaluating the efficacy of the Louisville EZ. The letters, written by Thomas Lambert and Sumei Zhang, discussed previously published studies by Lambert and Coomes (2001) and Zhang (2015) that produced opposing conclusions. Lambert and Coomes concluded that the EZ was not effective in encouraging investment or improving economic conditions while Zhang argued that the EZ achieved its expected goals in the long run by increasing economic growth in the manufacturing and service industry sectors. The letters and prior studies present interesting and relevant analysis topics for incentive evaluators. The authors used varied data and methods to draw their conflicting conclusions and the Forum letters, in particular, highlight their strengths and weaknesses as well as the challenges and benefits of conducting causal analysis of incentive program impacts.
The Louisville EZ was established in 1983 and terminated in 2003. In 1983, the EZ included 3.75 square miles of the city west and south of its central business district. It was expanded in 1984 and 1986 to include the city’s main airport, a large Ford Motor Company plant, a chemical manufacturing area, a green field industrial park, some older residential areas of the city, and the entire University of Louisville campus. After the 1986 expansion, the EZ included about 46 square miles. In addition, from 1988 to 1996, the airport underwent a substantial expansion funded by $600 million in local, state, and federal government funding. This was associated with an expansion in UPS facility operations at the airport and an increase from 1,000 to 14,000 UPS employees.
The study uses several approaches to examine the economic changes in the Louisville EZ. Several of the approaches use census tract data since the EZ is essentially conterminous with the census geography. The most informative approaches involved descriptive statistical analysis relating to EZ program activities and local socioeconomic conditions and a shift-share analysis of local employment.
The descriptive analysis comprises three parts: 1) a comparison of EZ and select non-EZ areas based on the change in employment and other socioeconomic measures from 1980 to 1990, using census tract data from the 1980 and 1990 decennial censuses; 2) a comparison of EZ and non-EZ areas based on the change in residential property values from 1992 to 1996 using state property assessment data; and 3) a survey of 851 out of 1,272 firms in the EZ that received tax and other financial incentives.
Generally, the growth rate comparisons on economic measures indicate that the original 1983 EZ lagged the non-EZ areas significantly on all measures. What’s more, while the 1984 and 1986 expansions improved the comparative performance of the EZ in a substantial way, the expanded EZ still lagged the comparison areas on some measures. The property value comparisons follow a similar track with the 1984 and 1986 expansions improving the comparative performance of the EZ. The surveys indicate that sales tax exemptions for purchases of equipment, building materials, and motor vehicles were, by far, the most used of the EZ incentives (i.e., used by more than 60% of the firms). However, 75% or more of the firms indicated that these incentives did not influence investment decisions. The remaining incentives, including a reduction in building permit fees and a property tax exemption, were used by less than 20% of the firms.
The shift-share analysis used ZIP code-level county business patterns data to assess employment growth and the competitive employment advantage of EZ and non-EZ areas. It indicated that the original 1983 EZ and the final EZ after the 1986 expansion lost jobs to other regions because of competition. In contrast, the non-EZ comparison areas gained employment from other regions. The only part of the EZ to gain jobs from other regions because of competition was the portion added in 1984, which included the airport expansion.
In his Forum letter, Lambert conducts additional descriptive analysis comparing the 1986 EZ and non-EZ areas based on: 1) change in various socioeconomic measures including unemployment rate, civilian labor force, poverty, average income, and average home value from 1980 to 2000 and 2) change in employment by industry sector from 1980 to 1996. The comparisons of socioeconomic measures show some improvement in the EZs relative to non-EZ areas with the addition of the post-1990 period. The employment comparisons indicate that transportation and public utilities were the only industry sectors to experience an employment increase, which Lambert attributes to the airport expansion and growth of UPS operations at the airport.
In her study, Zhang uses econometric techniques to conduct a causal analysis of the Louisville EZ and estimate its impact on employment growth. The study examines data from 1980 to 2000—nearly the entire life of the EZ. Instead of relying on census tract data as the unit of analysis, Zhang uses federally generated traffic analysis zones (TAZs) that can be perfectly matched to the EZ. Using TAZs offers several benefits to the analysis. They provide a richer source of socioeconomic data, providing residence-based information such as the population census, industry sector employment data, and commuting data for 1980, 1990, and 2000. Also, because there were more TAZs than census tracts in the EZ, the TAZ-level data set provides finer-grained information about the EZ and improves the explanatory power of the econometric model because the sample size is larger.
Zhang conducts a difference-in-differences (DID) analysis of the TAZ-level data using 1980 to 1990 as the pre-treatment period and 1990 to 2000 as the post-treatment period. This research design differentiates pre-treatment and post-treatment periods and compares the TAZs receiving the EZ treatment to TAZs not receiving the EZ treatment, which allows the researcher to evaluate whether a causal relationship exists. The focal variable in the DID model is a binary variable indicating whether a TAZ is located in the EZ. The economic outcome variable is the employment growth difference in a TAZ between 1990 and 2000 compared with 1980 and 1990. The pre-treatment period was determined by Lambert and Coomes' descriptive analysis and by Zhang's shift-share analysis, which together indicate that the economic performance of the areas in the EZ was relatively weak from 1980 to 1990 but strengthened from 1990 to 2000.
The DID research design also allows the researcher to control for observable factors other than the EZ that also influence EZ outcomes. These factors could be measures of local socioeconomic conditions, geographic location, or other government policies. If excluded from the DID model, the estimated impact of the EZ treatment could be biased. Lastly, Zhang uses an additional statistical procedure to correct for the impact of selection bias on EZ outcomes. In comparison with non-EZ areas, EZ outcomes could be biased low if the EZ is targeted to an extremely distressed area (e.g., the 1983 EZ) or biased high if the EZ is not well targeted and includes higher growth areas or areas with high growth potential (e.g., the 1984 and 1986 EZ expansions).
The econometric model estimates suggest that the EZ had varying causal impacts on employment by industry sector. Statistically discernible positive impacts of the EZ were estimated for manufacturing and service sector employment. Zhang also tested whether the estimated causal impact of the EZ on employment was due to the airport expansion by controlling for TAZ distance to the airport. This test did not change the statistical results, which contradicts the conclusions drawn in the Lambert and Coomes study and Lambert’s Forum letter about the impact of the airport expansion.
There are many relevant takeaways from the two studies and the debate about the efficacy of the Louisville EZ in the Forum letters. Here’s a few that I think are important: