How Has the Pandemic Continued to Affect Survey Response? Using Administrative Data to Evaluate Nonresponse in the 2022 Current Population Survey Annual Social and Economic[...]

Subscribe

In the latest release of its annual income report, Income in the United States: 2021, the U.S. Census Bureau tabulates the national income distribution using survey responses collected during February through April 2022 (with the majority of data collected in March) as part of the 2022 Current Population Survey Annual Social and Economic Supplement (CPS ASEC). However, as Research Matters blog posts from each of the last two years have explained (in 2020 and 2021), administration of the CPS ASEC has been negatively impacted by the ongoing COVID-19 pandemic, yielding lower response rates.¹ In-person CPS interviews have resumed since September 2021, so restrictions placed on the ASEC 2021 were no longer in place in 2022. Data collection operations in 2022 were similar to pre-pandemic methods, except in geographic areas where there was high risk for exposure to the virus, but response rates have remained low.

Figure 1 illustrates this phenomenon as it plots unweighted response rates to the monthly CPS through April 2022. While 76% of sampled U.S. households responded to the basic survey in March 2021 (during the busiest month of data collection for the ASEC), the response rate during March 2022 was 72%.

Source: Bureau of Labor Statistics, available at <www.bls.gov/osmr/response-rates/home.htm>.

Lower response rates, however, are not necessarily problematic if respondents and nonrespondents remain roughly similar, such that the resulting sample of respondents remains representative of the nation. To investigate this possibility, we combined administrative records, 2010 Census data, and American Community Survey responses to determine whether households that did and did not respond to the CPS ASEC differed, and whether these differences ultimately biased the resulting income and poverty estimates.Figure 2 plots earnings differences between respondents and nonrespondents using IRS Form W-2 earnings records linked by address. It shows that in 2020 respondents began to have higher earnings than nonrespondents, whereas in earlier years - 2017 to 2019 - the two groups had been much less different. That phenomenon has repeated in 2021 and 2022. Specifically, differences in median earnings between respondents and nonrespondents are statistically significant in 2017, 2018, 2020, 2021 and 2022, but each of the years from 2020 to 2022 have respondent-nonrespondent differences at the median that are significantly larger than the differences in each of the years 2017 to 2019, with the sole exception that differences in 2017 and 2022 are not significantly different from each other.²

Source: U.S. Census Bureau, Current Population Survey, 2017 to 2022 Annual Social and Economic Supplement (CPS ASEC) linked to IRS Form W-2 earnings data by address.

Still, the standard nonresponse adjustments could potentially have controlled for these unconditional earnings differences. In the usual course of survey data processing, the Census Bureau creates weights designed to adjust for nonresponse and to target weighted counts to independent population estimates by age, sex, race and Hispanic origin. To check whether these standard adjustments were sufficient to make respondents representative of the whole population, we tested whether survey response continued to vary by W-2 earnings even after controlling for all the factors in the weighting adjustment, as well as some other factors observed in the linked data. Figure 3 shows that earnings have indeed remained correlated with survey response since 2020, even after controlling for these other characteristics.³

Source: U.S. Census Bureau, Current Population Survey, 2017 to 2022 Annual Social and Economic Supplement (CPS ASEC) linked to administrative and survey data.⁴

The final step is to measure whether - and by how much - these differences between respondents and nonrespondents, even after adjusting for other characteristics, ultimately bias the resulting income statistics. As in previous years since the 2020 CPS ASEC, this measurement was estimated by implementing an alternative nonresponse adjustment that uses information from linked data sources that are not available in standard survey processing. This information includes linked administrative, decennial census and survey data available for respondent and nonrespondent households. Since low-income households have been less likely to respond, this adjustment shifts weight away from higher-income respondent households toward lower-income respondent households.

Using alternative weights created from the alternative nonresponse adjustment, we calculate alternative income and poverty statistics and compare them to the official (survey-only) estimates. We find that estimates of median household income and poverty rates were not statistically different in 2017, 2018 or 2019. This indicates that in pre-pandemic years the normal nonresponse adjustment in the survey weights sufficiently addressed nonresponse bias for these statistics.

However, Figure 4 shows that since 2020, the alternative income estimates have diverged from those estimated using the CPS ASEC survey data alone. Figure 4 plots the ratios of these estimates across the household income distribution. The horizontal line at one indicates the ratio at which the alternative-weight estimate would equal the survey-only estimate at that point in the distribution. While before 2020 these ratios hovered near one, since 2020 the alternative income estimates have been about 2% lower than the survey-only estimates.

Source: U.S. Census Bureau, Current Population Survey, 2017 to 2022 Annual Social and Economic Supplement (CPS ASEC) linked to administrative and survey data.

From 2020 to 2022, official poverty rates with the alternative weights were slightly - yet statistically significantly - higher than the survey estimates, by a half percentage point, four-tenths of a point, and three-tenths of a point, respectively. These differences contrast with the pre-pandemic years 2017 through 2019, which showed no statistically significant differences in poverty rates between the alternatively weighted estimates and the survey estimates.⁵

To summarize, since 2020, survey nonresponse has continued to bias income statistics upward by about 2% and official poverty rates downward by a fraction of a percentage point.

Definitions and information on confidentiality protection, sampling error and nonsampling error are available at <https://www2.census.gov/programs-surveys/cps/techdocs/cpsmar22.pdf>.

¹ These two blogs are available at <www.census.gov/newsroom/blogs/research-matters/2021/09/pandemic-affect-survey-response.html> and <www.census.gov/newsroom/blogs/research-matters/2020/09/pandemic-affect-survey-response.html>. More information about the design of the survey, including weighting for nonresponse, is available in Technical Paper 77 at <https://www2.census.gov/programs-surveys/cps/methodology/CPS-Tech-Paper-77.pdf> and in a summary of weighting in the CPS at <www.census.gov/programs-surveys/cps/technical-documentation/methodology/weighting.html>.

² We refer to survey years in the text, tables and figures to keep the references consistent and more clearly identify the 2020, 2021 and 2022 CPS ASEC as the ones affected by the pandemic. These surveys ask about income received in the previous year, however. Statistics from the 2020 CPS ASEC, for example, describe income received during 2019.

³ In particular, 2022 households with W-2 earnings greater than $100,000 were statistically significantly more likely to respond than the $25,000-$50,000 reference group, and this additional likelihood was also statistically significantly more than those in the pooled years 2017 to 2019 for households with W-2 earnings in the intervals $100,000-$150,000 and $200,000 and up.

⁴ This figure comes from a regression of household survey response on W-2 earnings and various demographic and socioeconomic characteristics. The baseline earnings group is $25,000 to $50,000. Therefore, the positive coefficient for "≥ 200,000" in 2022 means that households with $200,000 or more in earnings were more likely to respond to the CPS ASEC than households with $25,000 to $50,000 in earnings, after controlling for factors like race, Hispanic origin, age, etc.

⁵ The differences among the 2020, 2021 and 2022 differences between alternative poverty rates and survey poverty rates are themselves not statistically significant.