The Leadership Conference’s Comments on Internet Panel Pretesting

View a PDF of the comments here.

December 18, 2023

Ms. Sheleen Dumas
Department PRA Clearance Officer
Office of the Under Secretary for Economic Affairs
U.S. Department of Commerce
1401 Constitution Ave., NW
Washington, DC 20230

Submitted via

Re: Generic Clearance for Internet Panel Pretesting and Qualitative Survey Methods Testing (FR Doc. USBC–2023–0002, OMB Control No. 0607–0978)

Dear Ms. Dumas:

On behalf of The Leadership Conference on Civil and Human Rights, a coalition charged by its diverse membership of more than 240 national organizations to promote and protect the civil and human rights of all persons in the United States, and its Census Task Force co-chairs, Asian Americans Advancing Justice | AAJC and NALEO Educational Fund, we appreciate this opportunity to provide comments in response to the Census Bureau’s request for approval of the proposed extension of the Generic Clearance for Internet Panel Pretesting and Qualitative Survey Methods Testing, published in the Federal Register on October 17, 2023.

The Leadership Conference is the nation’s oldest, largest, and most diverse civil and human rights coalition and provides a powerful unified voice for the many constituencies we represent. Our coalition views the collection of useful, objective data about our nation’s people, housing, economy, and communities to be among the most important civil rights issues of our day. Our longstanding role as a Census Information Center has allowed us to lift up within our broad civil rights coalition the fundamental importance of comprehensive, high-quality data about our population, communities, and economy.

To that end, we value the importance of multi-modal testing to ensure that data collection methods and operations effectively count and measure equally well all communities within a survey universe, especially historically undercounted population groups. The accuracy and inclusiveness of test results must take precedence over cost-efficiency whenever less-costly testing methods could narrow the scope of research in a way that skews the broad application of the outcomes.

Relying solely or primarily on medium-scale internet pretests to identify effective question formats and wording, procedures, contact and communications strategies undoubtedly will leave out demographic subgroups that are unable to respond or less comfortable responding to surveys online. As we have seen in prior testing cycles, the Census Bureau might be forced to scale back its testing overall due to funding shortfalls, delays, and uncertainties, leaving fewer opportunities to test methods and operations using other important response options — including paper questionnaires, telephone, and in-person interviews — which are more likely to capture historically undercounted and hard-to-count population groups. We saw this regrettable scenario play out last decade, when the Census Bureau was forced to cancel or scale-back several large-scale field tests for the 2020 Census, including all tests planned for Indian country and in rural areas.

We recognize the importance of using cost-efficient methods to test survey methods, questions, and contact strategies. Certainly, testing via online response portals could preserve limited fiscal resources not only during the testing phases for a decennial census, the American Community Survey (ACS), and other vital surveys, but during the production phases, as well. Nevertheless, we strongly urge the Census Bureau to consider accuracy as the overarching goal of all phases of survey development and implementation, especially for the universal decennial census and related ACS.

Relying too heavily on testing via the internet and electronic communications (such as texting and emails), to the exclusion of alternative modes of contact and data collection, can easily narrow the range of communities the Census Bureau reaches, thereby skewing test results. Previous research and survey evaluations have shown that some demographic cohorts — whether by age, race, ethnicity, and gender, and by socio-economic characteristics such as place of birth, educational attainment, and income — are less comfortable responding to surveys online or face barriers to doing so, such as lack of access to broadband. In addition, the population cohorts that the Census Bureau is less likely to reach by internet may interpret questions (e.g., wording, format, terminology) differently than those who respond online, further limiting the usefulness or applicability of test outcomes. While testing solely via the internet might save resources in the short term, the fiscal benefits are illusory. A failure to fully evaluate methods and strategies with a fully representative sample of respondents, many of whom may be completely overlooked by a singular testing method, undoubtedly will result in higher costs during data collection (production), as the Census Bureau must conduct more costly operations, for a larger portion of households, in order obtain responses.

We support medium-scale testing using electronic methods, and we recognize the overall advantages to increasing internet response for all surveys and the census. Nevertheless, as we saw in the 2020 Census, heavy promotion of and reliance on internet self-response appears to have contributed to disparate self-response rates between communities with and without reliable broadband connectivity, such as lower income neighborhoods, rural areas, and on American Indian reservations. Analysis of census results clearly show that low self-response areas are likely to have higher undercount rates among historically undercounted population subgroups. Therefore, the Census Bureau should consider the 2020 Census experience in designing tests for the 2030 Census, ACS, and other demographic surveys. Research that relies too heavily on internet pre-testing to determine what methods to pursue in large-scale tests, in order to achieve cost-savings, could reduce the accuracy of field tests and the subsequent surveys or censuses themselves. It is paramount that any cost-saving measures do not contribute to persistent differential inequalities of outcome in the census and other surveys, especially for populations that have been historically undercounted.

Thank you for your consideration of our concerns and views. If you have any questions about these comments, please contact Meeta Anand, senior director of the census and data equity program at The Leadership Conference on Civil and Human Rights, at [email protected].


The Leadership Conference on Civil and Human Rights

Asian Americans Advancing Justice | AAJC

NALEO Educational Fund