Federal health officials at the Centers for Disease Control and Prevention and the U.S. Food and Drug Administration internally acknowledged that a key statistical tool used to monitor COVID-19 vaccine safety signals was “mostly useless,” according to newly disclosed documents obtained by Sen. Ron Johnson.
The records, subpoenaed from the U.S. Department of Health and Human Services (HHS), include internal emails, draft manuscripts and peer-review comments exchanged between CDC and FDA researchers from 2021 to 2023. The documents were later analyzed by scientists affiliated with Children’s Health Defense (CHD).
At issue is the use of empirical Bayesian (EB) data mining — a statistical method employed within the federal Vaccine Adverse Event Reporting System (VAERS) to identify potential safety signals by comparing rates of reported adverse events across vaccines.
Internal correspondence shows that researchers were aware of limitations in EB data mining during the pandemic, particularly in detecting adverse events linked to COVID-19 vaccines.
In a December 2021 email, FDA official Dr. David Menschik wrote to co-authors of a proposed study that there was “a considerable bias toward the null” when using EB data mining in the “current, unprecedented situation,” suggesting the method could fail to detect real safety signals.
The study, which researchers had hoped to publish in The Lancet Infectious Diseases, relied on EB data mining of early COVID-19 vaccine data. However, after a peer reviewer stated that the likelihood of detecting a safety signal using the method was “likely close to zero,” the authors dropped the data-mining component and removed contributors associated with that analysis.
In another September 2021 email, Menschik acknowledged that “data mining has blind spots” and emphasized the importance of complementary surveillance systems, such as the Vaccine Safety Datalink (VSD), to cover those limitations.
One issue raised in the documents involves “masking,” a statistical phenomenon that can occur when similar vaccines are compared against one another. For example, if both the Pfizer and Moderna COVID-19 vaccines generate high numbers of reports for a specific adverse event, comparing them against each other could obscure a potential safety signal.
According to the records, EB data mining struggled to detect myocarditis as a safety signal in part because both mRNA vaccines had elevated reporting rates, potentially diluting the statistical signal.
Sen. Johnson, in a March 23 letter to HHS Secretary Robert F. Kennedy Jr., described the masking issue using a hypothetical analogy, arguing that comparing two potentially harmful substances could obscure risks if both are included in the same baseline analysis.
The documents also reference a study examining tinnitus — ringing in the ears — following COVID-19 vaccination. Despite internal acknowledgment that EB data mining may not accurately detect such safety signals, researchers proceeded with publication.
READ ALSO: Bayer sues Pfizer, Moderna over alleged use of Monsanto GMO Tech in COVID vaccines
In August 2024, the findings were published in the American Journal of Otolaryngology. Rather than removing the EB data-mining results, the authors added a disclaimer stating that their findings “cannot definitively exclude the possibility” that COVID-19 vaccines may increase the risk of tinnitus.
Internal emails show that Judith Maro, Ph.D., of the Harvard Pilgrim Health Care Institute, questioned the usefulness of the method for COVID-19 vaccines, writing that it would be “mostly useless” because the database contained so many COVID-related reports that creating a meaningful comparator would be difficult.
Dr. Narayan Nair of the FDA’s Office of Biostatistics and Pharmacovigilance responded that officials were aware of the limitation “before and during the pandemic,” but suggested reporting the findings while acknowledging the weakness as a study limitation — an approach previously used in other papers.
The controversy also touches on whether the CDC conducted proportional reporting ratio (PRR) analyses — another statistical approach for detecting disproportionate adverse event reporting.
In 2022, CDC officials told media outlets, including a Reuters fact-checker, that PRRs were considered a “crude measure prone to many false signals.” Instead, the agency described EB data mining as a “gold standard” technique and said it would continue relying on it.
Former CDC Director Rochelle Walensky reiterated that position in a September 2022 letter to Johnson, stating that EB data mining was viewed as a more robust method to mitigate potential false signals.
However, according to documents previously obtained through the Freedom of Information Act, the CDC acknowledged that it had not conducted PRR analyses of VAERS data related to COVID-19 vaccines.
When asked recently about current analytical methods, the CDC declined to detail specific approaches but noted that, on May 8, 2025, the CDC and FDA expanded public access to VAERS data through the WONDER database and downloadable files to improve transparency.
The FDA did not immediately respond to requests for comment regarding whether EB data mining remains in use.
The newly released documents have intensified debate over the methodologies used to monitor vaccine safety during the pandemic and whether acknowledged statistical limitations were adequately communicated in published research.