For a short time, I got uplifted and excited about a study coming out of Los Angeles County and one coming out of Santa Clara County, both with some research roots at Stanford (not too shabby of an academic institution). The uplifting implication, based on some COVID-19 antibody testing, was that a lot more people had the society-halting coronavirus than previously expected — which would mean that the death rate of the virus is much lower than expected and we can more easily and more quickly get back to “normal life” as we watch cases and deaths fade away.
As I started to look into these studies more closely, I found they were peppered with enough holes that they may as well have been blasted by a shotgun. They have not been peer reviewed (which is critical to make sure there aren’t deep flaws in the methodology) and they do not confidently (if at all) tell us what the reporting about them has been telling us. I’ll go through the LA County one, and the same issues seem to apply to the other.
First of all, jumping back to the misleading “findings” for a moment, the study indicated that 4.1% of the LA County population had antibodies to COVID-19, approximately 55 times more of the population than the county’s official infection count. That would mean the associated deaths from COVID-19 represented a much smaller percentage of all infections than expected, which would mean that we humans could go out in society again with much less fear of catching a virus that might kill us.
“This study suggests that the covid fatality rate in LA County, currently estimated to be around 4%, is probably more likely between .15 and .09%, when accounting for all the infections that have not been counted,” a Los Angeles Times reporter tweeted.
One of the challenges probably warping the results, however, is that the testing methodology reportedly can’t distinguish between different coronaviruses. Many people testing positive for the virus might just have (or recently have had) the common cold. That would demolish the main finding of the study. As someone on Twitter summarized, “They used an assay that can not discern SARS-CoV-2 from common cold coronaviruses and MERS. This is one of many issues. But it means that they vastly overestimated the number of infections.”
Secondly, the methodology for finding people to respond to the survey was not ideal. The researchers solicited participation from people on Facebook, likely leading to people with symptoms or who thought they may have had the virus joining the study simply in order to get tested. (Recall that it’s been very hard for people to get tested.) While the researchers reportedly used a representative sample of the general population, there’s still “consent bias” from people choosing whether to participate or not. Also, there’s the possibility people were sharing the link in Facebook groups or with friends and family members who wanted to get tested.
One more issue is that the total sample size was 870, quite small for this topic.
The tests themselves may not even be up to the job. “[S]ome scientists have raised concerns about the accuracy of kits used in such studies because most have not been rigorously assessed to confirm they are reliable,” an article in Nature writes.
“The Santa Clara study reports using a kit purchased from Premier Biotech, based in Minneapolis, Minnesota. According to the preprint, the manufacturer’s kit performance data noted 2 false positives out of 371 true negative samples.
“But with that ratio of false positives to true cases, a large number of the positive cases reported in the study — 50 out of 3320 tests — could be false positives, says Marm Kilpatrick, an infectious disease researcher at the University of California Santa Cruz.”
Premier Biotech, the company producing the antibody tests in California, reportedly found a 90–95% accuracy in the test results, but the accuracy may be considerably worse than self-reported by the Chinese firm. The YouTube doctor above discussed research indicating that the test’s accuracy was far worse than 90–95% claimed. Getting to the cruz of this point, if you find that there’s a 4.1% infection rate but the testing is so inconsistent, then there’s a chance 0% of the population actually had the virus. Of course, that’s not the case, but it shows you how much we can trust the accuracy of the findings.
All you can say with some confidence is that no more than 5% of the population has been affected.
“All total garbage. Used lateral flow rapid tests. Majority of ‘positives’ are false positives with these. Totally unreliable. Rates 1-3% at most,” someone tweeted.
Furthermore, even if the mortality rate from COVID-19 was 0.15% as implied, if 300 million people in our country (most of the population) got it, that would mean 450,000 deaths. Not so uplifting.
This virus has already been overwhelming some hospitals, has led New York City to dig mass graves, and has countless doctors and nurses saying they feel like they’re “working in a war zone.
Whatever the story, it’s not looking good right now.
That's exactly right and is what is missed by lots of folks championing the LA/Santa Clara studies. If mortality rate was 0.001% but infected all of america in 24h, it would be a catastrophe.
— Simeon Kwan 關明德 (@simeonkwan) April 21, 2020
There may be unattributed deaths. It is risky to mix official and unofficial numbers.
But, this needs to be repeated all over. And if it holds up then some way needs to be devised of locking down the senior homes and retirement communities while opening the rest.
— GVF (@i_gvf) April 21, 2020
There’s much more regarding the flaws of this study here: