Stopping an outbreak is always time-sensitive, so you don’t really have time to double-check results before you initiate tracing contacts and isolating them. Also I definitely believe that false positives are related to true positives. What we really need is a test to tell us whether a symptomatic person is shedding virus and is therefore infectious. One night, a cab is involved in a hit and run accident. Hmm. But I think in early summer cases rose, then hospitalizations, then deaths. This is actually a thing I’ve heard advocated here in the US, reducing the maximum cycle count so as to avoid this issue. I haven’t run numbers on that, but by eye it looks to have a weekly modulation. Something odd going on right now in TX (and probably other states). Given the possibility of ‘stale’ PCR tests for weeks or even months after infection, if everyone who is admitted to hospital is tested, could that mess things up if there are relatively few currently symptomatic people but many cases in the recent past? Most of us in healthcare have a fairly good understanding of math but are not nuanced in the field of statistics. If presented with related base rate information and specific information, people tend to ignore the base rate in favor of the individuating information, rather than correctly integrating the two. Hence a 2 SD range of +/- 7% of the mean, which gives the right range. Base rate fallacy/false positive paradox is derived from Bayes theorem. So areas where the base positive rate is higher, the % of positives that are false positives is lower? Repeat the PCR test multiple times and see it come up negative repeatedly. The Times article, which is not so old—it’s from 29 Aug—is entitled, “Your Coronavirus Test Is Positive. That would imply that either testing was growing / shrinking in step with the spread / decline of the virus, or that New York was *right at* R=1 for quite a while. And cases are possibly messy because TX is reporting a lot of back-log old cases not counted in the “new daily”. (2) Should it indicate virulence and the likelihood of a person’s own mortality due to Covid? What counts as a “case” has changed over time. It’s kinda like when you find a burnt spot of ground: sure, that area may not be in flames now, but there sure was a fire, so you want to know whereit may have spread while it was burning. I’m not sure if it’s 10% or 50% but it’s undoubtedly more than 5% of the positive tests that are not true positives. The original question is why the % positive is so consistent. Of course it’s possible to contaminate with the synthetic positive control, but again, if everyone jumps on the positive result and does a re-test, re-testing will reveal it was spurious. I do not think that assumption is valid anywhere in USA over any period longer than a few weeks. Yeah, I’m not saying that entirely explains it either. A few options to consider: (1) Should a positive test only indicate presence or vestige of the virus? https://www.washingtonpost.com/graphics/2020/national/coronavirus-us-cases-deaths/?utm_campaign=wp_to_your_health&utm_medium=email&utm_source=newsletter&wpisrc=nl_tyh&wpmk=1. By base rate fallacy/false positive paradox, if the specificity of a test is 95%, when used in a population with a 2% incidence of disease -- such as healthy college students and staff -- there will be 5 false positives for every 2 true positives. The Indiana State Department of Health advised against a random testing program, as it felt overall data accuracy would be difficult. Therefore, of the positive results, only 60/ (60+97)≈38% will be correct! I think your .35% SD was intended as a percentage of the mean of ~.92%. The base rate … by I was wondering if you have any comment on the NY State Covid numbers. Conjunction fallacy – the assumption that an outcome simultaneously satisfying multiple conditions is more probable than … New Zealand has practically no positives even though it does 100.000s of tests, I think a positive test rate of <0.03% which I would take as an upper bound for false positives. If positive the person is quarantined and contacts are traced and tested. Restaurant occupancy, sporting events and other large gatherings are again limited at a greater level than state requirements. Their lateral flow assay monitoring (known high number of false positives) or the PCR testing, where whole countries like New Zealand can have no cases despite continued testing? (or a FN). The tests are "good enough" for diagnosing patients with symptoms but not nearly as effective when used for a random testing program. There’s just no common timeline upon which things can lead or lag each other in a way that shows up in the trackers. Another wrinkle for the measurement problem; both of contagious individuals and viral load sufficient to be related to death. Half a million passengers travelled in the U.S. on June 11, continuing a travel rebound that would mean, one commentator says, a full return to normal by the end of summer. The NFL contamination case in August is an example of how a high false positive rate tied into a situation in a lab. In the past few months, we've seen that one of these odd behaviors is attributed to a significant number of health-news headlines recommending vitamin C to purportedly assist one's immune response to COVID-19. You then analyze how often the test gives incorrect results. Yes, and this might be true in some places, but looking at the # of tests performed in NY it does not seem to be true there. Abstract: We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. It’s even possible (although I have no idea whether it is true in Texas) that the definition of a “COVID related death” has changed over time. *I’m sure IFR has dropped somewhat, but deaths did rise significantly in July…, Statistical Modeling, Causal Inference, and Social Science, In case you’re wondering . Also because of additional testing being available, Indiana is now performing at times 40,000 COVID tests per day. False negatives should not really occur in those with recent onset symptoms as viral shedding occurs prior to and for the first week or so of the clinical course. Of those, about 35 are positive each day, according to the university's dashboard. Such improvements to our sites include the addition of alt-text, navigation by keyboard and screen reader technology, closed captioning, color contrast and zoom features, as well as an accessibility statement on each site with contact information, so that users can alert us to any difficulties they have accessing our content. (The actual incidence of active COVID-19 in college age students is not known but estimated to be less than 0.6% by Indiana University/Fairbanks data. Panic happens because the media industry tends to engage in what can be described as a base rate fallacy (Hardman, 2015) which is the idea that people tend attribute a higher level of risk to a situation when they are not aware of the actual base rates of such phenomena. Cases are clustered in the city, with certain neighborhoods experiencing more cases than others. Purdue University made the decision in late spring to resume in-person classes for its fall session. For Covid 19, we have far more accurate figures from 20 February 2020 to the time of writing: 32,330 deaths. Maybe It Shouldn’t Be. Using the same test on patients with COVID-19 symptoms, because their incidence of disease is 50% or greater, the test does not have to be perfect. If at any time you have questions or concerns regarding accessibility, or experience technical issues, please contact us at firstname.lastname@example.org. Do you think we are at the limits of the test and there may be a significant amount of false positives? What do you make out of that? Yet those numbers would be only representative of the positivity of mass testing, not the prevalence of infective patients. In order to have the best experience on our sites, we recommend that users utilize the latest available versions of web browsers and assistive technology. Throw all those four groups in together if you want, but just understand you are not getting a true picture of what is going on. By those increased numbers of testing, 4% of our Indiana population is now being tested for COVID-19 every week. Bad decisions can be made because of a misunderstanding of statistics. Now I’m commenting on things I understand poorly, but wouldn’t you expect that the contamination rate would be fairly variable, depending on whether some lab tech got a bad night’s sleep or was fighting with their partner, etc.? And the questionably “false” positives where the sample is really positive in the PCR sense (there is actual COVID RNA) but the person is not sick or infectious (the viral RNA is old fragments of virus, not “live” infectious virus) will only occur if some of the population tested has had COVID in the past. The pretest probability of a patient having COVID-19 versus another diagnosis is dependent upon the community base rate of COVID-19. Contact traced people identified as being close to a COVID patient WITH symptoms (>10% incidence of testing positive for COVID) would also be another category and those identified by contact tracing who were near a person who tested positive WITHOUT symptoms (>1% incidence of having COVID) would be a fourth. Contact tracers are telling positive testers who have nowhere to isolate to be evaluated at their hospital emergency room. Without knowing the specificity of the test, the number of these positives that are false positives is unknown. Our efforts are ongoing. False positives might also occur due to cross-reactivity with other corona viruses. If the only variation of the numbers were from random sampling variation, then the standard deviation would be about 0.35%, based on 90,000 tests per day (test count data from https://coronavirus.jhu.edu/testing/individual-states/new-york ). The truncation value is usually 40 but I have seen 45. The cut-off for a yes/no test is determined based on the validation, typically a number near but below the truncation value. less than 0.6% by Indiana University/Fairbanks data. © 2020 MedPage Today, LLC. The check samples are inserted into the sample stream by the people collecting the samples. Only one has been hospitalized and none have died. By not reporting these groups separately, we really have no idea what's going on in our town. But keep in mind you can also do multiple primers (roughly checking for different viral genes) and see some but not others cross the threshold. Often there are false positives in a validation but the test will still have a specificity near 100%. Only 14% gave the correct answer of 2% with most answering 95%. Commingling of data in our county from the people tested WITH symptoms together with the randomly tested Purdue students WITHOUT symptoms has occurred. There would also be variation in the number of tests performed each day. We investigate whether potential socioeconomic factors can explain between-neighborhood variation in the COVID-19 test positivity rate. So far, 90% of the students who test positive do not develop symptoms. I don’t really know what to think about all this, but I’ll share with you. There's nothing like fear to generate abnormal behavior. New data from China buttress fears about high coronavirus fatality rate, WHO expert says ... seems like a classic statistical fallacy. Unfortunately, the lack of understanding of the statistical principle of base rate fallacy/false positive paradox has led to some confusing numbers. Let's take a closer look. What counts as “COVID related hospitalization” has changed over time. We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. Purdue has discussed using a serial testing protocol. So then would the picture of the “base rate fallacy” effect be different than if there were no heterogeneity and the base rate was uniform? > These are not randomized tests, through a sparse, clustered set of interactions with a great deal of heterogeneity. MedPage Today believes that accessibility is an ongoing effort, and we continually improve our web sites, services, and products in order to provide an optimal experience for all of our users and subscribers. The base rate is the actual amount of infection in a known population. There are both known positive and negative controls on those trays. The numbers have caused our county health department to move cautiously. Day after day the positive percentage stays in a tight range of about 0.85-0.99%. The number of tests doesn’t seem to be changing that much, so it would still imply an oddly flat curve. If the false positve rate is tied to the true positive rate via contamination, then that doesn’t explain why it is supposedly steady: it should rise and fall with the true positive rate. >>where whole countries like New Zealand can have no cases despite continued testing? Haven’t read all responses, but assume it is PCR tests and their false positives that are discussed. What should a “positive test” ideally indicate? The base rate fallacy, also called base rate neglect or base rate bias, is a fallacy. Up to this point, Purdue has done random testing on about 1,000 students per weekday. I also wonder if it could be an issue of defining “COVID related” hospitalizations. The last example brings me to what is perhaps the most pervasive reason behind the conjunction fallacy: we tend to ignore base rates. For example, this happens when scholars like Kahneman and Tversky attribute to their experimental subjects the errors of the so-called conjunction fallacy and base-rate fallacy, and also when it is claimed that someone has committed the gambler's fallacy (Woods , 478–492). The material on this site is for informational purposes only, and is not a substitute for medical advice, diagnosis or treatment provided by a qualified health care provider. Testing procedures might be different between countries too. So if that were why, then would we expect the trend to change soon (IE either hospitalizations to drop, or cases to rise)? Different places use different primers, equipment, and sample collection then different thresholds for what counts as a positive. When the incidence of a disease in a population is low, unless the test used has very high specificity, more false positives will be determined than true positives. I have worked with PCR data for a long time. why do you state that there is a high proportion of false positives? We might think that the rate of “hospitalizations” would drop followed by a drop in the rate of “deaths”. The difference in the numbers can be quite striking and certainly not inherently understandable. Typically specificity, 1- the false positive rate, is reported as 99.9%, not 100%, when there are no false positives. This shows that NZ is doing around 100-200k tests a month, https://www.health.govt.nz/our-work/diseases-and-conditions/covid-19-novel-coronavirus/covid-19-current-situation/covid-19-current-cases. We’re doing in the US as many tests every day as NZ has done EVER. It’s always tough when you’re looking at a press release to figure out what’s going on.”. How can the range be so narrow and stable? All rights reserved. MedPage Today is committed to improving accessibility for all of its users, and has committed significant resources to making our content accessible to all. Diversity in an approach is fine but the problem is that how the details vary over time and location are unavailable then all the numbers get treated the same. . A classic 1978 article in the New England Journal of Medicine reveals this problem. A witness claims the cab was green, however later tests show that they only correctly … Base rate fallacy/false positive paradox unfortunately becomes ignored when one does this. Since staff and students combined are 50,000 at Purdue University, 5,000 tests are done every week. No wonder FP and FN rates are all over the place than. Eight weeks ago, Indiana was performing 20,000 tests per day. this is why the U.S. health care system is the most expensive in the world, https://www.wvdl.wisc.edu/wp-content/uploads/2013/01/WVDL.Info_.PCR_Ct_Values1.pdf, https://www.medrxiv.org/content/10.1101/2020.08.03.20167791v1, https://coronavirus.jhu.edu/testing/individual-states/new-york, https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30424-2/fulltext, Are female scientists worse mentors? NZ went a long time with no positive samples, during that period I’d expect very low false positive rates. Base rate fallacy/false positive paradox is derived from Bayes theorem. But .00032 / .0092 is 3.5%, not .35%. Tests for the coronavirus range from 90% to 99% specificity. CP Scott: "Comment is free, but facts are sacred" To first order you might say the probability of a false positive is something like k * pp, where pp is the percentage of true positives and k is a number between say 0.003 and 0.1 but If pp = 0 then doesn’t matter how big k is you won’t get any. A classic explanation for the base rate fallacy involves a scenario in which 85% of cabs in a city are blue and the rest are green. If the only variation of the numbers were from random sampling variation, then the standard deviation would be about 0.35%, based on 90,000 tests per day (test count data from https://coronavirus.jhu.edu/testing/individual-states/new-york). At the empirical level, a thorough examination of the base rate literature (including the famous lawyer—engineer problem) does not support the conventional wisdom that people routinely ignore base rates. As of a week ago, our two local hospitals with a combined 350 beds had 18 patients admitted with a COVID diagnosis. You also do not know if a low virus concentration in the sample really means a low virus concentration, for example the swabbing may not have been done properly.