The 2020 Novel Coronavirus Outbreak | Thoughts on Probability and Statistics

Bayes’ Rule, Unreliable Diagnostic Testing, And Containing COVID-19

How false-negatives in diagnostic testing are leading to the release of infected people, motivating extreme containment measures. The COVID-19 outbreak, explained with Bayes’ Rule.

If you are reading this after 2020, please keep in mind that this post was written during the early stages of the COVID-19 pandemic, and hence, may not reflect a reality beyond this time.

We are currently in February 2020. Over the past month, a deadly virus has been spreading throughout China and the world, sending the infected to the ICU and trapping others in their homes. As authorities try to manage this crisis, they face the challenging issue of containment — sending the infected to quarantine, while allowing the non-infected to go free.

The Problem With Epidemics That Plagues The Authorities

Here is the scenario. You have a cough and a fever. There is a chance that you have caught COVID-19 — the virus spreading throughout the world. You don’t know what this chance is, and you don’t want to take chances, so you seek advice from your doctor.

The issue from the authorities perspective is different. You want treatment, but the authorities need to contain the spread of the virus. From their point of view, there are 4 main outcomes of your visit to the doctor.

1. If you are infected and diagnosed with coronavirus, they will quarantine you for the public benefit.
2. If you are not infected but you are diagnosed with coronavirus, they will wrongly quarantine you, causing you inconvenience. The public will suffer no major harm, but the authorities will have to expend a small amount of resources.
3. If you are infected and not diagnosed with coronavirus, they will wrongly release you, causing you to spread the virus. This puts the public in grave danger of an outbreak.
4. If you are not infected and not diagnosed with coronavirus, they will rightly release you and save some resources.

The two mistakes that the authorities can make is scenario 2 and 3. Scenario 2 is a minor inconvenience (if not done too often), but scenario 3 is the major issue which can cascade into a larger outbreak, even if only done once. If an outbreak occurs, they will have to do contact tracing for possibly hundreds of people, given the contagiousness and lethality of coronavirus. This will be extremely costly for them, so their primary interest is in minimizing the probability of the third scenario.

Thus, the authorities need to make an accurate diagnosis, so that they can avoid releasing the infected and quarantining the non-infected. To achieve this, the authorities first make an initial assessment of all suspected infections, whether they are patients at the medical clinic, or travellers from places with active outbreaks.

Initial Assessment of a Patient

There are clues which will hint at a COVID-19 infection.

• Location: there will be different probabilities of infection for people living in different places. People near the epicenter are more likely to be infected. People living near popular tourist destinations are more likely to be infected than people from isolated places like rural Alaska.
• Travel history: travellers from places near the epicenter or other outbreak locations are more likely to have the infection (hence, why travellers are screened).
• Social contacts: people who have close contact with the infected/those at risk of infection are more likely to be infected themselves.
• Symptoms: people who have symptoms characteristic of coronavirus, such as fever, cough, and shortness of breath are more likely to be infected.
• Imaging: people who show imaging features of coronavirus on an x-ray or CT scan are more likely to be infected.

Not all of these clues will be immediately available. Authorities at the airport can only screen people for temperature and identify travel history if needed. A doctor will know about the symptoms disclosed by the patient, but it is on the doctor to take the initiative to find and link the clues together. And an asymptomatic case may not have any clues.

You go to the doctor, and the doctor asks you a series of questions. You tell the doctor about your cough and fever. The doctor is suspicious and builds their own belief of whether you are infected or not. Their belief is strong enough to justify a coronavirus test, so they order you to take this test.

The Probability and Statistics Behind Diagnostic Testing

After gathering enough clues, a doctor may suspect coronavirus. To confirm this suspicion, the doctor will order a diagnostic test.

Misconceptions about what the diagnostic test result means

A diagnostic test is performed by collecting samples from your body (eg. mucus in the back of the nose) and looking for presence of the virus in those samples. It seems simple enough, and people have a lot of faith in science. This may lead people into making this first mistake.

Incorrect interpretation: a positive result means a patient has novel coronavirus, while a negative result means that a patient does not.

This is not true, because the test is not always reliable. There are many reasons why a test may give a misleading result:

• A patient in the very early stages of an infection may not excrete a detectable amount of virus.
• The virus itself may only exist deeper inside the body, hence being inaccessible by a swab test.
• There may have been accidental contamination of the sample.

In general, the people who make the testing kits will specify the reliability of the test.

Suppose that a company now markets their test as “90% accurate”. This can lead to another common mistake.

Incorrect interpretation: for a 90% accurate test, a positive result means 90% chance of being infected, and a negative result means 90% chance of not being infected.

This interpretation is also not true, but is actually surprisingly common in the medical community— the great psychologist Gerd Gigerenzer shows how doctors misinterpret the results of mammogram results article.

Bayesian probability explains what the diagnostic test really means

The correct way to evaluate a diagnostic test requires thinking in terms of Bayesian probability. To put it simply, Bayesian probability involves having a prior probability and then using new information to update it. In terms of diagnostic testing, the prior probability is the doctor’s belief about whether the patient is infected or not. The test result is used as information, and this changes the doctor’s belief.

We can formulate the reliability question in terms of math equations.

Prior probability

Posterior probability: how reliable is a positive result?

Posterior probability: how reliable is a negative result?

What we need to do now is to connect the doctor’s initial belief with the final belief. To do this, we use Bayes’ rule, which can easily be derived using basic facts about conditional probabilities.

The denominator is expressed like this.

We now have two new unknown probabilities.

1. The probability of a positive result given that a patient is infected. This number should be high — infected patients should be getting positive test results.
2. The probability of a positive result given that a patient is not infected . This number should be low — non-infected patients should be getting negative test results.

These two probabilities are actually measures of a test’s reliability, and they can be expressed in terms of two quantities: sensitivity and specificity.

Sensitivity

• Probability of a positive result given infection.
• The test is “sensitive” to the presence of coronavirus. If the coronavirus is present, the test will detect it.
• Ideally, close to 100%.

Specificity

• Probability of a negative result given no infection.
• The test is “specific” to coronavirus. If there is no coronavirus infection, the test will not detect anything, and returns negative.
• Ideally, close to 100%.

I should remind you here that I can express probabilities in terms of complements. Infection and no infection are mutually exclusive. Similarly, a positive and a negative result are also mutually exclusive. This means that our calculations will use these two equations, where A and B are events such as “the patient is infected”, or “the test result is positive”.

The probability of infection given a test result

We now have all the tools we need to interpret a test result

• Prior probability — initial belief
• Posterior probability — final belief
• Bayes’ rule — connects initial and final belief
• Sensitivity and specificity — allows us to do computations using Bayes’ rule

Combining these, we get

We now know how to incorporate the information from a test result to change an initial belief into a final belief. Next, we look at diagnostic tests with different levels of reliability, and see how their results might affect the beliefs of doctors.

The Application of Diagnostic Tests in Medicine

You followed the doctor’s advice and got a diagnostic test. The doctor now receives a positive or negative result for coronavirus and now needs to make a decision:

1. Prescribe any necessary medication and let you go.

2. Order you to be quarantined.

Decision Making in Medicine — Reducing Uncertainty And Taking Acceptable Risks

To decide when to treat, the doctor needs to have confidence in a diagnosis.

In general medical practice, a doctor can make two mistakes:

1. You are not infected, but the doctor decides that you are, and gives treatment.
2. You are infected, but the doctor decides that you aren’t, and does not give you the proper treatment / lets you go.

In order to avoid these mistakes, the doctor orders a test. They had a prior belief, and the test is used as information to reduce uncertainty. Once the doctor has enough certainty, the doctor can make a recommendation.

How much uncertainty can be reduced by a test?

We first look at a reliable test. Consider a test that has sensitivity and specificity at 98%. We show what the prior and posterior probabilities of infection are, given a positive or negative result.

• If you think the probability of someone being infected is 80%, then a positive result takes it to near 100% (near certain infection), while a negative result reduces the probability to under 10% (unlikely infection).
• Similarly, if you think that the probability of someone being infected is only 20%, then a positive result increases it to over 90% (likely infection), while a negative result takes it to near 0% (infection is nearly impossible).
• Any initial probability which is greater than 10% increases to a higher probability (>80%) after a positive result.
• Any initial probability which is less than 90% decreases to a lower probability(<20%) after a negative result.
• Since the test is able to change a large range of initial probabilities into final probabilities near 0% and 100%, it resolves the uncertainty of whether a patient is infected or not. Hence it is a reliable diagnostic tool.

The Ideal test

• The orange curve (bottom, for negative result) is very close to the x-axis, which would mean that a negative result virtually guarantees a zero probability of infection.
• The blue curve (top, for positive result) is straight up the y-axis and right, which would mean that a positive result virtually guarantees a certain probability of infection.

Diagnostic tests are not perfect — bad luck exists, decisions are always risks.

If the doctor has a reliable test, then a prior belief of 50% can be reduced to less than 5%, or some other very low figure. Unfortunately, these tests are not perfect. There is still a non-zero probability of actually being infected, so the doctor can make a mistake out of sheer bad luck.

How much risk is tolerable?

Because the doctor ultimately has to make a decision, even under uncertainty, the doctor has to decide what is an acceptable risk.

Examples of where risk is unacceptable:

• Near-certainty is required before starting a harsh treatment with irreversible side effects.
• A very low probability of having a lethal infectious disease is required before the patient is released.

Examples of where risk is more acceptable:

• Less certainty is needed if the condition is mild and not life-threatening, and the treatment does not have permanent effects.
• Less certainty is needed if the patient is in immediate mortal danger and requires medical intervention as soon as possible (the potential harm of treatment is less than the harm of no treatment).

Decision Making Without Reliable Diagnostic Tests

(From here, we switch focus from general medicine, back to the coronavirus outbreak.)

We’ve established that a doctor needs to resolve uncertainty using a diagnostic test before they can have the confidence to take an acceptable risk.

However, what happens if the diagnostic test is not reliable?

We try to use realistic values. Many flu tests have a specificity of around 90–95%. Keeping in line with this, I assume a 90% specificity (feel free to correct me if you have better information).

I assume a 40% sensitivity. This turns out to be close to the reported sensitivity for COVID-19 coronavirus tests.

Suppose that a patient lives in an area which recently had an outbreak, and is now presenting to the medical clinic with symptoms of coronavirus. If the doctor deems the probability of a COVID-19 infection to be 50% (likely infection), and orders this unreliable test for the patient:

• A positive result increases the probability to 80% (likely infection)
• A negative result decreases the probability to a value to 40% (possible infection).
• 40% probability of infection is still too high. Given that the virus is contagious and lethal, releasing the patient is an unacceptable risk.

With the reliable test:

• A 50% probability of infection can be decreased to 2% with a negative result.
• A COVID-19 infection can potentially be ruled out (if 2% is acceptable risk).

Using a single unreliable test, a patient‘s probability of infection cannot be reduced to an acceptable level, regardless of a positive or negative result.

The unreliable test cannot rule out infection.

No decision making (for epidemics) without reliable diagnostic tests

Now imagine a scenario where people are quarantined until the authorities are confident that they are not infected. If the test is unreliable, then it cannot prove that there is no infection. In other words, the test has no consequences on decision making — the patient will remain in quarantine either way.

With contagion potential, unreliable tests have no influence on decision making.

Mistakes happen when test reliability is not understood

What if a doctor does not know the reliability of a test?

A doctor would understand that COVID-19 is highly contagious and lethal, and thus, their level of acceptable risk would be very low.

Suppose they mistook an unreliable test for a reliable test. Their initial belief of 50% probability would drop to an actual 40% probability of infection, while in their mind, it would be 2% because they assumed that the test was reliable. If they went on to release the patient, this would potentially have disastrous consequences.

This is why the tools used in the clinic are backed by research. Doctors need certainty about what their tools are and aren’t capable of.

Checking multiple times with multiple tests.

If a doctor knows about the unreliability of a test and trusts their own judgement, then a single negative test result will not convince them. The doctor will dismiss the test result as an error.

How many negative test results does it take to convince the doctor?

Assume that if the probability is reduced to less than 5%, then the doctor will be sufficiently convinced that there is no infection.

After a clinical examination, a doctor believes that a patient has 50% probability of having COVID-19. He knows that the diagnostic test is unreliable, with 40% sensitivity and 90% specificity, and so he orders a series of diagnostic tests.

Every diagnostic test returns a negative result. The initial probability is 50%, and each test decreases the probability.

50% → 40% →30.8% → 22.9% → 16.5% → 11.6% → 8.0% → 5.5% → 3.7%

In the graph, you can see that the green point has 5.5% prior probability and ends up with 3.7% posterior probability, which is past the grey threshold of acceptable risk.

It takes 8 consecutive negative results to convince a doctor who starts with a probability of 50%. Note that if it takes X tests starting from probability Y, then it will take at least X tests for any probability greater than Y. Any patient with an initial probability of infection greater than 8% will require at least 2 tests to rule out an infection.

Is this a feasible solution?

The Role of Unreliable Tests in the COVID-19 Outbreak

The COVID-19 Diagnostic Tests Are Unreliable — Low Sensitivity

For the current outbreak of coronavirus, there have been reports of negative results for people who are actually infected — a false negative.

On the patients being tested with nucleic acid tests (NATs), used to diagnose COVID-19.

“Not all of those infected by it return positive NATs”.

“Even patients who definitely have the disease only come back positive 30 per cent to 50 per cent of the time.”

“Testing throat swabs (from potentially infected people) also returns a lot of false negatives.” — Professor Wang Chen, director of the Chinese Academy of Medical Sciences (source: Caixin Media, a Feb 8 article.)

The experts themselves are saying that the tests are not sensitive enough — the probability of a positive result for an infected patient is too low.

The tests being described by these experts are similar to the unreliable tests that I have been using as an example.

How is the containment of COVID-19 progressing?

Releasing or not accepting patients with a negative result

“Some patients whose CT scans clearly showed signs of viral infection but whose NATs tested negative were “released” back into their communities due to a shortage of hospital berths.” — anonymous doctor in the imaging department of a major Wuhan hospital.

(Source: Caixin Media)

“Though her doctor was almost certain she was infected with the virus, a throat-swab test she had taken came back negative, which meant the facility wouldn’t take her.”

(Source: Wall Street Journal)

While these events are driven more by a lack of resources, it is sad to see patients being rejected from the hospital because of low-sensitivity tests.

Ordering multiple tests, not releasing patients with a negative result

Wuhan’s top Communist Party official, Ma Guoqiang, cast doubt on the swab test on Monday, urging those with negative results to try again to be sure. “One needs to wait one day later, to get another negative, in order to rule out [the virus],” he said.

(Source: Wall Street Journal)

“Her clinical features are very typical of the coronavirus infection. We didn’t discharge her, because we still have enough beds and can’t run the risk of letting her infect someone else. And on the fifth try, the result finally came back positive.” — Bin Song, director of the radiology department at Huaxi Hospital in Chengdu.

(Source: Interview with Al Jazeera)

While multiple tests can provide confidence if a positive result is eventually reached, it has its costs, and the patients should have been quarantined and treated either way (since disconfirmation is nearly impossible).

Do tests even add any valuable information for Wuhan patients?

For a 40% sensitive, 90% specific test, we just saw that it takes 8 consecutive negative results to reduce a 50% probability of infection to less than 5% (where very low probabilities are required to reduce the risk of contagion). We also saw that it takes 2 consecutive negative results to reduce 8% to less than 5%. What is the probability of a person in Wuhan being infected, if they:

• Have already shown signs in a CT scan? (surely >50% probability?)
• Have a cough/fever? (surely >8% probability?)
• Are related to someone who has been infected?
• Have entered any hospital?

It seems like testing for the purposes of ruling out infection is not a feasible solution in an active outbreak zone. There are simply not enough resources to test everyone multiple times.

This may partially explain why…

… Authorities have already implemented large-scale containment measures — they assume that everyone is, or will be infected.

Mandatory quarantine.

Wartime controls

Zhangwan district in the city of Shiyan, Hubei, implements wartime controls. Residents are not allowed to enter or leave buildings, with the exception of medical workers and a few other workers with critical roles.

Of course, unreliable diagnostic testing isn’t the only reason for all of these events, nor is it even the most important reason. There are plenty of reasons why many of these decisions were made.

In terms of official policy and guidelines…

The NHC changes diagnostic criteria to no longer require lab tests

Given that low sensitivity tests cannot rule out infections and infectious people are dangerous to the public, the following news makes sense.

The National Health Commision changed the diagnostic criteria to add clinically confirmed cases as a category — cases which do not require a positive rest result, but only suspected cases with the imaging features of pneumonia.

The CDC does not even bother to test evacuees from Wuhan

The Centers for Disease Control and Prevention in the US did not even bother to give diagnostic tests to the evacuees who escaped Wuhan. We don’t know if they are using the same test which is used in Wuhan, but we know that the test is not sensitive.

“The test for the coronavirus often results in a false negative if given too early, when the person is infected but not yet showing symptoms.”

“A false negative could provide a false sense of security” — Dr. Christopher Braden, CDC official.

(Source: CNN)

The CDC knows that negative results do not add any useful information, and positive results will not have any effect on decision making. Evacuees will be kept in quarantine regardless of results, and sent to the hospital if symptoms develop.

Has the sensitivity problem been solved?

Feb 7: a Xinhua article claims that Wuhan is able to test 4,000–5,000 samples per day.

Feb 9: A New York Times report claims that Hubei has the capability of performing 6,000 tests per day.

“The local health department says the labs can run 6,000 tests a day.”

Source: New York Times, Feb 9

We have information on:

• Potential number of diagnostic tests
• Reported number of confirmed cases

We want to consider:

• What is the true number of infections (assuming confirmed cases require positive results)?
• What is the true number of diagnostic tests?
• What is the sensitivity of the diagnostic tests (has it been improved)?

We assume that they reserve the tests for the most obvious/severe cases — cases which have gone through many rounds of assessment in the clinic, radiology, etc. This justifies my next assumption that a very large proportion of the tested people are going to be infected (>90%). This assumption remains while the number of suspected cases/clinical confirmations is still significant.

Wuhan

Around the time of Feb 7, there are around 1,200–2,000 positive results per day.

Suppose that they can do 4,000 tests on 4,000 people, and the tests have 40% sensitivity, 90% specificity.

(1) Suppose all 4,000 are infected. Then on average, 1,600 people would test positive, which is exactly in line with the number of confirmed figures.

(2) Now suppose only 90% of tested people are infected — 3,600 are infected and 400 are not.

• Of 3,600 infected, 1,440 will test positive.
• Of the 400 not infected, 40 will test positive.
• In total, 1,480 will test positive. On the lower end, but in line with confirmed figures.
• An alternative is that there are more than 4000 tests per day.

(3) For 5000 tests a day and 90% infection, 4,500 people are infected.

• Of 4,500 infected, 1,800 test positive.
• Of 500 not infected, 50 test positive.
• In total, 1,850 people test positive. Another realistic figure.

Now suppose that the tests are sensitive: 90% sensitivity and specificity.

(4) Suppose 40% of 4000 tested people are infected — 1,600 infected, 2,400 not infected.

• Of 1,600 infected: 1440 test positive.
• Of 2,400 not infected: 240 test positive.
• In total, 1680 test positive, which is in line with confirmed figures.

Given that the article about test sensitivity was published Feb 8, the most realistic scenario is 90–100% infection rate among the tested, 30–50% sensitivity and 4000–5000 tests per day.

This seems to confirm that test sensitivity was too low at the time of February 7 — which justifies the authorities’ decision to add clinical confirmations.

Hubei

The data for Hubei around/before Feb 9 shows 2,000–3,000 new confirmed cases per day.

Suppose that they do 6,000 tests per day, with 40% sensitivity, 90% specificity.

(1) Suppose that 90% of people are infected — 5,400 infections.

• Of the 5,400 infected, 2,160 test positive.
• Of the 600 not infected, 60 test positive.
• In total, 2,220 test positive.

While this is in the range, it is on the lower end, so there may be more infections, more testing capabilities than reported, or possibly better test sensitivity (if other parts of Hubei use different tests from Wuhan).

Either way, it is clear that low sensitivity diagnostic tests are a major problem in this COVID-19 outbreak.