How Science Best Serves Society
Science is most effective when it is given the most amount of room to be wrong
In November of last year, a novel virus emerged in China and soon spread across the globe. In the early days of the pandemic little was known about the virus. Even today, despite an unprecedented fever of scientific activity and attention, precious little data has been able to shed much light on the situation. This hasn’t stopped science and health professionals from issuing public health recommendations. We’ve seen pandemics before and know how bad things can get. There was no time to wait for all the data to get in; we had decisions to make and a limited window of opportunity to make them in before any public health measure would be useless.
Given the rapidity of the virus’s spread, we made the best predictions and models using the limited data we had, along with some previous experience from other viral epidemics. We were obviously going to be wrong about a lot of things.
Now that we are a few months wiser, there is an important question we need to ask ourselves as scientists and public health officials: Are we willing to be wrong?
In theory, science and medicine are unbiased disciplines of exploration driven by evidence and objectivity. We ask questions, develop hypotheses, and design experiments to tease out a small piece of insight about our complex universe. However, science does not operate in a vacuum, nor is it carried out by robots. Science is a human endeavor — a noble endeavor to be sure, but one that faces the same human limitations and biases that other human-driven industries experience. Our finitude ensures mistakes will come.
One of science’s greatest virtues, however, is its self-correcting nature.
Scientists and medical professionals have held many incorrect, even inconceivable (from our vantage point today), positions throughout history. Yet when new evidence is unearthed that challenges the prevailing paradigm, corrections are made and paradigms are shifted in light of the new observations or analyses — though not without resistance, typically from well-established scientists, to be sure. In fact, one of the hallmarks of a young scientist is her desire to discover something that rocks the prevailing paradigm. This is how science works and why it has made many advances in health, technology, and how we understand the world.
As a scientist, I know firsthand the frustration that comes when conspiracy theorists and those who oppose mainstream science (without an evidential basis) loudly dismiss the conclusions of scientists who have devoted their whole life to studying their area of interest. I know what it feels like to have your reputation, even your career, riding on being right. And I know what it feels like to be wrong.
But being wrong is part of the process. Though it stings, public acknowledgement that scientists got something wrong is the very element that enables us to correct course. This doesn’t make science unreliable — it is the very aspect that sustains its reliability. Disagreement, discussion, and debate are indications of healthy science. And you don’t get these without an intellectual culture in which mistakes are reckoned with and improvements made.
The problem is science doesn’t always proceed this way. Sometimes, pride and politics threaten intellectual honesty and scientific rigor. With the coronavirus pandemic, for example, science and medical professionals have made public statements and predictions on a very public issue, and this has led to standing by these statements and predictions even when there turns out to be little evidence to back up the positions (or, worse, evidence to the contrary).
When things get political, scientific exploration and self-correction are hindered as public opinion and perception begin to play key roles in shaping our conclusions. We start to see what we want to see and choose to use rhetoric over empirical data to justify our positions.
I want to highlight three issues relating to the pandemic that show the negative effects of pride and politics on scientific exploration. I am not arguing for or against any of these as public health measures, but rather showing that the possibility of being wrong is quickly becoming an enormous intellectual liability for scientists and public health officials. And, as I mentioned, this is ruinous to the scientific enterprise.
The Question of Asymptomatic Spread
In a press briefing on Monday, June 8th, the World Health Organization’s (WHO) technical lead for coronavirus said: “From the data we have, it still seems to be rare that an asymptomatic person actually transmits onward to a secondary individual.” Almost immediately there was a backlash on social media and from other public health professionals, leading to the WHO backtracking on their prior statement the very next day. What’s the problem here?
The original statement was based on an honest review of the literature, and the reason for backtracking was not because their expert opinion about asymptomatic spread had changed, but rather because of the backlash generated by its disclosure. In fact, in their June interim guidance, the same findings are said even more explicitly:
Comprehensive studies on transmission from asymptomatic individuals are difficult to conduct, but the available evidence from contact tracing reported by Member States suggests that asymptomatically-infected individuals are much less likely to transmit the virus than those who develop symptoms.
Why is the question of asymptomatic spread so controversial? Because so much public policy has been decided, at least in part, on the basis of the probability of asymptomatic spread. Why should everyone wear face masks? Because we don’t know who has the virus but is asymptomatic. Why should we close the schools? At least partially because we don’t know who has the virus but is asymptomatic. Why the strict lockdown? Partially because we don’t know who has the virus but is asymptomatic.
It has been a central element in designing our social guidelines. So any softening of our position on the threat of asymptomatic spread is understandably poorly received.
To be clear, I am not disputing that asymptomatic spread is possible. We know there are asymptomatic patients, and thus it is reasonable to hypothesize that viral transmission can happen via these asymptomatic patients. The point I’m making is that the passionate reactions, even from scientists and doctors, to the news about the unlikelihood of asymptomatic spread are not based on experimental or even an abundance of observational evidence.
The literature on the asymptomatic spread of COVID-19 is interesting. Most of the papers I looked at simply assume that asymptomatic carriers play a large role in virus transmission, conflate asymptomatic and presymptomatic spread (though they are different), base their opinion on models rather than observational evidence, or cite other papers to show that asymptomatic spread has been documented.
This last point is particularly troubling because some papers imply that many cases of asymptomatic spread have been clearly documented but then cite this paper, which only gives disputable evidence that suggests asymptomatic spread happened in a particular family cluster.
The hypothesis of asymptomatic transmission of COVID-19 based on the evidence of asymptomatic carriers of the virus is a good hypothesis, and it was reasonable to initially base public policy on this hypothesis. However, the more data we collect, the more disputable this hypothesis becomes.
It is possible that the hypothesis is overstated (i.e., asymptomatic spread is rare) or that it is simply wrong. Are we willing to be wrong?
The Question of Masks
Similarly, the question of compelling the healthy public to wear face masks to keep the spread of the virus at bay is currently a prominent talking point for many public health officials.
Many assert that it is a scientific fact that face masks reduce transmission. But is there direct evidence to back this up? From the WHO interim guidance:
At present, there is no direct evidence (from studies on COVID-19 and in healthy people in the community) on the effectiveness of universal masking of healthy people in the community to prevent infection with respiratory viruses, including COVID-19.
But that doesn’t stop people from asserting that face masks in public are essential to slowing transmission. I’ve even seen images such as this one going around on social media that make specific claims about the risk reduction percentage when wearing face masks without any evidential basis (or cited sources).
The issue here again begins with a somewhat reasonable hypothesis: If everyone wears a mask, asymptomatic (or presymptomatic) carriers of the virus will be less likely to transmit the virus due to their face mask barrier. But somehow this hypothesis graduated to scientific fact without any direct evidence.
As we saw above, it is not even clear that asymptomatic carriers transmit the virus very often. Beyond this, cloth face masks (the face masks that are usually recommended for the public) are only good at blocking large droplets that we expel, such as when we cough or sneeze. However, they are not efficient at blocking smaller particles that may contain viral load. In fact, at least one study done in 2015 indicated that not only are cloth masks (in a clinical setting) ineffective for preventing influenza-like illnesses, they actually increased the likelihood of infection due to moisture retention, reuse, and poor filtration.
Official guidance on the public wearing face masks has been inconsistent, probably due to the lack of evidence of their effectiveness in healthy populations. Do these cloth masks afford any protection? Maybe, especially if it stops people from coughing on you. But then again, if they are coughing, then it is likely that they are symptomatic, and a better prevention measure would be for the symptomatic person to stay home (or wear a mask if they have to go out in public).
I'm not arguing that face masks have no value or that the guidance of the public wearing of face masks is unreasonable. I am arguing that it is not based on direct evidence. It is possible that the hypothesis that they afford any significant protection could be wrong. Are we willing to be wrong?
Examples of Scientific Confirmation Bias
Finally, I want to briefly touch on some examples of confirmation bias in the scientific literature. I have already mentioned the tendency for journal articles to overstate the effects of asymptomatic transmission to back up their claims for modeling and the recommendation of face masks. These misleading claims were likely unintentional cases of confirmation bias. The researchers already had an answer in mind, then they looked for a paper to confirm their answer and cited it with bold rhetoric.
An example can be seen in this recent paper and a media write-up about the study in which one of the paper’s authors states:
Our study establishes very clearly that using a face mask is not only useful to prevent infected coughing droplets from reaching uninfected persons, but is also crucial for these uninfected persons to avoid breathing the minute atmospheric particles (aerosols) that infected people emit when talking and that can remain in the atmosphere tens of minutes and can travel tens of feet.
What’s the problem here?
The paper does not clearly establish what they claim it does. If anything, Figure 3 suggests that the rate of infection was on the decline before the face mask mandate was made. There are many assumptions that the authors make, such as total compliance, appropriate mask usage, and that there were no confounding variables. They start measuring the effectiveness of face masks immediately after the mandate is given, even though it should take at least two weeks to see any effects due to the incubation period of the virus.
Further, they assume that any deviation from the projected infection rate (Figure 2) is directly from a specific public health measure, when it could just as well be that the virus is following its natural transmission dynamics (see Farr’s law) and would have deviated from the projected line with or without any public health measure.
What is clear, actually, is that the data presented does not make a clear case for public mask usage, whether or not masks are effective. It is ironic that the authors end their abstract with this statement: “Our work also highlights the necessity that sound science is essential in decision-making for the current and future public health pandemics.”
After publication, Kate Grabowski, a Johns Hopkins epidemiologist, noted on Twitter that three of her fellow faculty members requested that the Proceedings of the National Academy of Sciences of the United States of America (PNAS) retract this paper. As I noted above, this, too, is science: the public reckoning with bad forms of it.
Perhaps an even more striking case of confirmation bias can be seen in this article recently published in Nature, one of the most respected research journals. The authors argue that the lockdown measures which were taken in several European countries “had a large effect on reducing transmission” of the virus.
I’m all for data analysis to evaluate the effects of public health measures. However, we should evaluate all claims — especially the ones that essentially contend, “the data say our predictions and interventions were correct.” The data may well say that, but skepticism toward claims of this sort is both healthy and warranted.
I find two major flaws in the paper.
The first is that the authors use a model that incorporates assumptions — such as transmission rate and the effectiveness of public health interventions — that are hard to estimate and subject to bias. At least they state this upfront in the abstract.
The bigger issue with the paper, however, is that they did not use a control.
The researchers assume their model correctly estimates the deaths that would have happened if the lockdown had not been implemented, then compare what they predicted would have happened to the actual data for death rates from COVID-19. Showing that many more deaths occurred in the simulation than those that happened in reality, they confidently assert that the lockdown had a major positive effect in slowing viral transmission.
The problem is, they actually did have a control in the data they presented, but seemed to have overlooked it. To be clear, their model (as shown in the right-hand graphs in Figure 1) predicted that nothing less than a complete lockdown would have any significant effect on viral transmission. So, how can we test this claim? Well, we look at a country that never implemented a complete lockdown: Sweden. Interestingly, the article includes Sweden in the analysis (see Extended Data Figure 1 and Extended Data Table 1). What does the model predict when no interventions are implemented? 28,000 deaths up to May 4th. How many actual deaths did Sweden have? 2,769. That’s an order of magnitude off. Interestingly, their model for how many deaths predicted in the presence of public health interventions is 2,800 — spot on for the actual number of deaths (even though Sweden didn’t enforce a lockdown). Maybe this is why they didn’t notice the discrepancy in the “no intervention” model — the model that would correspond most closely with Sweden’s approach.
I am not suggesting these scientists are trying to do bad science, but rather that confirmation bias is a powerful blinder, even in science. And when the issue has a political nature to it (such as “Did we make the right decision when we locked down the country”), there is an enormous pressure to show that you were right. But what if we were wrong? Are we willing to be wrong?
I’m not arguing that we should or shouldn’t have taken a particular measure recommended by experts in response to this virus. In fact, I think most of the measures and precautions were very reasonable based on the limited data we had. What I am arguing is that for science to thrive in uncovering the deep secrets of our complex world, we have to be willing to be wrong, and be given the space to be wrong.
Distorting factors — like political values and points of pride — can easily take away this vital aspect of the scientific endeavor, relaxing scientific rigor, encouraging confirmation bias, and ultimately leading to less public trust of scientific claims. Even (or perhaps especially) in times of intense pressure and public scrutiny, we need to strive for intellectual honesty and the humility to admit when we may be wrong. Society depends on it.
