The Dystopia Where Nothing Gets Cancelled

We’re creating a marketplace of disinformation

The 1993 film Demolition Man imagines a future given over to surveillance, cultural conformity, and social control. There is even a literal “speech police” — an A.I. that automatically gives people tickets for using foul language. Some who don’t want to be subject to such restrictive controls live underground in order to preserve their freedom of speech and choice.

At one point in the film, leading rebel Edgar Friendly, played by Dennis Leary, mounts a passionate monologue defending personal liberty. He says “I want high cholesterol. I want to eat bacon, butter, and buckets of cheese, okay? I want to smoke a Cuban cigar the size of Cincinnati in a nonsmoking section. I wanna run through the streets naked with green Jello all over my body reading Playboy magazine. Why? Because I suddenly might feel the need to.”

Many of the most popular dystopias in science fiction are concerned with the tyranny of control, surveillance, and censorship. From 1984 to Minority Report and even to Star Wars, creators imagine that dystopia will result from powerful oligarchies restricting freedoms of speech, thought, and association.

In America, a libertarian impulse for unfettered free speech and choice is, of course, one of the most influential pillars of our politics. Discourse about censorship is longstanding, but modern technologies have lately increased its resonance. Arguments have flared up over topics like cancel culture, Google’s alleged bias against conservatives, and whether Twitter should have temporarily restricted Donald Trump Jr.’s account after he posted a video claiming masks are unnecessary. These issues are muddy, but the responses share one thing in common: People across the political spectrum tend to assert the merits of an unrestricted marketplace of ideas.

But what if dystopia could also come from openness?

It’s a counterintuitive (and even blasphemous, for some) notion since freedom is foundational in maintaining a liberal society. The reflex of many Americans is to defend it as an a priori value. But if Americans don’t wrestle with dystopias of openness, we may unwittingly create something more akin to a marketplace of disinformation — a culture of perpetual discussion where it becomes less and less likely that harmful ideas can ever be successfully debated out of society. In such cacophony, disinformation not only spreads but is incentivized.

One argument for the marketplace of ideas is that good ideas will eventually win out. The best evidence will ostensibly settle debates and become facts. In a way, the power of the marketplace of ideas is held to be that it “cancels” bad information. But in a radically open society, especially one that’s as technologically accelerated as ours, there’s also a risk that bad ideas — conspiracy theories, pseudoscience, eugenics, and more — are amplified and adopted by a growing number of people.

Every new communication technology has sparked similar conversations. Whether it’s television, radio, or Twitter, technologies carry both gifts and curses. To quote cultural theorist Paul Virilio, “When you invent the ship, you also invent the shipwreck.”

When extolling the virtues of the marketplace of ideas, we should consider how markets actually work.

What’s different about today is that through digital technologies like the internet and social media, bad ideas now have both a rapid and global reach. Though tech creates the potential for the spread of good ideas, the worst ideas thrive when commodified online.

Metrics such as views and shares give ideas worth, regardless of their veracity. And the more an idea is “worth,” the harder it is to push it out of the marketplace. Algorithms promote the content with the most engagement. And since the content that gets the most engagement is that which triggers strong emotions, social media algorithms amplify arguments that provoke rage and sow division. This creates competing interests with tech companies, between penalizing free speech that hurts society and rewarding free speech that engages users.

For example, anti-vaccination content shared on Facebook could make it harder for public health officials to contain the spread of Covid-19. Health misinformation has received almost four times as many Facebook views as information from reliable sources. The site’s algorithms are reliably steering users toward disinformation.

The possibility for bad actors to profit from the spread of bad ideas is only expanding. Since users can retain anonymity on large, free platforms, the entry costs (both economic and social) of dealing in bad ideas are low and decreasing. Aided by echo-chamber effects, bad actors, who can hide behind the plausible deniability of “censorship,” not only prosper but can find each other and form coalitions that have real effects on society. A gunman fired a rifle inside a Washington, D.C., pizzeria after reading false claims of children trapped there in a sex-slave ring; U.S. political candidates openly support the QAnon conspiracy theory; and military officials in Myanmar used Facebook to promote hate speech against the Rohingya Muslim minority. Those who argue that there have always been conspiracy theorists or that these instances are the regrettable costs of freedom are not taking the problem and its complexity seriously.

James Baldwin once said, “a complex thing cannot be made simple.” By having a more nuanced discussion, we can consider the concerns of both free speech advocates and tech critics who call for more content moderation. When extolling the virtues of the marketplace of ideas, we should consider how markets actually work — particularly in the context of social media. And though it goes against the cyberlibertarian impulse of Silicon Valley, those engaged in discourse about open debate need to do the hard and uncomfortable but more foundational work of negotiating where the boundaries are.

But in order to get there, we need to consider that while we fret over “cancel culture,” we could find ourselves disinformed into dystopia. And that could be a science-fiction dystopia even worse than the ones we tend to imagine.

This piece contains edits from Amy Nordum of MIT Technology Review