Five Stages of Content Moderation

Substack is the latest company to discover that its users don’t want to share a platform with Nazis. 

In November, The Atlantic ran a story by Substack writer Jonathan Katz pointing out that a significant number of Nazis and other white nationalists were not only publishing on the platform but also profiting from it with paid subscriptions. In early December, a large number of prominent Substackers posted an open letter asking the company: “Why are you platforming and monetizing Nazis?” 

In late December, Substack responded with what is now a familiar and vapid argument about the virtues of free speech. The argument was weightless for several reasons. First, free speech means that you can’t be jailed for speech, not that you are guaranteed access to any given platform. Second, as Substack author Ken White noted, Substack already “censors” content. The platform has a hate-speech policy, an anti-doxxing policy and an anti-pornography policy. As White notes, “we’ve established what Substack is, now we’re just haggling over the price.” Finally, Substack’s co-founder argued that deplatforming Nazis “doesn’t make the problem go away—it makes it worse.” It’s not at all clear what problem he thinks is made worse by deplatforming Nazis, but there’s a mountain of evidence, still growing, that deplatforming works. 

Those who do not learn from history are condemned to repeat it, and we’re not talking about ancient history here. Most of the learning has taken place over the last decade, as one company after another goes through the five stages of content moderation: 

1. Denial: “Because our platform is inherently good, we don’t have any need for robust content moderation.”
2. Free-speech mouth noises: “Despite our platform being inherently good, we can see some problems, but our unwavering commitment to free speech precludes us from taking any robust action to solve the problems.”
3. Bargaining: “Despite our unwavering commitment to free speech, we acknowledge that we still have a problem, a big one, and we need to pacify users/activists/regulators/shareholders. We’re still not engaging in any meaningful moderation, but can we interest you in this one-time gesture? Perhaps a statement about our desire to humbly listen and learn?” 
4. Inadequate token effort: “We’ve taken down all the beheading videos, and from now on, we’ll take beheading videos almost as seriously as we take DMCA violations. Everything else remains the same.” 
5. Acceptance: “We acknowledge we are part of society, and society has rules, and henceforth we’re going to make a semi-credible effort to moderate content in such a way that most of these bad headlines finally go away—which everyone knows is the most important thing.”

Substack this week entered stage three—bargaining—after being confronted by one of its biggest and most influential publishers, Platformer. Platformer ran a big story on the Nazi issue last week and threatened to take its business elsewhere. As a result, Substack agreed to throw a couple of high-profile Nazi accounts into the volcano in order to appease the Cancel Gods. This is an act of bargaining because the company told Platformer it’s not changing its policies—it’s essentially a one-off gesture, designed to make the bad headlines go away. “If we do this, will you stop?”

In some ways, the fact that Substack has advanced to the bargaining stage offers a glimmer of hope. After Elon Musk gutted Twitter’s Trust and Safety department, a number of other tech companies followed suit with mass layoffs. For a while, it looked like Silicon Valley as a whole was planning to set the clock back to 2012. 

One day after its much-touted removal of a handful of Nazi accounts, Substack author and terrorism financing expert Jess Davis published an expose showing how 75 Nazi and extremist newsletters are making serious money on the platform, possibly millions of dollars per year. There’s no guarantee that Substack will advance to stage four, but one thing is certain: The bad headlines have only just begun.