‘A Permanent Nightmare’: Pinterest Moderators Fight to Keep Horrifying Content Off the Platform

Moderators reported seeing child pornography content ‘every couple hours’

On a good day, Blake’s job as a Pinterest moderator meant clicking through an endless dragnet of fetish material, adult nudity, and “sexy lady” miscellany. Hide, delete, strike, or ban? These decisions took mere seconds. In fact, the pornography label was considered so straightforward that new hires were reportedly trained on it.

But one day in 2017, Blake discovered something horrifying. Within a network of sexual child images sat an innocently named Pinterest board. “Something like ‘My Pics,’ or ‘My Computer,’” Blake told OneZero. The board was set to secret, visible only to its owner. It contained photos of an infant and a toddler, “nothing violating in any sense, but photos you’d take if you were babysitting,” Blake says. But because the board orbited other accounts that posted child abuse content, Blake suspected these children were at risk of sexual exploitation. They escalated the issue to a manager, asking if Pinterest should investigate the user’s IP or report the photos to law enforcement.

But to their dismay, Blake said they were told that “no imminent threat” was present. “I remember being like, ‘What the fuck.’” The account was ultimately banned for violating Pinterest’s policy against inappropriate images of minors, yet “no external or even further internal actions were taken.”

OneZero spoke to three former Pinterest moderators, or trust and safety specialists, who worked at the company between 2016 and 2019. These individuals served as frontline workers, keeping harmful content at bay. According to their testimonies, they had inadequate resources and minimal mental health support. They worked full-time as contractors for Pinterest’s Trust and Safety team, earning between $25 and $32 per hour. All three signed nondisclosure agreements (NDAs) with Pinterest, and because of that, OneZero is referring to them using pseudonyms.

The pornography label was considered so straightforward that new hires were reportedly trained on it.

In an email to OneZero, a spokesperson for Pinterest declined to comment on the size of its current moderation team. “The majority of our moderators are Pinterest employees working in-house,” the spokesperson said. “We also work with an outsourced team.”

Since its founding in 2009, Pinterest has promoted itself as Silicon Valley’s nicest platform, one that credits Midwestern women, scrapbookers, and home designers as early adopters, according to the New York Times. Pinterest not only rejects the label of “social network”; it actively seeks to be a “safe and happy place for inspiration, self-improvement, and salted caramel cookie recipes,” CEO Ben Silbermann told the New York Times. When the Washington Post reported on political disinformation appearing on the platform, the company blamed Facebook as the origin point. Facebook, Twitter, and Google later testified before Congress about Russian meddling, but Pinterest was excluded from the summons.

Pinterest has garnered largely positive reviews from technology outlets, including OneZero, and its progressive values have buttressed a series of bold policy moves aimed at combating misinformation. In 2018, the company launched a search ban on fake medical advice, such as cancer cures, according to the Wall Street Journal. Last year, it removed hundreds of domains and groups that spread health misinformation and began serving verified medical facts in response to vaccine-related searches, reported The Guardian. And when civil liberty advocates criticized the company for glorifying “plantation weddings,” Pinterest started warning users that such images may be controversial, wrote BuzzFeed News. These deliberate choices to protect the platform’s health, sometimes at the cost of rapid growth, have helped the company frame itself as the “anti-Facebook.”

The company’s moderation strategies, however, have their flaws. This month, OneZero reported on harmful content that lives on the platform. That includes sexualized images of young girls tagged with a code name for child pornography, racist imagery, medical hoaxes, conspiracy theories, and hyperpartisan propaganda. Because Pinterest may hide problematic content rather than remove it entirely, these pins persisted on the website and on search engines such as Google, allowing OneZero to see what slipped through the cracks. Pinterest told OneZero that it relies on users to report offensive content and keeps the site clean using “proactive tools,” such as automated tracking technology, for images of child abuse.

But Pinterest’s moderation efforts aren’t exclusively defined by a set of guidelines and algorithms. Human moderators perform this work as well: determining the course of action for violating pins, investigating suspicious users and their networks, making tough decisions around “gray area” content, and protecting Pinterest’s reputation as a company that generally gets this stuff right. Moderation is often grueling, thankless labor that can traumatize workers. Only in recent years have companies like Facebook been forced to acknowledge the emotional impact of spending days reviewing content that depicts extreme violence, abuse, and harassment. “I was dealing with what couldn’t be automated,” said Jordan, a former Pinterest moderator.

Behind the scenes, Pinterest moderators ingest a huge amount of content. According to sources OneZero spoke to, as of 2018, a single worker could review up to 8,000 images per day, depending on the subject matter. Sources say Pinterest’s earliest moderators could choose areas of moderation to work on, which included pornography, hate speech, or graphic violence. Moderators used a tool that surfaced daily content reported by Pinterest users and detection algorithms. Using this tool, images appeared in a queue next to a series of hotkeys allowing moderators to take a variety of actions: hiding or deleting content and accounts, issuing a “strike” or penalty, escalating content to another Pinterest team, and banning a user altogether. Metadata also showed where an image came from or who repinned it, which helped moderators to build a broader picture of a user’s activity.

“We’ve implemented tool changes to give agents more control over their experience: allowing agents to set audio off by default, converting images to black & white from color (which reduces the realism), allowing agents to turn off auto-play video, deduplicating images to reduce exposure of violating content to agents, etc.,” Pinterest’s spokesperson said. “We’re improving and expanding our machine learning models to reduce the burden on agents when possible.”

Moderators told OneZero that internal detection tools were sometimes flawed, making their jobs harder. Sometimes algorithms failed to infer context, and a board of classical art would be reported for nudity. Harmless images were occasionally mistaken for child pornography. “There was an acceptable false-positive rate internally, and the automation was never perfect,” Jordan said.

“I was dealing with what couldn’t be automated,” said Jordan, a former Pinterest moderator.

Like Facebook and Twitter, Pinterest uses image hashing to create digital fingerprints for child abuse photos. These are strings of characters that can be cross-checked against a database of violating hashes run by the National Center for Missing and Exploited Children. Matches are removed immediately, and their associated accounts are suspended, but moderators told OneZero that child pornography was widespread on the platform. “I’d run into it once every couple of hours,” Jordan said.

Moderators told OneZero that sexualized images of minors were uniquely traumatizing. These images weren’t always explicit — sometimes they resembled catalogue photos and required moderators to contextualize them within the overall behavior of a user. “We had so many boards where people thought they were being smart by calling them ‘Summer Season,’ but would contain hundreds of images of little girls dressed in bikinis,” Blake said. “We knew how to determine whether it was moms looking at these for their kids or fucking perverts.”

When child abuse was detected, moderators would escalate the images to dedicated staff who’d report them to law enforcement. At one point, the Trust and Safety team reportedly kept a map with pins showing the locations of arrests and convictions stemming from their work.

More frequently, though, moderators said they struggled to address “edge cases” that could reasonably be hidden or deleted according to Pinterest’s rules. While the platform’s guidelines are clear on what it may “remove, limit, or block the distribution of,” moderators told OneZero that enforcement of these actions was largely arbitrary.

Topics like harassment and political content were notoriously challenging, as they could be steeped in context that needed unpacking. A board of World War II photos could arguably exist for multiple, conflicting reasons. So, to assess Nazi-related imagery, moderators were reportedly asked to interrogate whether something affirmed hateful ideology or issued a call to action. Other cases required moderators to consider people’s unconventional interests. One such case involved a board called “Angels,” which contained photos of children who had died. The board became a moderation issue when someone contacted Pinterest because their own child was included among the images, and they wanted them removed.

“Certain days, I’d go in and say, ‘I’m not in the fucking mood right now — delete delete delete,’” Blake recalled. “I’d ask supervisors how we made decisions around deleting or hiding a board, and they’d say, ‘I don’t know, go into the board, scroll down maybe twice, and if you’ve counted 10 violating images, then delete it.’ It was so nebulous, so up to my own judgment.”

Pinterest’s Policy team worked closely with moderators to create and refine rules around problematic content. This relationship included exercises meant to clarify when something should be hidden or removed. One exercise, related to gore, used the example of medical students creating boards of graphic, bloody images in preparation for an exam. Moderators would allow these users to keep their boards but would hide them from search and recommendation results on the platform.

Moderators told OneZero that Pinterest leaned heavily on the hide option. “They would push hide as a solution,” Jordan said. “I don’t know if that was intentionally sinister. Everything seemed like a tightrope, and the perfect decision seemed out of reach.”

Pinterest told OneZero that, currently, moderators complete a seven to ten week training process to familiarize themselves with the company’s content policies. According to Pinterest, moderators also undergo routine alignment checks, one-on-ones with managers, and weekly policy sessions to ensure they’re making “consistent and correct decisions.”

Within the company itself, moderators told OneZero they felt like “second-class citizens,” occupying a vital yet deliberately invisible role. Sometimes, this manifested in physical ways: According to sources OneZero spoke to, as of 2019, moderators were stashed away in a corner of the company’s San Francisco headquarters, with their computers facing the wall so the images they viewed would remain unseen by other employees. Though it’s not uncommon for platforms to silo their moderation teams (Facebook, for example, contracts this work off-site and has its own “war room” for on-site moderators), Pinterest moderators felt tangibly out of sight and out of mind.

“I would see something horrifying, stand up from my desk, and walk to get an espresso and some free cheese. Then I’d go out into the alley near a dumpster and smoke.”

“This [seating arrangement] created a vacuum of awareness and contributed to our underresourcing,” Jordan said.

The chasm between Pinterest’s cheerful brand and the heinous content on its platform never sat right with some moderators. Like many technology companies, Pinterest employees enjoy generous perks, such as catered food, free alcohol, comfy quiet rooms, and free arcade games.

“It was surreal,” Blake said. “I would see something horrifying, stand up from my desk, and walk to get an espresso and some free cheese. Then I’d go out into the alley near a dumpster and smoke.”

The moderators who OneZero spoke to were hired as contractors on one- to two-year contracts between 2016 and 2019 and worked on the Trust and Safety team. The limited time frame, Jordan says, was explained as a “sort of a benefit to us. Like, no one should be doing this job for a long amount of time.”

“We’ve hired fixed-term employees in the past to handle temporary increases in report volumes, respond to fixed-term review projects to help improve our machine learning models, and reduce response time for certain high-priority categories of reports,” Pinterest’s spokesperson told OneZero.

Early on, the content moderation team consisted largely of white women from the technology industry, sources told OneZero; later hires included recent college graduates with nontechnology backgrounds. Some moderators told OneZero that Pinterest was irresponsibly vague about the role of moderators during the interview process, making no mention of the gruesome material they’d be required to see. One moderator said they were only shown a board of swastikas and asked to identify which of the images violated Pinterest’s policies.

“We are transparent during the pre-hire and hiring process about content to which agents could potentially be exposed to and set realistic expectations regarding how much and what type of exposure can be expected in their role,” Pinterest’s spokesperson said. “We ensure that moderators aren’t assigned a single type of content to review,” noting that moderators cycle through different queues weekly.

A third moderator recalled their team feeling left out during Pinterest’s IPO last year, when it was priced at a valuation of $10 billion. “The IPO millionaires are coming!” wrote the Wall Street Journal. Indeed, many Pinterest employees became richer that day, including CEO Silbermann, whose net worth hit an estimated $1.6 billion. According to this source, contract moderators received zero shares and could only watch as colleagues celebrated their newfound wealth. “Equity was not part of our packages,” said Alex, a former Pinterest moderator. “So it was like, ‘Great for everyone who has equity!’”

“All of our current moderators who are employees have received equity,” Pinterest’s spokesperson told OneZero. “We do not offer equity to contractors or fixed-term employees.”

Moderators told OneZero that Pinterest did provide them with full benefits. But, they pointed out, mental health support was inadequate given the disturbing nature of their work. As of last year, moderators said they had access to 30-minute sessions with a therapist who visited Pinterest’s headquarters once every six weeks. They also reportedly received a monthly credit for massages through the on-demand massage startup Zeel.

Pinterest told OneZero that moderators have had access to a dedicated wellness program since 2014, and that it continues to evolve its offerings. These benefits currently include “onsite and remote counseling, options for post-employment counseling, training and speakers on mindfulness, self-care resources, resiliency and other wellness topics.”

“At Pinterest, our content moderators receive a range of specialized training because we don’t expect people to come in off the street and immediately recognize coded self harm language or historical hate symbols,” Pinterest’s spokesperson said. “If a moderator discovers that they’re struggling with a particular subject, we take them off that workflow no questions asked, and if they decide that this job is no longer for them, we partner with them to do non-content-oriented projects to give them time to find a new position within or outside Pinterest.”

Social platforms have yet to fully reckon with the psychological damage inflicted on their moderators, but the tide is slowly changing. Earlier this year, Facebook agreed to pay $52 million to moderators who developed post-traumatic stress disorder and other mental health issues on the job, reported The Verge.

Pinterest’s Trust and Safety team has always been small — just 11 people in 2018, according to The Atlantic — and their shared trauma fostered a sense of closeness. Moderators described seeking catharsis through gallows humor — using a Slack channel to post “darkly heinous things that were funny,” Blake said. They’d eat lunch, do adult coloring books, and occasionally bake treats to share. A group of moderators would frequently go for walks at a flower market near the office. Some of these friendships outlasted Pinterest’s employment contracts.

None of the moderators OneZero spoke with harbor ill will against Pinterest, but they wish the company had listened to them more. “There needs to be less division between the tech side and content side,” Alex said. They share the belief that Pinterest’s moderation team was undervalued to varying degrees. This chronic disregard — or “Faustian neglect in exchange for aggressive growth,” according to Blake — has arguably degraded the health of Pinterest’s platform. The question also remains as to whether social platforms can ever truly be healthy and what percentage of moderation should fall on humans versus algorithms.

“For me, I think it really comes down to scale and the actual DNA of the product itself,” Jordan said. “As the thing grows, if your goal as a product is to get anybody and anyone onboard and monetize that… a discovery platform will be a permanent nightmare.”

Have a tip about Pinterest or moderation at a tech platform? You can contact Sarah Emerson securely from a non-work device on Signal at +1 510-473-8820 or email [email protected].