Facebook Knew Its Algorithm Was Radicalising People. It Chose Engagement Anyway.
On a morning in May 2021, Frances Haugen walked out of Facebook's offices for the last time. She had
On a morning in May 2021, Frances Haugen walked out of Facebook’s offices for the last time. She had spent the previous months doing something unusual for a Silicon Valley employee: quietly photographing internal documents. Thousands of pages of research, memos, and presentations — studies Facebook had commissioned, received, and largely ignored. She gave the evidence to the Wall Street Journal, to Congress, and to the Securities and Exchange Commission.
What the documents showed was not complicated. Facebook knew its algorithm was amplifying hate, extremism, and misinformation. It had known for years. It kept doing it because stopping would hurt engagement, and engagement drove advertising revenue.
That is the whole story. Everything else is detail.
The Algorithm Was Not an Accident
Facebook’s feed algorithm does not show users what their friends posted in chronological order. It shows what the algorithm predicts will keep them on the platform longest — ranking content by the likelihood of a reaction, share, or comment. These are the metrics it optimises for.
The problem is that the content generating the most reactions is not the most truthful, most useful, or most socially constructive. It is the content that makes people angry.
An internal presentation from 2016, reviewed by the Wall Street Journal and cited in Haugen’s testimony, showed that researcher Monica Lee had found Facebook was not only hosting extremist groups but actively promoting them. The presentation concluded that 64 percent of all extremist group joins were attributable to Facebook’s own recommendation tools — predominantly the Groups You Should Join and Discover features. The figure comes from a single internal presentation and has not been independently verified, but it was Facebook’s own assessment of its own systems.
The algorithm was identifying users with adjacent interests and nudging them toward increasingly extreme communities, automatically and at scale, because those groups had high engagement rates. It was not designed to radicalise anyone. It was designed to maximise time on platform. Radicalisation was a reliable side effect.
In 2017, Chris Cox, Facebook’s then chief product officer, formed a task force to examine whether maximising engagement was contributing to political polarisation. It was. Addressing the problem, the task force found, would produce a measurable decline in engagement. Facebook did not make the change.
The European Parties Who Complained
By 2019, the algorithm’s effects had become visible at the level of democratic governance. Major political parties across Europe filed a formal complaint with Facebook about the impact of its recommendation systems on political discourse.
An internal report obtained by Haugen documented their position: the parties said they felt strongly compelled to skew negative in their Facebook communications, and that this had pushed them toward more extreme policy positions. They knew those positions were harmful. They also knew that moderation would cost them reach in an environment Facebook had designed.
European political parties were telling Facebook directly that its platform was distorting how they governed. Facebook received the complaint, filed it, and made no meaningful change.
What Happened Before the 2020 Election
Ahead of the November 2020 US presidential election, Facebook implemented what its internal documents termed safety systems — algorithmic modifications designed to reduce amplification of inflammatory content and slow the spread of misinformation. The systems worked. Engagement fell.
After the election result was called, Facebook disabled them. According to Haugen’s testimony and supporting documents, the safety systems were rolled back because reduced engagement meant reduced advertising revenue. The company wanted growth back.
In the weeks between the election and 20 January 2021, Facebook’s algorithm amplified content from accounts using the platform to organise political activity. Among the material circulating were posts with photographs of armed partisans and explicit language about restoring the republic by force.
On 6 January 2021, a crowd stormed the United States Capitol. Prosecutors cited Facebook posts as evidence in subsequent legal proceedings — not as proof that the platform caused the attack, but as documentation of how it was used to organise and inflame. The safety systems were switched back on after the attack.
Haugen told Congress: Facebook had changed those safety defaults in the run-up to the election because it knew they were dangerous. It reversed them when the election was over because it wanted the growth back. “That really feels like a betrayal of democracy,” she said.
Facebook’s response was that social media was not the primary cause of political polarisation, that the company had not prioritised profits over safety, and that it had made significant investments in safety infrastructure.
Myanmar: Where the Algorithm Contributed to a Genocide
In Myanmar, Facebook was not simply a platform — it was, for most of the country, the internet itself. Following a sharp reduction in SIM card costs in 2014, smartphone adoption accelerated rapidly. Handsets came pre-loaded with Facebook. Many businesses had no website, only a Facebook page. By 2016, estimates suggested that between 35 and 85 percent of Myanmar’s internet-connected population used the platform actively.
The Myanmar military had been seeding anti-Rohingya content on Facebook for years. Internal studies dating to 2012 indicated that Meta was aware its algorithms could produce serious real-world harm. In 2016, Facebook’s own research acknowledged that its recommendation systems were compounding the problem of extremism.
Amnesty International’s 2022 report concluded that Facebook’s algorithms had proactively amplified content inciting violent hatred against the Rohingya from as early as 2012. The military’s campaign had deep structural and historical roots that preceded Facebook’s presence in the country — but the platform, most human rights scholars argue, served as a significant accelerant of state-orchestrated violence that was already under way. Despite sustained warnings from civil society organisations, journalists, and its own staff, Facebook did not respond with anything approaching adequate urgency.
In 2014, the company had actually attempted to help. It published a sticker pack as part of a civil society anti-hate initiative, allowing users to post affirmative messages in response to violent content — slogans like Think before you share. According to reporting on the crisis, Facebook’s algorithm then registered the stickers as positive engagement signals and began promoting the content they were attached to. The tool designed to counter hate amplified it instead. This account has not been traced to a specific document in the Haugen files, but it has been reported consistently across multiple credible outlets covering Facebook’s role in Myanmar.
In 2017, the Myanmar military conducted what it described as clearance operations against the Rohingya. Hundreds of thousands of people were displaced; credible estimates place those killed in the thousands to tens of thousands, with rape and torture reported at systematic scale. The UN fact-finding mission in 2018 determined that Facebook had served as a useful instrument for vilifying the Rohingya. That same year, Facebook commissioned a human rights impact assessment. The report concluded that the company was not doing enough to prevent the platform from fomenting division and inciting violence.
Facebook acknowledged this. It made some changes. A former Meta employee told Amnesty International that the company treated lives in the Global South as categorically less worthy of attention than those in the United States and Europe. “If 1,000 people died in Myanmar tomorrow,” the employee said, “it is less important than if 10 people in Britain die.”
What Facebook Called “Harmful Non-Violating”
The documents Haugen copied included an internal classification that captures the company’s decision-making in four words: harmful non-violating.
This referred to content Facebook’s own researchers had identified as damaging — to users, to specific communities, to democratic institutions — but which did not technically breach the platform’s stated policies. Facebook neither removed it nor reduced its algorithmic amplification. The harm was documented. The content remained.
The category existed because removing policy-violating content was a defensible business decision with clear rules. Reducing the reach of harmful-but-compliant content would have required a values-based choice: accepting less engagement in exchange for less harm. That choice was not made.
The documents also captured what happened when employees raised concerns internally. In the days after January 6th, one wrote on an internal message board: Haven’t we had enough time to figure out how to manage discourse without enabling violence? An employee who responded supportively of the leadership received the reply: Welcome to day one.
The Testimony and What Came After
Frances Haugen testified before the US Senate on 5 October 2021. She was specific, documented, and composed. She told senators that Facebook had consistently chosen to maximise its own growth rather than implement safeguards on its platform, while concealing from the public and government officials internal research that illuminated the scale of the harm.
“The result has been more division, more harm, more lies, more threats and more combat. In some cases, this dangerous online talk has led to actual violence that harms and even kills people.”
Facebook’s market capitalisation fell by six billion dollars within 24 hours of her 60 Minutes interview. The company characterised the documents she had taken as stolen and suggested she may have broken the law. Federal whistleblower protections covered her disclosures to the SEC and Congress. The legal threat did not proceed.
More than 40 state attorneys general, hundreds of school districts, and dozens of individuals subsequently sued Meta over features alleged to have harmed children. The Federal Trade Commission moved to reopen its 2020 landmark five-billion-dollar settlement with the company, seeking to bar Meta from monetising children’s data.
In January 2025, Mark Zuckerberg announced the end of Meta’s third-party fact-checking programme, citing concerns about censorship. The specific legislative reform Haugen had urged Congress to pursue — amending Section 230 to create liability for algorithmic amplification — remains essentially unchanged.
The documents are part of the historical record. The algorithm has been modified — Meta has reduced political content recommendations and adjusted ranking signals since 2021 — but the underlying logic, maximising engagement above other considerations, has not been structurally displaced. The system is still running.
The Simple Version
The Facebook algorithm extremism problem was not a technical failure. The engineers understood what the algorithm was doing. Researchers documented it. Internal memos were written. European governments complained formally. Warnings about Myanmar were filed. Safety systems that demonstrably reduced polarisation were tested and found effective.
At every decision point, the company chose to continue.
Not because Facebook’s leadership wanted ethnic violence in Myanmar or a riot in Washington. But because the systems producing those outcomes also produced engagement, and engagement produced revenue, and revenue was what the company was built to maximise.
Outrage is more engaging than calm. Fear is more engaging than reason. Division is more engaging than agreement. The algorithm did not select for these things deliberately — the people who built it chose to optimise for engagement, and then declined to change course when the consequences became undeniable.
Facebook had the documents. It had the research. It had the formal complaints from governments. It had the bodies in Myanmar. It kept the algorithm running because the algorithm was working exactly as designed. That some of it has since been modified does not change what the choices were, or when they were made.
Sources
- CBS News — Facebook Whistleblower Frances Haugen Details Company’s Misleading Efforts on 60 Minutes, October 2021 https://www.cbsnews.com/news/facebook-whistleblower-frances-haugen-misinformation-public-60-minutes-2021-10-03/
- MIT Technology Review — Frances Haugen Says Facebook’s Algorithms Are Dangerous. Here’s Why, October 2021 https://www.technologyreview.com/2021/10/05/1036519/facebook-whistleblower-frances-haugen-algorithms/
- Amnesty International — Myanmar: Facebook’s Systems Promoted Violence Against Rohingya, September 2022 https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/
- NPR — Four Takeaways from Facebook Whistleblower Frances Haugen’s Testimony, October 2021 https://www.npr.org/2021/10/05/1043377310/facebook-whistleblower-frances-haugen-congress
