This post is part 2 in a series about automated content moderation. Read the first post here.

When whistleblower Frances Haugen leaked a set of documents from Meta in 2020, among the revelations was a jarring statistic: The company’s algorithms designed to detect terrorist content incorrectly deleted nonviolent Arabic-language content 77 percent of the time, while failing to detect hate speech under the company’s own policies in many instances. Meta’s own transparency report released later that year demonstrated similar findings. Five years later, researchers in the region report that overzealous moderation remains a problem, while paths to remedy have all but collapsed.

Where these systems are faltering in Arabic, they’re positively failing in less-resourced languages. As a 2025 report from the Center for Democracy and Technology found, labeled datasets in certain languages and dialects such as Maghrebi Arabic and Kiswahili contain inconsistencies, bias, and inaccuracies due to the limited hiring of annotators who actually speak the languages as well as shifts in the languages themselves. An investigation into ChatGPT’s outputs in several low-resource languages demonstrates the depth of problem.

But language disparities are just one of several concerns as automated moderation becomes more widespread. From the systemic suppression of content from Palestine to the repeated misclassification of LGBTQ+ content as adult or explicit material, these varied examples demonstrate the risks of overreliance on automated moderation—and the need for stronger safeguards.

Transparency, Cultural Competence, Appeals

As we discussed in Part 1 of this series, automated systems can process content at a scale that humans never could, potentially enabling better moderation at scale and alleviating the psychological load on ill-paid moderators whose jobs require them to view incredibly disturbing content. But automated systems also reproduce existing biases, struggle to understand context, and often make mistakes that disproportionately affect journalists, activists, artists, and other vulnerable and marginalized communities.

As Rachel Griffin wrote in 2023, “Perfectly accurate moderation is not only technically out of reach but intrinsically impossible.” Despite those intrinsic flaws, there is a great deal companies, policymakers, and civil society can do to help ensure that highly-automated systems operate in ways that respect human rights, minimize predictable harms, and provide meaningful accountability when they fail. If companies are going to continue relying on automation to moderate users’ speech—and there is little reason to believe they won’t—then accountability must evolve alongside these technologies.

That evolution can start with committing to the Santa Clara Principles 2.0. These principles, first outlined in 2020 and re-launched in 2021 after substantial international input, reflect the needs and expectations of the global community and specifically address automation. The first Foundational Principle states:

Companies should ensure that human rights and due process considerations are integrated at all stages of the content moderation process, and should publish information outlining how this integration is made. Companies should only use automated processes to identify or remove content or suspend accounts, whether supplemented by human review or not, when there is sufficiently high confidence in the quality and accuracy of those processes. Companies should also provide users with clear and accessible methods of obtaining support in the event of content and account action. 

Drawing on the Santa Clara Principles 2.0, international human rights standards, and years of research documenting the shortcomings of automated moderation, we propose eight recommendations for policymakers thinking about regulation and companies deploying AI-assisted content moderation systems.

1. Automated technologies should help, not replace, human moderators. For example, automated systems can help flag and prioritize content for review, while humans can interpret context, handle sensitive cases, and refine system performance.
2. Companies must be transparent about when and how automation is used in content decisions.
3. Companies must regularly audit their automated systems for bias, with particular attention to low-resource languages, vulnerable and marginalized communities, and conflict zones.
4. Users must have the ability to appeal, and to provide context when they believe human or automated moderation decisions have wrongfully removed their content. Appeals should be promptly evaluated and decided by human moderators.
5. Companies should regularly assess the human rights impact of their moderation decisions, and issue public statements of the results
6. If they rely on third-party vendors, companies should carefully (and regularly) audit those vendors for compliance with these same principles
7. Lawmakers should avoid promoting and passing legislation that effectively or explicitly mandates automated moderation systems
8. Policymakers should also refrain from attempting to dictate platforms technical and design choices to favor or disfavor particular expression.

These recommendations understand that automated content moderation isn’t just a technical problem for clever engineers and product teams to solve. Because content moderation shapes public discourse and fundamental rights, its design and oversight must respond to the concerns of policymakers, civil society, independent researchers, and the communities most affected by these systems.

This is the second post in a 2-part series on automated content moderation. Read the first post here.

This blog post is part 1 of a 2-part series. The second part sets out recommendations for companies and policymakers.

Six years ago—one month into a global pandemic—we argued that the automated moderation processes many platforms were rapidly adopting should be highly transparent, easily appealable, and temporary. We warned that "protocols adopted in times of crisis often persist when the crisis is over."

That warning proved prescient. The use of automation and artificial intelligence (AI) to identify, flag, and moderate content has become the new norm—a permanent feature of how platforms govern speech online. In this two part series, we’re take stock of this new norm, and considering what platforms can and should do to ensure that AI serves online expression rather than stifling it.

A brief history of automated content moderation

From spam filtering and keyword blacklists to the hash-matching technologies used to identify child sexual abuse material and terrorist content, automated technologies have been used in commercial content moderation for many years. While these tools have long posed risks to freedom of expression, their use was, for quite some time, relatively limited in scope.

Then, in 2017, a blog post published by Facebook (now Meta) described the company's "fairly recent" use of artificial intelligence to identify, classify, and remove violent extremist content. At the same time, Facebook emphasized caution, noting that it did not want to suggest there was "any easy technical fix."

Just one year later, Mark Zuckerberg appeared before the U.S. Senate's Commerce and Judiciary Committees and disclosed that "99 percent of the ISIS and Al Qaida content" removed by Facebook was flagged by AI "before any human sees it." He also stated that Facebook was "developing A.I. tools that can identify certain classes of bad activity proactively and flag it for our team at Facebook." At the time, we raised concerns about the ethical implications of using AI in this manner.

Then came 2020. The sudden reduction of the human moderation workforce, combined with a dramatic increase in social media use—and with it, a surge in misinformation—created the perfect conditions for platforms to expand their reliance on AI-driven moderation. It quickly became apparent that companies'—and particularly Meta's—approach to moderation during the pandemic represented a backslide in transparency, freedom of expression, and access to remedy. The increased reliance on automation was a significant factor.

The costs and benefits of AI content moderation

We knew in 2020 that the use of AI to moderate content would present problems for online freedom of expression. Today, those problems are well-documented. A 2025 joint declaration by special rapporteurs and representatives of the United Nations (UN), Organization for Security and Co-operation in Europe (OSCE), Organization of American States (OAS), and African Commission on Human and Peoples’ Rights (ACHPR) states:

“The use of AI content moderation can lead to over-removal, discrimination and censorship. Reliance on inherently biased datasets and opaque training processes can amplify pre-existing inequalities, risking homogenisation of expression, and erasure of linguistic and cultural diversity.”

EFF and many of our allies have documented these impacts. For example, our 2019 paper co-authored with Witness and Syrian Archive examined the impact of extremist content regulations—and their implementation through automation and AI—on human rights documentation. A 2020 report from Human Rights Watch highlighted the consequences of these removals, noting: "There is no way of knowing how much potential evidence of serious crimes is disappearing without anyone's knowledge."

The Center for Democracy and Technology's recent series on content moderation in the Global South demonstrates persistent inequities in content moderation of four “low-resource” languages—so-called because the relative scarcity of training data makes it more difficult to develop equitable and accurate AI models for them. 

Content moderation often disproportionately impacts vulnerable and historically marginalized groups, and AI content moderation is no different. GLAAD recognizes the role AI plays in scaling content moderation but notes that “when moderation systems lack nuance, transparency, and human oversight, they can fail to curb harassment and wrongly suppress legitimate LGBTQ content.”

These failures are not incidental. They are a predictable consequence of deploying automated systems to make complex judgments about language, culture, context, and identity at scale.

All of that said, automated content moderation can offer important benefits. The primary one: helping to spare human content moderators who must review content that varies from whimsical to horrific, often for little pay and with devastating mental health consequences. Outsourcing this work to the bots can offer some relief—though it’s worth noting that the humans hired to train the AI models face a similar dynamic.

In addition, AI models could potentially be trained over time to be more precise, accurate, and dynamic, helping to mitigate over-censorship and disinformation. The jury is still out on whether this potential will be realized; what we do know is that new approaches to the persistent problem of over and under-enforcement are desperately needed.

Automated moderation is no longer an experiment

Getting the balance between real costs and potential benefits depends a lot on the details: how automated systems are designed, trained, implemented, and audited.  

Despite advances in the sophistication and scale of automated moderation systems, many of the transparency, accountability, and due process safeguards advocated by civil society, researchers, and human rights experts have yet to be fully realized. At the same time, automated systems have become increasingly central to how platforms enforce their rules and govern online speech.

The question today is not whether companies will use AI to moderate content, but under what conditions they should do so. And now as ever, the answer is not that the public should just trust that platforms’ deployment of increasingly powerful systems will serve, rather than inhibit online expression. In fact, as automated systems become more sophisticated and more deeply embedded in platform governance, the need for transparency and accountability becomes more urgent. 

This is part 1 of a 2-part series. You can read the second part here.

This is the second installment of a blog series reflecting on the global digital legacy of the 2011 Arab uprisings. You can read the first post here.

From Russia—where wartime censorship and more stringent platform controls have choked dissenting voices—to Nigeria, with its aggressive takedown orders turning social media into political battlegrounds, and to Turkey, where sweeping “disinformation” laws have made platforms heavily policed spaces, freedom of expression online is under attack. Per Freedom House’s 2023 Freedom on the Net Report, 66% of internet users live where political or social sites are blocked, and 78% are in countries where people have been arrested for online posts. New social media regulations have emerged in dozens of countries in the past year alone.

The online landscape looks markedly different than it did fifteen years ago. Back then, social media was still new and largely free from legal restrictions: platforms moderated content in response to user reports, governments rarely targeted them directly, and blocks (when they happened) were temporary, with censorship mostly focused on whole websites that VPNs or proxies could easily bypass. The internet was far from free, but governments’ crude tactics left space for circumvention.

Those early restrictions, as crude as they were, marked the start of a rapid evolution in online censorship. Governments like Thailand, which blocked thousands of YouTube videos in 2007 over critical content, and Turkey, which demanded takedowns from YouTube before blocking the site entirely, tested legal and technical pressures to mute dissent and force platforms’ compliance. By 2011, governments weren't just reacting—they had learned to pressure platforms into becoming instruments of state censorship, shifting their playbooks from blunt blocks to sophisticated systems of control that simple VPNs could no longer reliably bypass. Governments across the region were watching closely, and by the time the 2011 uprisings began, they were prepared to respond.

Looking Back

After learning that a Facebook page—We Are All Khaled Said, honoring a young man killed by police brutality—sparked Egypt’s street protests, Western media hailed online platforms as engines of democracy. Revolution co-creator Wael Ghonim told a journalist: “This revolution started on Facebook.” That claim was debated and contested for years; critically, Facebook had suspended the page two months earlier over pseudonyms violating its real-name policy, restoring it only after advocates intervened. 

Once the protests moved to the streets, Egypt’s government—alert to social media’s power—quickly blocked Facebook and Twitter, then enacted a near-total shutdown (more on that in part 4 of this series). As history shows, the measures didn’t stop the revolution, and Egyptian president Hosni Mubarak stepped down. For a brief moment, freedom appeared to be on the horizon. Unfortunately, that moment was short-lived.

Egypt’s Digital Dystopia

Just as the Egyptian military government quashed revolution in the streets, they also shut down  online civic space. Today, Egypt’s internet ranks low on markers of internet freedom. The military government that has ruled Egypt since 2013 has imprisoned human rights defenders and enacted laws—including 2015’s Counter-terrorism Law and 2018’s Cybercrime Law—that grant the state broad authority to suppress speech and prosecute offenders.

The 2018 law demonstrates the ease with which cybercrime laws can be abused. Article 7 of the law allows for websites that constitute “a threat to national security” or to the “national economy” to be blocked. The Association of Freedom of Thought and Expression (AFTE) has criticized the loose definition of “national security” contained within the law, as “everything related to the independence, stability, security, unity and territorial integrity of the homeland.” Notably, individuals can also be penalized—and sentenced to up to six months imprisonment—for accessing banned websites.

Articles 25, which prohibits the use of technology to “infringe on any family principles or values in Egyptian society,” and 26, which prohibits the dissemination of material that “violates public morals,” have been used in recent years to prosecute young people who use social media in ways in which the government disapproves. Many of those prosecuted have been young women; for instance, belly dancer Sama Al Masry was sentenced to three years in prison and fined 300,000 Egyptian pounds under Article 26.

Beyond Egypt: Regional Trends

Egypt’s trajectory reflects a wider regional and global pattern. In the years following the uprisings, governments moved quickly to formalize legal authority over digital space, often under the banner of combating cybercrime, terrorism, or “false information.” These laws often contain vaguely worded provisions criminalizing “misuse of social media” or “harming national unity,” giving authorities wide discretion to prosecute speech.

In Qatar and Bahrain, a social media post can result in up to five years in jail. In 2018, prominent Bahraini human rights defender Nabeel Rajab was convicted of “spreading false rumours in time of war”, “insulting public authorities”, and “insulting a foreign country” for tweets he posted about the killing of civilians in Yemen and sentenced to five years imprisonment. 

Two years later, Qatar amended its penal code by setting criminal penalties for spreading “fake news.” Article 136 (bis) sets criminal penalties for broadcasting, publishing, or republishing “rumors or statements or false or malicious news or sensational propaganda, inside or outside the state, whenever it is intended to harm national interests or incite public opinion or disturb the social or public order of the state” and sets a punishment of a maximum of five years in prison, and/or 100,000 Qatari riyals. The penalty is doubled if the crime is committed in wartime.

Now, as war has once again reached the region, these laws are being put to the test. Bahraini authorities have arrested at least 100 people in relation to protests or expression related to the war, while Qatar has arrested more than 300 people on charges of spreading “misleading information.”

And in the UAE, at least 35 people—most or all of whom are foreign nationals—have been arrested and “accused of spreading misleading and fabricated content online that could harm national defence efforts and fuel public panic,” according to the Times of India. The arrests fall under the UAE’s 2022 Federal Decree Law No. 34 on Combating Rumours and Cybercrimes which—says Human Rights Watch—is, along with the country’s Penal Code, “used to silence dissidents, journalists, activists, and anyone the authorities perceived to be critical of the government, its policies, or its representatives.”

From Regional Practice to Global Pattern

Today roughly four out of five countries worldwide have enacted cybercrime legislation, a dramatic expansion over the past decade, with many governments adopting or revising such laws in the years following the Arab uprisings. 

Outside the region, other nations have repurposed these laws to police speech. In Nigeria, journalists have been detained under the Cybercrime Act, with dozens of prosecutions documented since 2015. Bangladesh’s Digital Security Act has been used in thousands of cases—including hundreds against journalists—while in Uganda, authorities have prosecuted political critics under computer misuse laws for social media posts. 

Cybercrime laws are only one piece of a broader toolkit that governments now deploy to control digital spaces. Over the past decade, authorities have introduced sweeping “disinformation” laws, platform liability rules, age verification laws, and data localization requirements that force companies to store data domestically or appoint legal representatives within national jurisdictions. These measures give governments leverage over global technology firms, enabling them to demand faster content removals, obtain user data, or threaten steep fines and throttling if platforms fail to comply. Rather than relying solely on blunt instruments like blocking entire websites, states increasingly govern speech through layered regulatory systems that pressure platforms to police users on the state’s behalf.

The platforms too have changed. The same social media companies that were once championed as tools of democratic mobilization now operate in more constrained environments—and often act as willing participants in repressing speech. Facing financial penalties and the prospect of being blocked entirely, many companies expanded compliance with takedown requests after 2011, as can be seen in the companies’ own transparency reports. They later invested heavily in automated technologies that remove vast quantities of content before it is ever publicly available.

Rights groups around the world, including EFF, have warned that these dynamics disproportionately impact historically marginalized and vulnerable groups, as well as journalists and other human rights defenders. Research by the Palestinian digital rights organization 7amleh and reporting by Human Rights Watch have documented how content moderation policies, government pressure, and opaque enforcement mechanisms increasingly converge—leaving activists, journalists, and human rights defenders caught between state censorship and platform governance.

The New Architecture of Repression

Looking back now, it’s clear that, fifteen years ago, governments were caught off guard. They crudely blocked platforms, shut down networks, and scrambled to contain movements they did not fully understand. But in the years since, states have systematically adapted, transforming what were once reactive measures into durable systems of control.

Today’s controls are embedded in law, outsourced to platforms, and justified through the language of security, safety, and order. Cybercrime statutes, disinformation frameworks, and platform regulations form a layered architecture that allows states to shape online expression at scale while maintaining a veneer of legality. In this system, repression is often procedural, bureaucratic, and continuous.

The question is no longer whether the internet can enable dissent, but whether it can still sustain it under these conditions.

This is the second installment of a blog series reflecting on the global digital legacy of the 2011 Arab uprisings. Read the rest of the series here.