In courts across the country, publishers and creators are suing large AI companies for training generative AI models on copyrighted works. It’s easy to see these cases as a David-and-Goliath showdown between authors and publishers fighting for a cut of Big Tech company profits. But this fight is actually much larger. It affects billions of people who aren’t represented in the case at all, including researchers, artists, activists, journalists, people with disabilities, and anyone who accesses and shares content online.   

That’s why EFF is stepping in to defend the public’s rights in these cases. Today, EFF filed an amicus brief in In re OpenAI, explaining how the court’s ruling on fair use could impact people worldwide. If fair use gets narrowed, it will impede work that benefits the public, like medical research and investigative journalism, and even limit access to privacy protection tools.

Let’s be clear: creators and workers deserve support as AI changes their fields, and Big Tech companies must be held accountable for the harms they cause. But giving copyright holders more control over the web won’t accomplish either of these goals. If the plaintiffs prevail in these cases, courts would have to rewrite copyright law in ways that hurt everyone except powerful media and tech companies.

Rewriting Fair Use Makes Everyone Worse Off

Copyright exists for one reason: to stimulate creativity and innovation in service of the greater public good. Unfortunately, the plaintiffs in this lawsuit are arguing to limit everyone’s fair use rights, in ways that would thwart the basic purpose of copyright. The harms of shrinking fair use will go far beyond generative AI, impacting everyone.

Initial speculation that AI models would be infringement machines, churning out works highly similar to those on which they were trained, has proven unfounded. And courts have already found fair use in cases where copyrighted works were used to train generative AI models, recognizing that training uses those works for a fundamentally different purpose.

Plaintiffs in these cases argue that, even if the use is transformative, using their works to train AI is still unfair, because the AI models could generate new, non-infringing works that might compete with plaintiffs’ own works. They also argue that scraping or downloading the data without permission—or in some cases, from websites containing infringing copies—was independently infringing and/or rendered any subsequent uses of that data unfair.

These arguments would put beneficial research and reporting at risk. First, if copying without permission rules out fair use, the law will never protect transformative uses that benefit us all. Researchers, journalists, and other watchdog groups gather information the same way AI companies do—but they do it to serve the public interest, and often to hold Big Tech itself accountable. For example, journalists and nonprofit researchers have used “crawling” or “scraping” to investigate potentially anticompetitive practices by tech companies, and research misinformation and hateful content posted on X. Their data acquisition techniques are virtually identical to those used by AI giants—but they lack those companies’ deep pockets.

Requiring users to show that copyrighted material was lawfully obtained would also threaten everyday internet use. As EFF’s brief explained, that requirement would “cast legal doubt on the billions of copies we collectively make every day through routine online activities like opening emails, clicking on an embedded link, visiting websites, sharing pictures, streaming music, and watching videos. All of this activity requires intermediate copying.”

Fair use is what lets us all use the internet without being compelled to track the source of every image, video, and article that we click on, and make sure the rights are all cleared.

Additionally, while rightsholders are understandably concerned about competition from AI generated works, copyright law exists to promote new creation, not protect authors who published first. Under the plaintiffs’ theory, no one could stand upon the shoulders of giants. Copyright could prevent people from writing new books inspired by themes in a favorite novel or genre, or prevent academic researchers from publishing literature reviews that distill insights from prior research—simply because readers might prefer those new publications over those that came first.

Finally, general-purpose AI models are neither designed nor usually used to create works targeted at any particular market. These models have been used to generate new insights and tools for medicine, science, journalism, art, and accessibility. For instance, an AI-powered tool called Be My AI uses Open AI’s GPT models to provide visual descriptions for people who are blind or have low vision. Generative tools have also accelerated scientific research, and researchers have used these tools to develop new medical treatments and medications for conditions previously thought to be incurable.

Mandatory Licensing Would Entrench Big Tech

Some creators and publishers argue that AI developers should just get a license. But requiring permission for every work used would give the largest companies massive advantages over small competitors and impede academic research. As Professor Lawrence Lessig recently noted:

If all training requires a license, then none but a handful of tech giants will be able to afford the costs of training. That’s not because authors are greedy but because the copyright system is the least efficient property system known to man.

Moreover, creators are unlikely to benefit from these licensing arrangements in many cases. Media and entertainment giants have exploited the imbalance between themselves and underpaid working artists for decades. Trying to help creators by giving them more rights under copyright is, as Cory Doctorow has written, like trying to help a bullied kid by giving them more lunch money for the bully to take. Artists aren’t getting good deals from record labels or publishers now; there’s no reason to think these companies, which already own the copyrights, will treat artists more fairly once they control the data that powers AI.

The Future of AI Is Too Important to Leave to Big Content or Big Tech

AI technologies will affect all of us, and copyright law is ill-equipped to address many of these consequences. The rules governing these technologies should be rules that serve the public—not just the small number of companies that can afford to litigate copyright battles for decades, or settle for billions.

New records obtained by EFF expose additional issues with Medicare’s AI-driven prior authorization program and show why we need stronger legal limits around government use of automated decision-making. 

EFF sued the government back in March for information about the Wasteful and Inappropriate Service Reduction (WISeR) model, a new Medicare program that uses AI to evaluate prior authorization requests for certain medical services. We previously released records showing that WISeR has resulted in delays and denials of care, operational chaos, and reports of patient harm. Today, we’re publishing over 800 new pages of records, including:

• Requests for information (RFIs) or requests for corrective action plans (CAPs) sent from the Centers for Medicare & Medicaid Services (CMS) to four of the six WISeR vendors, along with the vendors’ responses.
• Applications of 66 companies to the WISeR model, including those of the 6 selected companies. 
• Task orders and contracts for the evaluation, implementation, and monitoring of the WISeR model.

Documents obtained through EFF’s lawsuit are available here.

These newly released documents show that the WISeR rollout was rushed and rocky from the start, highlighting the dangers that arise when governments treat AI like a silver bullet for complex problems. The documents also highlight the need for more transparency about safeguards against inappropriate, AI-driven denials. The public still lacks the information needed to assess whether clinician review is providing meaningful oversight of AI systems and whether appropriate data is being used to evaluate prior authorization requests. 

EFF has long argued that AI has no place in consequential government decisions about peoples’ rights or access to benefits, such as immigration enforcement and policing decisions. That’s because using automated decision-making in these contexts risks decisions that violate people’s rights or deprive them of benefits they’re entitled to based on biased or inappropriate considerations and without proper due process or accountability if there is an error. 

By outsourcing decision-making to private AI companies without sufficient transparency, testing, or legal limits, WISeR jeopardizes seniors’ access to medical care. Congress should step in to place greater legal limits on the use of automated decision-making in Medicare. 

WISeR’s Rollout Was Rushed and Rocky 

CMS launched the WISeR program just six months after announcing it, ignoring calls from medical providers to delay its rollout. In records previously released by EFF, CMS acknowledged this “extremely condensed timeline.” 

New records suggest that WISeR’s unusually rapid rollout contributed to widespread problems in the first few months of the program. CMS did not disclose the extent of these issues to the public, even as agency employees privately contacted vendors about potential noncompliance with program requirements. 

In the first three months of the program, CMS sent RFIs or requests for CAPs to at least four of the six WISeR vendors. These requests, which describe delays, technical failures, and a lack of communication with providers, suggest that widespread problems during the program’s rollout were not limited to one vendor or state. 

These letters underscore the severity of the problems with the program. For example, CMS requested information from one vendor, Virtix, about numerous reported issues, including “determination turnaround times that exceed established guidelines,” “denials that appear inaccurate or inconsistent with applicable criteria,” and “inadequate customer service.” CMS’s RFI to Zyter–a vendor that uses Palantir software infrastructure–observed that the agency had received reports that “Zyter is not issuing determination letters in a timely manner” and communications indicating that it “may lack adequate processes for responding to and escalating provider/supplier concerns.” The vendor Innovaccer received two CAP requests and a “Performance Notice of Non-Compliance” over its failure to implement a fully functional portal for WISeR providers and suppliers.

Excerpts from CMS’ letters to vendors describe problems during WISeR’s rollout:

Vendors’ responses to CMS provide further evidence that WISeR’s rushed rollout contributed to its problems. Innovaccer claimed it warned CMS months in advance that it might not be ready to provide full portal functionality by the launch date, even before signing a Participation Agreement. Another vendor claimed that “technical and workflow stabilization challenges were encountered due to the compressed implementation timeframe” it faced after joining the WISeR program a month after the other vendors. 

The issues reflected in CMS’s letters to vendors, and the vendors’ responses, illustrate the risks of deploying automated decision-making systems without proper testing or safeguards. While vendors’ responses describe fixes for the problems CMS identified, after-the-fact remedies cannot undo the harms that patients and providers already experienced during WISeR’s rushed rollout. 

More Transparency Needed About AI Safeguards

The public still knows far too little about any safeguards that WISeR vendors may have adopted—in part because the government chose to conceal some of this information. Transparency about these safeguards is critical to protecting patients, especially because WISeR’s payment structure gives vendors, as the government put it, “an incentive to deny as many claims as possible.” 

The public has limited means to evaluate the effectiveness of even the safeguards that weren’t redacted from records released to EFF. For example, one vendor’s application touts safeguards like “Human-in-the-loop (HITL) workflows,” “Continuous model monitoring,” and “Transparent audit trails.” Yet it remains unclear whether these measures effectively prevent improper prior authorization denials. Some evidence suggests they may not: the company denied more requests than it approved in the first few months of the program, and CMS inquired about “denials that appear inaccurate or inconsistent with applicable criteria.”

Ultimately, the records obtained by EFF raise more questions than they answer about the safeguards used by WISeR vendors. 

Having a Human in the Loop Is Probably Not Enough

CMS has emphasized having a human in the loop as a key safeguard against inappropriate, AI-driven denials. The WISeR FAQ page on the CMS website states that “Non-affirmations will require the review of a human clinician and cannot be performed solely by technology.” But whether human review provides any meaningful protection depends on the details of its implementation, such as whether human reviewers have enough time to evaluate requests and practical authority to override AI recommendations. We still lack many of those details for WISeR vendors. 

A growing body of scholarship points to the substantial limitations of “human in the loop” policies as guardrails for AI systems. AI-generated recommendations can unduly influence human decisions, leading reviewers to over-rely on automated judgements and rubber stamp them as correct. Even worse, at least one health insurance company actively pressured clinical reviewers to accept its AI systems’ recommendations. Another company built a system that allowed clinical reviewers to reject claims without opening patient files; clinical reviewers using this system spent an average of just 1.2 seconds on each case.

Is the Right Kind of Data Informing Companies’ Decisions?

Some WISeR vendors’ applications offer general descriptions of the types of data informing their decisions. But far more details are needed to allow medical professionals to assess whether the data inputs to these AI systems are appropriate for evaluating prior authorization requests. 

Inappropriate training data or input data can lead to biased, unfair, or inaccurate decisions. CMS guidance for Medicare Advantage organizations using AI to make coverage decisions emphasizes the importance of basing decisions on individual patients’ circumstances. It cautions against using “an algorithm that determines coverage based on a larger data set instead of the individual patient's medical history, the physician’s recommendations, or clinical notes.” 

WISeR vendors’ applications reference data sources like “provider behavior,” “patient demographics,” “utilization management datasets,” “provider and payer performance metrics,” and “historical claims data.” Whether these sources could introduce bias or give too much weight to population-level patterns depends on how vendors use them to inform individual decisions. But we don’t yet have these important details about vendors’ data and algorithms. By outsourcing decision-making to private companies’ AI systems, the government has made it harder for the public to evaluate whether decisions are being made fairly.

Next Steps

As the government expands its use of automated decision-making, the potential for consequential mistakes increases rapidly. The records we’ve obtained so far raise questions about the rushed rollout and lack of guardrails for automated decision-making by the U.S. government. EFF urges transparency, oversight, and legal limits to protect people from the harms of automated decision-making in high risk systems.  

If you use the internet at all, you may have noticed that a large percentage of sites suddenly went down earlier this week—and all your nerdiest friends were talking about "The Vulnerability Apocalypse” and this thing called PHP that powers nearly 70% of the websites on the internet. You may find yourself wondering: what is PHP, what just happened, what is the vulnerability apocalypse, and what do I need to do?  

To find out what you can personally do read the first blog post in this series!  

What Is PHP?  

PHP is a programming language used mainly to create interactive websites. Facebook, Wordpress, Wikipedia, Etsy, and other major sites all use or were originally built on PHP. It is a long-established mainstay of web development. But because it is open-source, maintenance of these critical systems depends on a small number of volunteers working on this in their free time. The disaster scenario that happened this past week is the perfect example of why we need more money and people supporting open-source projects. Corporations should be contributing both. 

What Is the Vulnerability Apocalypse?  

Common Vulnerabilities and Exposures (CVEs) are the way that security professionals file reports about vulnerabilities in software. The National Vulnerability Database (NVD), run by NIST, is the official national database of CVEs. In other words, CVE reports managed by NIST are the closest thing we have to a canonical database of security vulnerabilities in software. The NVD currently has around 400,000 CVEs that have been filed and accepted since 1988. In just the month of October, 13,000 new CVEs have been submitted. This means that 3% of critical software security bugs that have ever been officially logged were found in the last month alone. And it's likely that by the end of the year over a quarter of the software vulnerabilities that have ever been found will be from 2026.  

 

This research is being driven almost entirely by the cybersecurity capabilities of major AI models from companies like Anthropic, OpenAI, and Google. Historically it has been popularly assumed by cybersecurity experts that finding unknown (zero-day) exploits in any specific project and turning them into exploits was expensive. The knowledge to discover them was not widespread, and it was a hard task even for those initiated into the arcane ways of vulnerability hunting. Google even dedicated an entire team to it.  

But now thanks to major AI models, vulnerabilities are cheap, and anyone of moderate technical skill with a few dollars and access to a frontier AI model can find vulnerabilities in a few minutes. For example, recent vulnerabilities in PHP were found with a prompt to an AI model that literally said "find bad vulnerabilities in PHP."  

What Happened This Week? 

In 2024, a potentially severe vulnerability in PHP was discovered during an external security audit. PHP’s maintainers couldn’t agree on a solution, but more importantly, they couldn’t find a way to exploit the vulnerability. Because it didn’t seem like an active threat, they ended up deprioritizing a fix in favor of working on more urgent security issues. Then, in August 2026, security researchers working with the latest models from OpenAI and Anthropic independently discovered the vulnerability again, and another one—and now, with a way to exploit them.  

We now know that the vulnerabilities were so severe that they would have allowed even moderately inventive hackers to take over any website that uses PHP and then steal customer data, launch further malware attacks, spy on users, take websites offline, or launch a massive denial of service attack.^[1] 

Because of the severity of these vulnerabilities, several of the involved researchers got together with the makers of PHP to patch the vulnerabilities and disclose them in a coordinated manner. The idea was that hopefully everyone would get a chance to quietly release and then apply fixes for the vulnerabilities before the whole world found out about it. 

But news already started to leak on October 20th. There was a post on a widely read forum “Hacker News” and other posts elsewhere. One of the researchers accidentally published a video early. Companies reported seeing active exploitation of the vulnerabilities going back as far as October 17th. On October 21st, six hours before the planned press embargo, one company, Chainguard, jumped the gun and released a patch early, essentially giving anyone instructions about exactly where to find the vulnerabilities and how to weaponize them. This sent system administrators everywhere scrambling to patch their systems before they were taken over by hackers. And it’s going to take a long time to patch all these systems. At the time of this writing over 33 million websites are still running on vulnerable versions of PHP.  

The internet is run largely on open-source software, mostly maintained by a small handful of skilled volunteers occasionally getting a small stipend, not by major corporations and highly paid engineers. These are projects you have perhaps never heard of, like PHP, OpenSSL, and SQLite, but these programs run on nearly every computer in the world and are maintained by teams as small as one person. When a vulnerability is found in these projects, it can be devastating.  

PHP isn’t the only critical open-source software with a high severity vulnerability this week either. Other widespread open-source projects like Node.js and GitLab also had severe security bugs reported.  

The small number of already overworked open-source maintainers are now facing a deluge of vulnerability reports, some of which are false, but many of which are very real. Burnout in open source has never been higher.  

We Need Stronger Support for Open Source Projects

Much of the foundation for the modern multibillion-dollar internet industry is built on open-source products that were created and still maintained—for free—by dedicated volunteers in their spare time. It’s an imperfect system, and it can lead to errors.  

One of those critical, under resourced open-source products is PHP. And while vulnerabilities in PHP were privately found years ago, they were left unfixed because they didn’t seem as impactful as other bugs at the time. These same vulnerabilities were recently rediscovered and proved to be trivially exploitable by security researchers using advanced AI models. While in the middle of the widespread, global effort to correct this mistake and deploy fixes, the problem was leaked by some of the security researchers who had gotten early access to the fixes. Unfortunately, this left millions of websites, ranging from personal blogs to government sites, vulnerable to hacking that could result in data theft, malware distribution, and worse.  

Open-source projects like PHP and others are the backbone of the entire internet. If they don’t get the resources they need, they and the internet they power, are at risk of falling apart. Corporations (whose products would not exist without these products) should be actively contributing to their maintenance. 

^{1. A technical analysis of the vulnerabilities is out of scope for this post and has already been written by the original researchers.}

It is a commonly held truth among computer geeks that there is no such thing as bug-free software, or a secure computer. Any piece of software that anyone actually uses will have a security vulnerability somewhere in the code. The more complex that software is, the more likely there will be a vulnerability. Fortunately for everyone, finding severe vulnerabilities was generally considered an expensive and difficult task requiring years of training and deep knowledge of how computers and software worked. Until recently there was a steady background hum of vulnerabilities being found and patched, with major vulnerabilities that could cripple the entire internet only happening every few years. Now that is no longer the case.  

You may have noticed your nerdiest friends freaking out about the “Vulnerability Apocalypse” this week and waving their arms in panic about something called PHP. You may have also noticed that some of your favorite websites were down for a bit. In the last two weeks of October alone, there have been multiple, potentially internet-breaking vulnerabilities reported and patched. Many of those vulnerabilities were in the open-source software that is the foundation of the vast majority of the internet.

For more detailed information on what happened, and why supporting open-source projects matters, check out our other blog.

This is all because frontier AI models, such as those created by OpenAI, Anthropic, and others, have made it incredibly cheap and easy to find previously unknown vulnerabilities (zero-days) in open- and closed-source code. Three percent of all software vulnerabilities ever found in history were found just in the month of October. This year alone researchers have found nearly a quarter of all software security issues which have ever been discovered. 

If exploiting critical software is now cheaper and easier than ever, what does that mean for us as individuals and what does that mean for society as a whole? TL; DR: It’s going to get worse before it gets better. But you can protect yourself with basic digital security hygiene that is now more important than ever.  

What Do I Need to Do? 

There absolutely needs to be a bigger conversation about how we got here and how to fix the systemic issues involved, including how corporations that depend on open-source code should contribute to their upkeep, and how big tech should be regulated.  

But for now, this news is a compelling reason to tighten up your digital hygiene. A lot of computers are going to get hacked a lot more often, and unfortunately, it isn’t going to get better any time soon. Thankfully, EFF has compiled a comprehensive set of security advice in our Surveillance Self-Defense guides. To start, we recommend making sure your browser and all applications are always up to date with the latest patches.   

You should also consider installing a tracker blocking tool, like EFF’s Privacy Badger (now on Safari), and remain vigilant for phishing attempts. Additionally, install a password manager and use unique passwords for each website, so that if one site is compromised, the rest of your passwords are safe. You should also be aware that data breaches may now be more likely, and the websites you visit may experience more attacks than usual, which means that some services may be disrupted.  

If you host a blog or other website, make sure that either you or your hosting provider has patched this particular issue.  

For additional tips on protecting yourself and others, head over to our Surveillance Self-Defense site. Additionally, if you’d like to support any of EFF’s open-source projects like Privacy Badger, Rayhunter, and Certbot, visit our donate page. 

California Gov. Gavin Newsom's executive order is an opportunity for a needed, thoughtful conversation about artificial intelligence and its potential harms. Everyday Californians are feeling real anxiety about the risks of artificial intelligence, and as an organization that works to ensure technology empowers people, EFF welcomes this order as a way for the state of California to lead a much-needed dialogue that addresses these concerns. 

DONATE TO EFF

ensure ai serves people, not power

Nonetheless, the most immediate and current concerns with this technology are not about sci-fi scenarios concerning rogue super-intelligence. They are happening right now through biased algorithmic decision-making for employment or government benefits, AI-powered surveillance systems such as Flock cameras, and artificially inflated personalized pricing. People want state and federal leaders to act, and we urge Gov. Newsom to develop thoughtful policies to address those concerns. Today’s EO is a good start. 

To that end, EFF supports the focus on expanding the reporting requirements under SB 53 (2025) for loss-of-control incidents, alongside third-party investigations. We urge the administration to consider how to make these third-party investigations available for smaller developers. As the Government Operations Agency prepares its recommendations for the governor, we urge leaders to also realize that the effectiveness of kill switches in advanced AI systems remains an area of active research. As such, they should ensure that - as we’ve previously mentioned - any technology regulation targeting cybersecurity practices at AI labs must be careful, precise, and practical. Moreover, we also caution that government-controlled kill switches run the risk of being used as a form of retaliation against protected speech, as demonstrated by the Trump Administration’s retaliatory actions against Anthropic earlier this year.

Ultimately, true safety requires California to focus on concrete, immediate, and urgent harms of AI technologies by ensuring that algorithmic decision-making in both the government and private sectors respects people’s rights and well-being. We urge Gov. Newsom and the state of California to develop thoughtful policy in collaboration with those most at risk of harm to address these and other concerns.

Secure messaging platforms, like Signal, WhatsApp, and recently, encrypted RCS, operate on a straightforward assumption: the content at each end of a conversation is private to the participants in the conversation. End-to-end encryption helps provide the mathematical guarantees that the companies who operate these messaging platforms cannot access the contents of messages. But there’s no way to guarantee what happens once the message arrives on a phone. As more devices and services introduce more artificial intelligence (AI) features into messaging apps, that line begins to blur. 

When AI features are computed entirely on device, it’s less concerning. Yet sometimes the computing requirements are heavy enough that the computation has to be done on a company server. Tech companies tell us they have a solution for this: trusted execution environments (TEEs). But do server-side TEEs really solve the problem?

TEEs exist to serve many different functions, ranging from digital rights management (DRM) content protections to securely storing information in your phone's mobile wallet, but for our purposes, we’ll be focusing on how tech companies use them for their AI tools. 

The basic idea is straightforward: most consumer devices aren’t powerful enough to handle the sorts of AI features companies want to offer, so sometimes they send data off your device to more powerful cloud servers to do the computing, then display the results on your device. Since your data is leaving your device, there’s a privacy compromise. For example, if you ask for a messaging app to summarize a conversation, it may offload that computing power to a cloud server, sending the entire contents of your messages to the cloud, then back to your phone.

TEEs supposedly offer a way to keep those requests private. There are several implementations out there, like Apple’s Private Cloud Compute, Google’s Private AI Compute, and WhatsApp’s Private Processing. It’s not just the big tech players, we’ve seen chatbots built with TEEs as well.

TEEs can provide more security and privacy than simply running in the clear, but they are fundamentally different from actual encryption or running locally. Despite the promises of some tech companies, they will never be able to match that level of security and privacy. Because of that, a user’s device should never automatically send data to a TEE. Let’s dig through the reasons why.

What Exactly Is a TEE, Anyway?

A TEE is a hardened section of the computer that runs software in a way that’s supposed to be secret even from other processes running on the machine. TEEs also let users check that the code being run is the code that they think is running, and not backdoored code instead, using a process called “attestation.” You may have also heard this referred to as a “secure enclave,” or heard the brand names SGX or TrustZone.

The intention of a cloud-based TEE is simple: a company can run a server in their data center, but still process data that you provide on your behalf without being able to see that information themselves.

Is a TEE Secure?

In practice, we've seen multiple cracks and hacks every year that show that it is possible to get at that data. That’s because while encryption relies on math, TEEs rely on engineering to provide their security. Standard encryption algorithms are created by years-long processes collaboratively produced by mathematicians around the world and are based on problems that have been studied for decades. The math is reliable, and there is no shortcut to breaking it that would not also upend fundamental understandings of mathematics as a field. 

The collective understanding of every mathematician in the world is that standard encryption algorithms are not breakable to the best of the world’s collective knowledge. No responsible engineer builds a system based on a new encryption method until after it’s been offered up for prodding.

Engineering, on the other hand, doesn’t work like that. Every individual system is the product of a group of engineers who put it out into the world, and each product will have its own quirks and bugs that have to be individually discovered and patched. These bugs are found after the system is built, not before. No one has yet built a system that is unbreakable. On the contrary, there is new research all the time that finds new ways to break into TEE systems. They’re patched as they come up, but they’re unlikely to ever become perfect, and certainly not any time soon. 

TEEs in particular are a hard engineering problem because the encryption key is physically right there on the device. Building a TEE means keeping a key fully separate and inaccessible while it’s on the same physical device as parts of the system that shouldn’t have access to the key.

Many attacks on TEEs involve “side channels.” In a side channel attack, the attacker measures the electrical impulses or other effects to figure out the timing of operations inside the TEE, then uses that to figure out the key being used. Once they have the key, they can read all the data. Compare that to end-to-end encryption, where the key is never on that machine in the first place, so an attacker would have to also run a similar attack on the user’s device.

Companies who turn to TEEs to protect data want to both have the key on the server and have it protected while still performing complex operations like running an LLM, which makes it much more difficult to protect those keys.

That being said, a TEE versus plaintext on a server is the difference between being able to easily read the data and having to do a bunch of specialized work to get at the data. That work often involves accessing the physical machine. This is most relevant for protecting against mass surveillance, and for many people, that might just be enough security.

But that's the core of the problem. “Secure enough for most cases” and “encrypted as in math” are not the same thing, and it’s important not to conflate the two. And services that currently offer “encryption as in math” have a real downgrade in security when they switch to security based on TEEs. 

If you want to dive into the myriad security issues and limitations of TEEs we’ve seen so far, they’re well documented here, here, here, and here.

What Does This Have To Do With LLMs and AI?

Sometimes organizations want to offer an LLM that can respond to queries in a private manner. On-device LLMs exist, but they’re limited in size. So, when organizations want to offer the ability to answer queries without being able to see the conversation, they turn to TEEs. That’s a useful way to run a chatbot that’s reasonably private. This is what Apple, Google, WhatsApp, and others are doing.

Why not turn to encryption? After all, LLM inference is just a bunch of math like any other things a computer does. It takes input to a (really big) function and gives an output. We have the math to do that computation in a way that hides the inputs and outputs from the one running the computation, it's just super expensive. It’s called homomorphic encryption, and no one’s figured out how to do it fast enough that it makes sense for this sort of computation.

Instead, the allure of a TEE is that it will run that computation for you inside of a special opaque section of a server. TEE manufacturers try to make it as hard as possible for the person running the TEE to peek inside. But you still have to trust the operator to not put a stethoscope to the box to try to figure out what's happening inside. 

In this case, it’s reasonable to consider these systems “privacy-preserving,” but not “encrypted.” That distinction is important, especially when we talk about how the TEEs interact with secure messaging. When someone using an end-to-end encrypted chat app asks an LLM to summarize, review, or store those messages, the content of those messages is leaving the device and going to an unencrypted third-party server somewhere. That’s a major threat to the privacy of secure chat apps, and one that’s increasingly hard for users to take control of.

How Does This Translate to Practical Advice?

The answer to this is going to vary based on an individual’s threat model, but a good rule of thumb is that a user’s device should never automatically send data to a TEE. When the person holding a phone can choose what information is sent, even if it’s a chunk of data like “unread messages,” they have the opportunity to pause and consider if that data might be too sensitive to risk sending.

In contrast, when data is sent automatically, the automatic sending becomes a feature of the system as a whole. If the system was previously end-to-end encrypted, adding automatic exfiltration makes the whole system no longer end-to-end encrypted.

Developers: don’t build systems that automatically send data off a device to a TEE, especially when it’s coming from an app that is otherwise end-to-end encrypted.

Users: if developers ignore us and build that system, turn off any automatic data sending features. Take a second to think about how much you’re willing to risk sending data when you choose to send it off the device.

So, What Should I Be Concerned About?

TEEs are useful for security in a number of circumstances. Your phone likely has a TEE where it keeps the key that encrypts your biometric unlock data and the base of the keychain where passwords are stored. It also enables certain backup systems, like how you can restore a phone with your passcode or restore WhatsApp or Signal backups.

But when we’re talking about cloud processing, it’s important to be clear this isn’t the same as end-to-end encryption and doesn’t offer the same level of privacy. 

Most of our private lives are on our phones and in our messages. We’ve worked for years to secure those messages, with major wins like encrypted RCS, and the continued user experience improvements of Signal and WhatsApp. We’ve even seen real improvements to backup security with features like Advanced Data Protection that bring end-to-end encryption for a variety of data outside of messaging, like notes and photos. 

But as companies roll out AI features that interact with these encrypted services, pulling data off devices and into a cloud-based TEE, they’re eroding the privacy protections of end-to-end encryption and risk causing serious confusion around what data is protected and what isn’t.

With doomsday AI scenarios dominating the news, lawmakers are rightly concerned about reports concerning security breaches at major US AI labs, such as the OpenAI–Hugging Face incident and the many others reported in its aftermath. As they consider potentially regulating frontier AI, they should focus any new legislation on the immediate, demonstrated risks from those incidents. 

Post-incident reports show that the Hugging Face incident could have been mitigated or prevented by following longstanding cybersecurity best practices, like stronger sandboxing and monitoring. Any new legislation should focus on closing gaps in existing law to prevent AI companies from taking unreasonable risks with the public's security.

When an AI developer or deployer runs a test or a task that has a high likelihood of causing harm to third parties—for instance, by breaking into someone else's computers—there should be clear minimum safety requirements. Such tests should run in a properly sandboxed test environment, disconnected from other systems, and be monitored and logged. Following these fundamental best practices would have prevented or substantially mitigated all of the incidents at AI labs that we currently know about.

That said, any proposal must be flexible enough to evolve with changing technology. Minimum safety requirements specific only to current AI technologies are likely to become obsolete; legal standards tied to well-established cybersecurity best practices are far more likely to stand the test of time. Tying any new mandates to evidence-backed security protocols also protects the public without impeding future AI development.

Strong legislation should also mandate and fund independent third-party investigations into any serious security incidents that may occur during AI labs’ tests of new tools, and make reports of these investigations available to the public. This important transparency measure would go a long way toward providing public oversight of the industry.

As with any technology regulation, those targeting cybersecurity practices at AI labs must be careful, precise, and practical.