California Gov. Gavin Newsom's executive order is an opportunity for a needed, thoughtful conversation about artificial intelligence and its potential harms. Everyday Californians are feeling real anxiety about the risks of artificial intelligence, and as an organization that works to ensure technology empowers people, EFF welcomes this order as a way for the state of California to lead a much-needed dialogue that addresses these concerns. 

DONATE TO EFF

ensure ai serves people, not power

Nonetheless, the most immediate and current concerns with this technology are not about sci-fi scenarios concerning rogue super-intelligence. They are happening right now through biased algorithmic decision-making for employment or government benefits, AI-powered surveillance systems such as Flock cameras, and artificially inflated personalized pricing. People want state and federal leaders to act, and we urge Gov. Newsom to develop thoughtful policies to address those concerns. Today’s EO is a good start. 

To that end, EFF supports the focus on expanding the reporting requirements under SB 53 (2025) for loss-of-control incidents, alongside third-party investigations. We urge the administration to consider how to make these third-party investigations available for smaller developers. As the Government Operations Agency prepares its recommendations for the governor, we urge leaders to also realize that the effectiveness of kill switches in advanced AI systems remains an area of active research. As such, they should ensure that - as we’ve previously mentioned - any technology regulation targeting cybersecurity practices at AI labs must be careful, precise, and practical. Moreover, we also caution that government-controlled kill switches run the risk of being used as a form of retaliation against protected speech, as demonstrated by the Trump Administration’s retaliatory actions against Anthropic earlier this year.

Ultimately, true safety requires California to focus on concrete, immediate, and urgent harms of AI technologies by ensuring that algorithmic decision-making in both the government and private sectors respects people’s rights and well-being. We urge Gov. Newsom and the state of California to develop thoughtful policy in collaboration with those most at risk of harm to address these and other concerns.

Secure messaging platforms, like Signal, WhatsApp, and recently, encrypted RCS, operate on a straightforward assumption: the content at each end of a conversation is private to the participants in the conversation. End-to-end encryption helps provide the mathematical guarantees that the companies who operate these messaging platforms cannot access the contents of messages. But there’s no way to guarantee what happens once the message arrives on a phone. As more devices and services introduce more artificial intelligence (AI) features into messaging apps, that line begins to blur. 

When AI features are computed entirely on device, it’s less concerning. Yet sometimes the computing requirements are heavy enough that the computation has to be done on a company server. Tech companies tell us they have a solution for this: trusted execution environments (TEEs). But do server-side TEEs really solve the problem?

TEEs exist to serve many different functions, ranging from digital rights management (DRM) content protections to securely storing information in your phone's mobile wallet, but for our purposes, we’ll be focusing on how tech companies use them for their AI tools. 

The basic idea is straightforward: most consumer devices aren’t powerful enough to handle the sorts of AI features companies want to offer, so sometimes they send data off your device to more powerful cloud servers to do the computing, then display the results on your device. Since your data is leaving your device, there’s a privacy compromise. For example, if you ask for a messaging app to summarize a conversation, it may offload that computing power to a cloud server, sending the entire contents of your messages to the cloud, then back to your phone.

TEEs supposedly offer a way to keep those requests private. There are several implementations out there, like Apple’s Private Cloud Compute, Google’s Private AI Compute, and WhatsApp’s Private Processing. It’s not just the big tech players, we’ve seen chatbots built with TEEs as well.

TEEs can provide more security and privacy than simply running in the clear, but they are fundamentally different from actual encryption or running locally. Despite the promises of some tech companies, they will never be able to match that level of security and privacy. Because of that, a user’s device should never automatically send data to a TEE. Let’s dig through the reasons why.

What Exactly Is a TEE, Anyway?

A TEE is a hardened section of the computer that runs software in a way that’s supposed to be secret even from other processes running on the machine. TEEs also let users check that the code being run is the code that they think is running, and not backdoored code instead, using a process called “attestation.” You may have also heard this referred to as a “secure enclave,” or heard the brand names SGX or TrustZone.

The intention of a cloud-based TEE is simple: a company can run a server in their data center, but still process data that you provide on your behalf without being able to see that information themselves.

Is a TEE Secure?

In practice, we've seen multiple cracks and hacks every year that show that it is possible to get at that data. That’s because while encryption relies on math, TEEs rely on engineering to provide their security. Standard encryption algorithms are created by years-long processes collaboratively produced by mathematicians around the world and are based on problems that have been studied for decades. The math is reliable, and there is no shortcut to breaking it that would not also upend fundamental understandings of mathematics as a field. 

The collective understanding of every mathematician in the world is that standard encryption algorithms are not breakable to the best of the world’s collective knowledge. No responsible engineer builds a system based on a new encryption method until after it’s been offered up for prodding.

Engineering, on the other hand, doesn’t work like that. Every individual system is the product of a group of engineers who put it out into the world, and each product will have its own quirks and bugs that have to be individually discovered and patched. These bugs are found after the system is built, not before. No one has yet built a system that is unbreakable. On the contrary, there is new research all the time that finds new ways to break into TEE systems. They’re patched as they come up, but they’re unlikely to ever become perfect, and certainly not any time soon. 

TEEs in particular are a hard engineering problem because the encryption key is physically right there on the device. Building a TEE means keeping a key fully separate and inaccessible while it’s on the same physical device as parts of the system that shouldn’t have access to the key.

Many attacks on TEEs involve “side channels.” In a side channel attack, the attacker measures the electrical impulses or other effects to figure out the timing of operations inside the TEE, then uses that to figure out the key being used. Once they have the key, they can read all the data. Compare that to end-to-end encryption, where the key is never on that machine in the first place, so an attacker would have to also run a similar attack on the user’s device.

Companies who turn to TEEs to protect data want to both have the key on the server and have it protected while still performing complex operations like running an LLM, which makes it much more difficult to protect those keys.

That being said, a TEE versus plaintext on a server is the difference between being able to easily read the data and having to do a bunch of specialized work to get at the data. That work often involves accessing the physical machine. This is most relevant for protecting against mass surveillance, and for many people, that might just be enough security.

But that's the core of the problem. “Secure enough for most cases” and “encrypted as in math” are not the same thing, and it’s important not to conflate the two. And services that currently offer “encryption as in math” have a real downgrade in security when they switch to security based on TEEs. 

If you want to dive into the myriad security issues and limitations of TEEs we’ve seen so far, they’re well documented here, here, here, and here.

What Does This Have To Do With LLMs and AI?

Sometimes organizations want to offer an LLM that can respond to queries in a private manner. On-device LLMs exist, but they’re limited in size. So, when organizations want to offer the ability to answer queries without being able to see the conversation, they turn to TEEs. That’s a useful way to run a chatbot that’s reasonably private. This is what Apple, Google, WhatsApp, and others are doing.

Why not turn to encryption? After all, LLM inference is just a bunch of math like any other things a computer does. It takes input to a (really big) function and gives an output. We have the math to do that computation in a way that hides the inputs and outputs from the one running the computation, it's just super expensive. It’s called homomorphic encryption, and no one’s figured out how to do it fast enough that it makes sense for this sort of computation.

Instead, the allure of a TEE is that it will run that computation for you inside of a special opaque section of a server. TEE manufacturers try to make it as hard as possible for the person running the TEE to peek inside. But you still have to trust the operator to not put a stethoscope to the box to try to figure out what's happening inside. 

In this case, it’s reasonable to consider these systems “privacy-preserving,” but not “encrypted.” That distinction is important, especially when we talk about how the TEEs interact with secure messaging. When someone using an end-to-end encrypted chat app asks an LLM to summarize, review, or store those messages, the content of those messages is leaving the device and going to an unencrypted third-party server somewhere. That’s a major threat to the privacy of secure chat apps, and one that’s increasingly hard for users to take control of.

How Does This Translate to Practical Advice?

The answer to this is going to vary based on an individual’s threat model, but a good rule of thumb is that a user’s device should never automatically send data to a TEE. When the person holding a phone can choose what information is sent, even if it’s a chunk of data like “unread messages,” they have the opportunity to pause and consider if that data might be too sensitive to risk sending.

In contrast, when data is sent automatically, the automatic sending becomes a feature of the system as a whole. If the system was previously end-to-end encrypted, adding automatic exfiltration makes the whole system no longer end-to-end encrypted.

Developers: don’t build systems that automatically send data off a device to a TEE, especially when it’s coming from an app that is otherwise end-to-end encrypted.

Users: if developers ignore us and build that system, turn off any automatic data sending features. Take a second to think about how much you’re willing to risk sending data when you choose to send it off the device.

So, What Should I Be Concerned About?

TEEs are useful for security in a number of circumstances. Your phone likely has a TEE where it keeps the key that encrypts your biometric unlock data and the base of the keychain where passwords are stored. It also enables certain backup systems, like how you can restore a phone with your passcode or restore WhatsApp or Signal backups.

But when we’re talking about cloud processing, it’s important to be clear this isn’t the same as end-to-end encryption and doesn’t offer the same level of privacy. 

Most of our private lives are on our phones and in our messages. We’ve worked for years to secure those messages, with major wins like encrypted RCS, and the continued user experience improvements of Signal and WhatsApp. We’ve even seen real improvements to backup security with features like Advanced Data Protection that bring end-to-end encryption for a variety of data outside of messaging, like notes and photos. 

But as companies roll out AI features that interact with these encrypted services, pulling data off devices and into a cloud-based TEE, they’re eroding the privacy protections of end-to-end encryption and risk causing serious confusion around what data is protected and what isn’t.

With doomsday AI scenarios dominating the news, lawmakers are rightly concerned about reports concerning security breaches at major US AI labs, such as the OpenAI–Hugging Face incident and the many others reported in its aftermath. As they consider potentially regulating frontier AI, they should focus any new legislation on the immediate, demonstrated risks from those incidents. 

Post-incident reports show that the Hugging Face incident could have been mitigated or prevented by following longstanding cybersecurity best practices, like stronger sandboxing and monitoring. Any new legislation should focus on closing gaps in existing law to prevent AI companies from taking unreasonable risks with the public's security.

When an AI developer or deployer runs a test or a task that has a high likelihood of causing harm to third parties—for instance, by breaking into someone else's computers—there should be clear minimum safety requirements. Such tests should run in a properly sandboxed test environment, disconnected from other systems, and be monitored and logged. Following these fundamental best practices would have prevented or substantially mitigated all of the incidents at AI labs that we currently know about.

That said, any proposal must be flexible enough to evolve with changing technology. Minimum safety requirements specific only to current AI technologies are likely to become obsolete; legal standards tied to well-established cybersecurity best practices are far more likely to stand the test of time. Tying any new mandates to evidence-backed security protocols also protects the public without impeding future AI development.

Strong legislation should also mandate and fund independent third-party investigations into any serious security incidents that may occur during AI labs’ tests of new tools, and make reports of these investigations available to the public. This important transparency measure would go a long way toward providing public oversight of the industry.

As with any technology regulation, those targeting cybersecurity practices at AI labs must be careful, precise, and practical.