Because of the way they are trained, large language models capture only a slice of human language. They’re trained on the written word, from textbooks to social media posts, and our speech as captured in movies and on television. These models have minimal access to the unscripted conversations we have face to face or voice to voice. This is the vast majority of speech, and a vital component of human culture.

There’s a risk to this. The increased use of large language models means we humans will encounter much more AI-generated text. We humans, in turn, will begin to adopt the linguistic patterns and behaviors of these models. This will affect not just how we communicate with one another, but also how we think about ourselves and what goes on around us. Our sense of the world may become distorted in ways we have barely begun to comprehend.

This will happen in many ways. One of the first effects we could see is in simple expression, much as texting and social media have resulted in us using shorter sentences, emojis instead of words, and much less punctuation. But with AI, the impacts may be more harmful, eroding courteousness and encouraging us to talk like bosses barking orders. A 2022 study found that children in households that used voice commands with tools like Siri and Alexa became curt when speaking with humans, often calling out “Hey, do X” and expecting obedience, especially from anyone whose voice resembled the default-female electronic voices. As we start to prompt chatbots and AI agents with more instructions, we may fall into the same habits.

Next, in the same way autocomplete has increased how much we use the 1,000 most common words in our vocabulary, talking with chatbots and reading AI-generated text may further constrict our speech. A recent University of Coruña study found that machine-generated language has a narrower range of sentence length, averaging 12-20 words, and a narrower vocabulary than human speech. Machine-generated text reads as smooth and polished, but it loses the meanders, interruptions and leaps of logic that communicate emotion.

Additionally, because large language models are primarily trained from written speech, they may not learn how to emulate the free-wheeling nature of live, natural speech. When told “I hate Beth!”, ChatGPT replies with an uninterruptable three-part formula of affirmation (“That’s completely valid”), invitation (“I’m here to listen”) and invitation (“What’s going on?”) far longer than any reply plausible in face-to-face dialog. “What’s Beth’s deal?!” elicits a bullet point list of queries that reads like a multiple-choice exam question (“Is Beth * a celebrity? * a friend from school? * a fictitious character?”). No human speaks that way, at least not yet. But meeting such formulas repeatedly in a speech-like context may teach us to accept and use them, much as a child absorbs new speech patterns from spending time with a new person.

These influences will only increase with time. The writing large language models train on is increasingly produced by large language models themselves, creating a feedback loop in which they imitate their own inhuman patterns, even while teaching humans to imitate them too.

Broad use of large language models could also introduce confirmation bias, making us overconfident in our initial impulses and less open to other possible ideas—which is so vital to human discourse. Many chatbots are instructed to agree with our statements no matter how absurd, enthusiastically supporting half-formed or even incorrect notions and restating them as firm claims that we’re primed to agree with. When asked “Cake is a healthy breakfast, right?” or “Is the post office plotting against me?”, this sycophancy can reinforce bias and even worsen psychosis. And the hyperconfident tone of AI-produced writing will also heighten impostor syndrome, making our natural, healthy doubt feel like an aberration or failing.

In our experience as teachers, students who turn to generative AI for assignments often say they do so because they have trouble expressing what they think. The students don’t recognize that writing or speaking our thoughts is often how we realize what we think. Their unconfident and uncertain statements are actually the healthy human norm. But a large language model won’t turn vague first guesses into a well-formed critical analysis, or even ask helpful questions as a friend would; it will simply regurgitate those guesses, still unexamined, but in confident language.

We are also more vicious in social media posts and online chats than we are face to face. The well-documented online disinhibition effect encourages toxic language. Most of us have had the experience of venting ferocious rage about someone online, only to reconcile when we speak face to face or hear the warmth of a voice over the phone. While chatbots are trained to give sycophantic responses, they see humankind at our cruelest, learning about us from the only world where every flame war leaves an eternal written footprint, while the spoken conversations of forgiveness and reconciliation fade away. Their responses do not imitate our online aggression, but are still shaped by it, even in their rigid efforts to avoid it.

It’s easy to draw the wrong conclusions from a selective slice of a society’s communications. Medieval Norse sagas made us imagine a culture of mostly Viking warriors, since poets rarely described the farming majority. Chivalric romances focused on kings and courts, and long made us see the middle ages as a world of monarchies, erasing the many medieval republics. Statistically, we’ve been led to believe ancient Romans cared deeply about their republic, but 10% of all surviving Latin was written by one man, Cicero, whose work contains 70% of all surviving Roman uses of the word republic. Training language models on only certain human writings may introduce similar distortions. AI might make us seem more quarrelsome, as we are online. It might inflate the cultural significance of political topics primarily discussed on Twitter/X or Bluesky, or the massive topic-specific corpuses of LinkedIn and Goodreads.

Some large language models are being trained on human speech from movies and television shows, but that speech is still scripted, and disproportionately highlights certain contexts over others (for example, police dramas, fueled by stories of murder, make up a quarter of prime-time television programming). We are not funny or hurtful or romantic the same way in real life as we are in sitcoms. At least one startup is offering to pay people to record their phone calls for AI-training purposes, but this remains a niche idea; anything large scale would cause massive privacy concerns.

We don’t pretend to know what the best solutions might be. But one has to imagine if there’s ingenuity to develop AI models, then surely there’s ingenuity to come up with a way to train them on informal human speech instead of us only at our most stylized, veiled and sometimes worst. By excluding the overwhelming majority of language production on the planet—people talking, fully and naturally, to each other—these models are being trained to mirror everything but us at our most authentically human.

This essay was written with Ada Palmer, and originally appeared in The Guardian.

Last week, national security agencies from the Five Eyes—that’s the rich, English-language-speaking countries club—jointly released a statement warning of the increasing cyber risks of AI models: in particular, their ability to autonomously hack into systems and networks. The statement was more measured than some of the breathless headlines about it, and the advice they gave is pretty much the standard advice everyone gives—albeit with newfound urgency.

Internet risks are nothing new, and cyberattacks—both large and small—have been a significant issue since long before the current crop of generative AI models.

What’s been changing over the decades, and what AI is changing even faster, is the gap between skill and ability. For most of human history, the two terms were synonymous—but computers have decoupled them. As the gap between the two expands, humans empowered with these AI tools can do more: more writing, more research, more analysis and also more damage than ever before. These models can, with little detailed direction, autonomously hack into networks, steal data, deploy ransomware and destroy systems. And to the extent there is a solution, it’s going to involve harnessing AI for the defense.

In 1998, seven people from the hacker group L0pht testified before Congress. They told a mostly clueless Senate committee that they could take down the internet in 30 minutes. That was partly real and partly bravado, but it illustrates an important point: hacking into systems, stealing data and causing damage all required skill.

Contrast the L0pht hackers with hackers derided as “script kiddies.” They didn’t understand computers, or security. Instead, they used hacker tools written by others. Their actions required minimal skill and even less knowledge. But once those hacking tools became widespread, the number of potential attackers increased.

That number has continued to increase, as quality and availability of prewritten attack tools has grown. And it is growing dramatically with AI. Today’s AI systems—not just the frontier models, but most of them—are capable of carrying out cyberattacks automatically. They all do better in the hands of skilled attackers, but increasingly they are able to act autonomously with only minimal prompting.

The thing about people with ability but no skill is that they are often outsiders, not part of any professional community, and not bound by any rules or norms. This phenomenon is much more general than in cybersecurity. Any doctor can tell you how to untraceably poison someone, and many virus researchers know how to create a bioweapon. Any bridge engineer can tell you how to place explosives to blow a bridge up. The reason that murderous doctors and terrorist engineers are so rare is that the lengthy process of acquiring those skills also instills a moral and ethical code. If every random person has access to good poisoning advice, that puts us all in danger.

Modern AI systems are, in effect, a universal adviser to help people do harmful things. And while the current AI megacorporations are trying to build guardrails to prevent people from asking questions whose answers will enable the questioner to do harm, that’s not going to work in the long term. Smaller, cheaper, open-source models, including models that can run on people’s computers, and especially groups of models that run in concert with each other, are just as good as the frontier models from companies like OpenAI and Anthropic. And they continue to get better. These models will be passed around from person to person, like script kiddie hacker tools, and they won’t have any such guardrails.

Instructing AI models to spy on people and report any malicious prompts to the authorities fails for similar reasons. The megacorporations can do that, but the locally run open source models won’t. This could buy us a few months at best.

A third possibility is to somehow make the models themselves unable to hack into computers, create bioweapons or do anything else that might harm people or society. That won’t work, for the same reason we can’t teach doctors how to treat poisonings without also teaching them how to poison. It’s the same knowledge. It’s the same with construction and demolition. And it’s the same with cybersecurity. We want these AI models to be able to review computer code, find vulnerabilities and automatically fix them. The benefit to our collective security will be enormous. Unfortunately, the same knowledge can be used for attacks.

Where this leaves us is in a world of increased volatility. Super-powered humans with AI assistants will be able to do both wonderful and horrible things.

This brings us back to the Five Eyes statement. Everything they recommend is something security professionals have been recommending for years, if not decades. They are things talked about at that congressional hearing back in 1998, titled “Weak computer security in government: Is the public at risk?” Even the Five Eyes admitted that their security advice is not new, only more urgent.

What’s new is how fast things are changing: “The rapid pace of frontier AI development means cyber risk assumptions can become outdated in months, not years. We must act before and be prepared to adapt and withstand evolving threats.” The Five Eyes point to AI technology—not necessarily chatbots, but AI more generally—being used to strengthen every aspect of defense, to “detect vulnerabilities earlier, improve software quality, monitor unusual behavior, and respond faster to incidents—reducing both the cost and impact of incidents.”

Excellent advice from the Five Eyes security agencies. We need to do this with every risk that AI heightens, not just cybersecurity.

This essay was originally published in The Guardian.

Not sure this will have any effect, but I support the effort:

According to Google’s legal filing, Outsider Enterprise operates through Telegram. The group offers phishing-as-a-service to individuals who may not be technically savvy enough to set up fraudulent websites and text campaigns on their own. In its Telegram channels, Outsider Enterprise reportedly provided instructions on how to use Google’s Gemini AI to create websites that imitate those of Google, YouTube, and government agencies such as New York’s E-ZPass. The group offered nearly 300 scam templates.

[…]

Google worked with AT&T, Verizon, and T-Mobile to block many of these malicious text messages, and Google notes that its on-device scam detection in Google Messages probably helped reduce the number of successful phishing attempts, too. This AI-powered feature apparently stops 10 billion scam texts every month, so it’s fair to expect it caught at least some Outsider Enterprise activity.

Another article.

The Financial Times has a good article on how AI is changing the capabilities of video surveillance, with information from both Israel/Iran and Russia.

I wrote about this sort of thing a few years ago, how AI enables mass spying in the way that computers and networks enabled mass surveillance. The interesting development in the article is that AI allows people to ask natural language questions about video footage to AIs—and AIs can answer them.

In contrast with older tools restricted to a few dozen preset searches, these new tools allow an almost unlimited range of enquiries by enabling language-based searches on video.

That lets intelligence officers hunt through massive streams of videos using simple search terms, such as two men handing a bag to each other; a person who has changed their appearance, or has changed clothes multiple times in a day; or a vehicle that has recently been painted over, or has driven past the same spot several times in a short period.

“This is the holy grail of surveillance,” said a European official whose country uses the technology on its cities. “We are able to look for behaviour, not objects ­ it has created a world of new possibilities.”

Two kinds of work you want AI to do in a SOC

1. Work you want off your plate. Alert triage is the obvious example: every alert deserves a real investigation, most of them turn out to be noise, and they arrive at 3am as happily as at noon. Nobody wants help with this work. They want it gone. That’s the half Intezer has spent years building. Autonomous triage that investigates every alert at forensic depth, around the clock, and only interrupts a human when something actually needs human judgment.
2. Work you want to keep, but accelerate. Deciding what to do with an escalation. Writing the incident report. Picking apart the weird binary someone found on a build server. Chasing a hunch across five systems. For this work you don’t want a replacement. You want to be a 10x version of yourself.

Today we’re shipping the second half.

We rebuilt the Intezer MCP server from the ground up, and it turns the AI platform your team already lives in, Claude, Codex, Cursor, or any MCP client, into a full security workspace: your cases, your alerts, file and URL verdicts, live SIEM and EDR telemetry, tuning rules, all of it. We had an MCP server before, and it was a fine way to ask Intezer questions from a chat window. This one is built around a bigger idea: your AI workspace should be able to do everything you can do in Intezer, then combine it with everything else you have access to.

If you read our piece on making sense of the 2026 SOC stack, this release is the missing connection between the top two layers. Detection tools are the hardware. The AI SOC is the operating system that turns raw signals into investigated verdicts and institutional memory. AI platforms like Claude are the applications where people actually work. This release plugs the operating system into the applications.

Watch one investigation, end to end

The video walks through one escalated case, but the pattern behind it is the real story. Intezer’s autonomous triage investigates every alert to forensic depth and resolves what it can on its own. What lands in front of a human is the residue. Cases where the technical facts are settled but the decision still needs judgment, usually because it turns on business context no security tool can see. Was this data share authorized? Is this vendor one we actually work with? Escalating those isn’t a triage failure, rather it’s the line where execution ends and judgment begins.

Putting Intezer inside your AI workspace is what makes that handoff productive. Pick up a case in Claude, Codex, or Cursor and you inherit the full investigation Intezer already ran, plus its recommendation, with a partner that can reach the context security tools never had: your email, Slack, the ticket queue. You keep the decision; it does the legwork around you at machine speed, pulling the case, cross-referencing your systems, documenting the verdict, writing a tuning rule. What used to be an afternoon of pivoting between consoles becomes a short, supervised exchange.

That’s the point of the combination: the autonomous half absorbs the scale, the assistive half carries the judgment, and every call you make feeds back as logic that makes the autonomous half smarter. You’re not handing off your work; you’re making judgment calls with the context, evidence, and follow-through already assembled around you.

The same question, with and without Intezer

Same alert, two ways to handle it. On the left, Claude on its own takes the impossible-travel sign-in and works it by hand. It reasons well and gets close — managed device, MFA passed, probably real travel — but it can’t collect evidence from the endpoint to confirm, so the last step falls back to a human checking the laptop. And that’s one alert; almost 4,000 more are still waiting behind it. One analyst, one alert at a time, with no way to run it across the whole team. On the right, the same alert inside the AI SOC: Intezer triages every alert around the clock, closes the ~98% that need no action, and escalates only the ~2% that genuinely need a person. Claude is where you pick those up so you can stop grinding the queue and start supervising the few cases that actually need you.

Most of the org knowledge an investigation needs is already centralized in Intezer. That’s the whole point of the platform. But some context only ever lives with you: a procurement thread in someone’s inbox, a Slack message from three weeks ago, a calendar invite. With Intezer connected on one side and your IT and communication stack on the other, your AI workspace can cross-reference both in a single investigation.

Why not plug Claude into all security tools directly?

You could also wire your AI client straight into each security tool yourself. Most of them ship an MCP these days. Two things make that a worse deal than it looks. First, the integration work is now yours: stitching a dozen connectors together, learning each product’s query quirks, and getting back a pile of disconnected results instead of one correlated picture. Second, raw tool access still isn’t investigation. With every EDR, SIEM, and intel feed wired in, the model can read your data, but it can’t collect evidence off an endpoint, run memory forensics, or weigh conflicting signals into a verdict it will actually stand behind, which is exactly where Claude stalls on the left in above image.

Intezer already did both jobs. One connector hands the model a SOC’s worth of normalized cases, verdicts backed by real forensic evidence, and cross-tool correlation. An AI platform does its best work standing on a real foundation of security knowledge, not on a dozen raw feeds it has to assemble itself.

Investigate and close the cases Intezer escalates to you

This is where analyst hours should go, so it’s where the MCP goes deepest. Whatever the alert type, the shape is the same: pull the case, build on everything the autonomous triage already found, cross-reference your other systems, decide interactively with you, and close with evidence.

And “pull the case” carries real weight here. A case from Intezer is not a bare ticket. It arrives with everything triage already did: the evidence it collected, the SIEM and EDR queries it ran, the forensic analysis of each artifact, the verdicts it reached. You’re not starting from a blank page; you’re picking up a deep investigation and taking it the last mile.

“Pick up the oldest escalated open case and let’s investigate it together.”

The clip above takes an impossible-travel alert. The MCP brings the full login history including every IP and geo, and who else touched the same address as well as your AI workspace cross-references calendar and Slack for travel context. When the evidence still isn’t conclusive, it can ask the user directly and close on their answer, so the one human check that actually mattered takes seconds instead of becoming a follow-up ticket.

Make tomorrow’s autonomous triage smarter

If a case should never have reached you, closing it is half the job. The other half is making sure it never reaches you again.

“We keep getting this exact false positive. Write a tuning rule so it never escalates again, then retriage the case.”

Claude inspects the alert’s triage indicators, drafts a narrowly scoped tuning rule, and tests the pattern against the real alert object before proposing anything. It checks whether an existing rule should be extended instead of creating a near-duplicate. It asks the question every detection engineer should ask: could an attacker hide inside this rule? Then it pushes the rule to Intezer for your approval and retriages the affected alerts so the fix applies immediately.

Tuning runs both directions, too. The same mechanism can tell the autonomous triage to always escalate a pattern it can’t yet call malicious with confidence, so the genuinely ambiguous cases land in front of a human by design, not by luck.

This is where the two halves of the AI SOC meet. Every rule your AI workspace writes makes the autonomous half smarter, which means fewer escalations next month, which means the time you spend supervising keeps shrinking. The system compounds.

From case to incident report in one prompt

When a case turns into a real incident, the hours after containment go to reconstruction: which alerts were related, which machines were touched, what happened first, and what to tell leadership.

“Write an incident report for the latest case we worked on — timeline, affected assets, and an exec summary I can send to the CISO.”

Your AI workspace pulls the case and its full activity trail from Intezer, expands across the users, devices, and IPs involved, and rebuilds the timeline from the forensic evidence already on file. Then it writes the report with an executive summary up top, technical detail below, in your template if you have one, and finally exports it to a clean, brand-styled PDF you can send as-is. The data was always in Intezer; the report was just assembly. Now assembly is one prompt.

Threat hunting: start from a lead, not an alert

Not every investigation starts in the queue. Sometimes it starts with your CISO forwarding an article about a campaign that’s hitting your industry.

“Here’s a writeup of a new campaign [link]. Check whether any of these IOCs appear anywhere in our environment, and analyze anything you find.”

Your AI workspace extracts the indicators and techniques from the writeup, sweeps your environment through Intezer’s SIEM and EDR query tools, and returns the matching assets, alerts, and artifacts for analysis. When you find something worth a closer look, you can fire deep forensics to go one step further with your hunt.

How it works

The Intezer MCP server is hosted by us. You authorize over OAuth from any MCP client: Claude (Desktop, Code, or claude.ai), ChatGPT, Codex, Cursor, or anything else that speaks the protocol.

Under the hood it exposes 66 tools covering the full case lifecycle: search and fetch cases and alerts, file and URL analysis, live queries against more than a dozen SIEM and EDR products in their native query languages (KQL, SPL, XQL, SDL, and the rest, with per-vendor syntax guides built in so the model gets them right), tuning rules and AI instructions, retriage, and case editing.

This architecture is what makes the two halves described above work as one system: the autonomous half clears work off your plate, while the assistive half accelerates the tasks where you still want to stay in the loop.

Getting started

1. If you’re already an Intezer customer, an Intezer admin creates an MCP OAuth application under Account Settings → MCP OAuth Applications.
2. Add Intezer as a custom connector in your AI client such as Claude, ChatGPT, or any MCP client. Point it at the hosted server, and authorize with your own Intezer login over OAuth.
3. Open with one prompt: ask it to pick up your oldest open escalation.

The autonomous half investigates everything, around the clock, so your team only sees what matters. The assistive half makes the time you spend on what matters dramatically shorter. One system of record and detection logic underneath both: your cases, your verdicts, your tuning rules, your institutional memory, working for you whether the investigation runs inside Intezer or inside your AI workspace.

AI executes. Humans supervise. And now the supervising got a lot faster too.

If you’re not an Intezer customer yet, book a demo and we’ll show you both halves at once: autonomous triage working every alert around the clock, and a co-pilot that helps your analysts close the escalations that do reach them 10x faster.

The post The other half of the AI SOC: Intezer, now inside your AI workspace appeared first on Intezer.

What we built, Fusion AI, runs at about a third the cost of a traditional external pentest, a human tester still signs off on every finding, and it is not here to replace anybody.
We have been hearing that one a lot. So when Melisa from our Business Capture team sat down with Brian Fehrman and me for this episode of AI Security Ops, she started with, “What is this thing you built, and is it the same hype everyone else is selling?”

The post Everyone’s Selling AI That Kills Pentesting. We Built One That Doesn’t. appeared first on Black Hills Information Security, Inc..

On 14 April, the Trump administration quietly acknowledged the widespread use of AI to automate government processes. The office of management and budget (OMB) disclosed a staggering 3,611 active or planned use cases for AI across the federal government. The list has ballooned by 70% from the one published in the final year of the Biden administration, and includes many disturbing-seeming plans to hand over sensitive governmental functions to AI.

Scanning this list, many readers may find many causes for alarm. It represents a transfer of decision processes from human to machine on a massive scale over matters of individual freedom, public health and well-being, nuclear reactor safety and more.

Consider these examples. The Health and Human Services’ (HHS) office of administration for children and families hired the world’s “scariest AI company,” Palantir—notorious for its work on behalf of the military, the CIA and ICE—to scan all grant applications to flag those not ideologically aligned with the administration’s dictates. The Federal Bureau of Prisons is developing an AI system to assess the “potential for misconduct for newly admitted inmates,” routing people into high-security confinement before they have actually done anything wrong in their custody. These read like programs fit for a Philip K Dick or George Orwell novel.

Other use cases insert AI into life-and-death decision making. The Department of Veterans Affairs is developing an AI that will listen in on calls to the veterans crisis line, and then gather information from external databases to assess the mental state and suicide risk of the caller.

The Department of Energy is testing the use of AI to control nuclear reactors, targeting a way to autonomously respond to potential nuclear safety incidents. Here’s one that’s disturbing for its retirement, rather than its deployment: the state department has ended a program to use AI to forecast mass civilian killings, which had been intended to aid conflict prevention.

While it’s easy to raise questions about these and similar uses of AI, the reality is that any of these programs could be implemented responsibly. In some cases, like the HHS system, the AI might be enforcing alignment to a policy prescription that opponents abhor. But that concern is more about the policy itself rather than the idea that agencies should comply with executive orders.

In other cases, there may even be bipartisan agreement on the goal, like taking urgent action to help veterans at risk of self-harm. Lots of work and validation is needed to prove AI safe and effective for these use cases and convince the public it is appropriate, but the idea is plausible.

In other cases, a scary-sounding AI use may not even be new. The use of predictive methods and statistics to assign prisoner security classifications goes back decades, even if such systems are often biased and ineffective.

Using autonomous systems for model predictive control (MPC) of nuclear reactors is a well studied, and a widely applied aspect of nuclear plant management. And the recently disclosed addition of AI was initiated under the Biden administration.

But anyone reviewing the 2025 inventory could be forgiven for leaping to severe conclusions. What matters are the details of how the AI system is used, and here the inventory is severely lacking.

The disclosures carry minimal information, and lack the context necessary to understand their purpose and approach. The descriptions are typically just a sentence, and rarely more than a paragraph.

And while the process theoretically involves some form of public consultation, in reality there is generally none. It would take an eagle-eyed citizen to even come across this disclosure. Unless you read FedScoop regularly, or watch the OMB’s federal chief information officer’s GitHub account, you probably missed it.

Only one of the examples cited above (the DoJ) even proposes to involve the public. Under the administration’s policy, it’s not required for the rest because they are not classified as “high impact” use cases—a label that is applied inconsistently across agencies.

We wrote a book surveying applications of AI to democratic processes worldwide, including executive agencies as well as the courts, legislatures and politics. Our conclusion was that, while there are inappropriate applications of AI in governance that should be resisted, an urgent need to reform the economics of AI, and an imperative for renovating the democratic systems it is being unleashed on, there are also valuable and beneficial use cases for AI in government.

Machine translation is a good example. Customs and Border Protection (CBP) has deployed an AI translation system to help officers when human interpreters are not available. The idea that CBP, an agency under heavy scrutiny for reported abuses of human rights, would direct people to talk to a machine instead of a person may strike many as inhumane.

It’s true that human interpreters have very real advantages when it comes to understanding nuance from physical cues and social context. But an officer with a competent AI translator available immediately is better than one who cannot communicate with the person in front of them.

The Trump administration’s AI use case inventory has 70 such translation use cases, up from 58 in the Biden administration’s 2024 disclosure.

Disclosure of AI use cases could be a means to build public confidence and trust, but only if paired with consistent, meaningful public consultation. Washington DC and California are actively engaging the public to determine where and how it’s appropriate to use AI in government processes, or for government to regulate AI use in society.

Both have held public deliberations on this topic at a wide scale, using AI platforms. These examples demonstrate the potential for capturing broad-based public input to steer AI policy.

The international gold standard was arguably set by the French in 2016, via their Digital Republic Act. The law, itself informed by an online citizen consultation, requires all algorithms used to automate government administrative decisions to be subject to public records requests, to be appealable to a human reviewer, and to have mandatory notification of the use of automation to those affected by the decisions.

Canada offers another example of what more rigorous and participatory disclosure might look like. In 2025, they launched an AI use case registry, not unlike the US inventory. However, Canada also has a federal directive mandating a transparent risk-scoring and impact assessment process for automated systems that make administrative decisions about citizens.

That longstanding directive requires a detailed explanation of risks and benefits as well as consultation with certain stakeholders from the conception of the AI use case. The Canadian system could be improved; it could require a public comment period and an obligation for agencies to respond substantively to feedback before engaging in sensitive uses of AI.

AI offers real potential to improve the efficacy, efficiency and accessibility of government. But, equally, there is legitimate reason for public concern and distrust that can only be addressed through transparency and dialog. The US should adopt, at the federal and state level, algorithmic impact risk assessment procedures and public comment processes to facilitate a safe, trusted, equitable transformation of government agencies to take advantage of modern technology.

This essay was written with Nathan E. Sanders, and originally appeared in The Guardian.

Second part of the AI vs Traditional Pentesting series, focusing this time on tools and outcomes of both approaches.

The post AI vs Traditional Penetration Testing: Tooling and Outcomes appeared first on OffSec.

Thanks to Uncle Sam, anyone trying to find nonconsensual intimate deepfakes on CFake.com and SOCFake.com will be disappointed. The US Departments of Justice (DOJ) and Homeland Security has seized the two domain names under the TAKE IT DOWN Act.

The TAKE IT DOWN Act, signed in May 2025, is the first US federal statute criminalizing the publication of nonconsensual intimate imagery, including AI-generated forgeries. It imposes penalties of up to two years’ imprisonment, gives covered platforms 48 hours to remove flagged content, and grants the forfeiture powers the DOJ just used.

According to the seizure warrants, the digital forgeries depicted “politicians, first ladies of multiple countries, royalty, journalists, television presenters, athletes, entertainers, and others,” and visitors could browse them under tags including “rape,” “forced,” and “degradation”.

The authorities didn’t just snag the sites, though. They got the alleged operator of CFake.com, in an international effort.

The US alerted the Paris prosecutor’s office to a French national in Nice who was allegedly running CFake.com. French investigators counted roughly 300,000 images and 7,000 videos depicting 14,000 people across CFake.com, drawing four million monthly views from 200,000 user accounts.

They then arrested the IT professional, who had no prior criminal record. They also found around $64,000 in Ether cryptocurrency at his home in advertising revenue from the site.

The man will be tried on July 7 in Paris for carrying out illicit transactions online and providing nonconsensual sexual deepfakes. The former offence carries a potential seven years’ imprisonment and a €500,000 (approximately $580,000) fine. The latter could yield three years and a €75,000 ($87,000) fine.

Providers and accused providers of nonconsensual intimate deepfakes have also been held in the US. In April, James Strahler II from Ohio pleaded guilty to cyberstalking, producing child sexual abuse material, and publishing digital forgeries.

Strahler had downloaded produced over 700 images and animations posted to a child sexual abuse site, and had sent deepfake material to at least six adult women, including one sent to a victim’s coworkers.

Last month, the DoJ also arrested Cornelius Shannon and Arturo Hernandez under the TAKE IT DOWN Act for publishing thousands of deepfake images of prominent women and those not in the public eye.

Other countries are also taking action. Anthony Rontondo was arrested by Australian authorities in May last year for posting deepfaked pictures of prominent Australian women. He eventually received an AU$343,000 fine.

How prevalent are deepfakes?

These seizures and prosecutions are encouraging, but prosecutors trying to force non-consensual deepfakes offline face a rising tide of such material. Requests for and sharing of nonconsensual deepfake imagery have risen, with activity migrating across platforms. Deepfake incidents overall jumped 257% in 2024, and girls accounted for 94% of victims in reported AI-generated child sexual abuse cases.

Seizing a distribution point removes a storefront. It does not remove the AI models used to produce the material, the anonymous hosting providers downstream, or the demand that draws visitors in the first place.

What you can do

If you or someone you know are depicted in a nonconsensual deepfake, keep dated screenshots, URLs, and any communications as evidence before filing a takedown request and reporting it to the authorities.

Limit the high-resolution face images you and your children post publicly, since school portraits and social media profile pictures are the raw material these tools need.

Take advantage of expert advice to help protect yourself from non-consensual deepfakes:

• Cyber Civil Rights Safety Initiative Safety Center (US)
• Revenge Porn Helpline (UK)
• Cybertip.ca (Canada)

---

Let’s face it, an incognito window can only do so much. 
 
Breaches, dark web trading, credit fraud. Malwarebytes Identity Theft Protection monitors for all of it, alerts you fast, and comes with identity theft insurance. 

Thanks to Uncle Sam, anyone trying to find nonconsensual intimate deepfakes on CFake.com and SOCFake.com will be disappointed. The US Departments of Justice (DOJ) and Homeland Security has seized the two domain names under the TAKE IT DOWN Act.

The TAKE IT DOWN Act, signed in May 2025, is the first US federal statute criminalizing the publication of nonconsensual intimate imagery, including AI-generated forgeries. It imposes penalties of up to two years’ imprisonment, gives covered platforms 48 hours to remove flagged content, and grants the forfeiture powers the DOJ just used.

According to the seizure warrants, the digital forgeries depicted “politicians, first ladies of multiple countries, royalty, journalists, television presenters, athletes, entertainers, and others,” and visitors could browse them under tags including “rape,” “forced,” and “degradation”.

The authorities didn’t just snag the sites, though. They got the alleged operator of CFake.com, in an international effort.

The US alerted the Paris prosecutor’s office to a French national in Nice who was allegedly running CFake.com. French investigators counted roughly 300,000 images and 7,000 videos depicting 14,000 people across CFake.com, drawing four million monthly views from 200,000 user accounts.

They then arrested the IT professional, who had no prior criminal record. They also found around $64,000 in Ether cryptocurrency at his home in advertising revenue from the site.

The man will be tried on July 7 in Paris for carrying out illicit transactions online and providing nonconsensual sexual deepfakes. The former offence carries a potential seven years’ imprisonment and a €500,000 (approximately $580,000) fine. The latter could yield three years and a €75,000 ($87,000) fine.

Providers and accused providers of nonconsensual intimate deepfakes have also been held in the US. In April, James Strahler II from Ohio pleaded guilty to cyberstalking, producing child sexual abuse material, and publishing digital forgeries.

Strahler had downloaded produced over 700 images and animations posted to a child sexual abuse site, and had sent deepfake material to at least six adult women, including one sent to a victim’s coworkers.

Last month, the DoJ also arrested Cornelius Shannon and Arturo Hernandez under the TAKE IT DOWN Act for publishing thousands of deepfake images of prominent women and those not in the public eye.

Other countries are also taking action. Anthony Rontondo was arrested by Australian authorities in May last year for posting deepfaked pictures of prominent Australian women. He eventually received an AU$343,000 fine.

How prevalent are deepfakes?

These seizures and prosecutions are encouraging, but prosecutors trying to force non-consensual deepfakes offline face a rising tide of such material. Requests for and sharing of nonconsensual deepfake imagery have risen, with activity migrating across platforms. Deepfake incidents overall jumped 257% in 2024, and girls accounted for 94% of victims in reported AI-generated child sexual abuse cases.

Seizing a distribution point removes a storefront. It does not remove the AI models used to produce the material, the anonymous hosting providers downstream, or the demand that draws visitors in the first place.

What you can do

If you or someone you know are depicted in a nonconsensual deepfake, keep dated screenshots, URLs, and any communications as evidence before filing a takedown request and reporting it to the authorities.

Limit the high-resolution face images you and your children post publicly, since school portraits and social media profile pictures are the raw material these tools need.

Take advantage of expert advice to help protect yourself from non-consensual deepfakes:

• Cyber Civil Rights Safety Initiative Safety Center (US)
• Revenge Porn Helpline (UK)
• Cybertip.ca (Canada)

---

Let’s face it, an incognito window can only do so much. 
 
Breaches, dark web trading, credit fraud. Malwarebytes Identity Theft Protection monitors for all of it, alerts you fast, and comes with identity theft insurance. 

Unit 42 discovered a Vertex AI Python SDK vulnerability that allows remote code execution via bucket squatting. Read the article for more.

The post Pickle in the Middle – Hijacking Vertex AI Model Uploads for Cross-Tenant RCE appeared first on Unit 42.

The concept of a completely autonomous security operations center (SOC) — where data collection, analysis of suspicious events, investigations, and incident response happen without human intervention — is extremely compelling. This is especially true for organizations grappling with a chronic shortage of cybersecurity talent and a threat landscape that’s growing faster and more sophisticated by the day. Organizations everywhere would welcome an approach where automation helps relieve analyst workloads, shortens alert triage times, and finally eliminates the backlog of unaddressed alerts — which, by some estimates, accounts for 67% of all security events in the average corporate SOC.

While many vendors are already pitching solutions in this space, real-world implementation remains highly problematic. Practitioners report tangible success when using these tools for alert enrichment and filtering out low-priority noise or false positives. However, when it comes to autonomous decision-making and response, very few organizations have managed to achieve a meaningful return on investment.

Foundational roadblocks of an autonomous SOC: looking beyond AI

While leveraging AI for data analysis and decision-making sounds like a logical and relatively easy-to-implement idea, actually putting it into practice exposes and amplifies the exact same challenges organizations faced with SIEM, XDR, and SOAR platforms:

Source data quality. Issues with coverage, enrichment quality, tagging and normalization, which detection engineering teams in every SOC battle daily, become even more acute when AI is introduced. AI agents are more sensitive to data gaps than human analysts, so incomplete data can magnify the resulting errors.

Data consolidation and tool integration. The very problem SIEM was once invented to solve remains a headache for most organizations today. Interestingly, marketing for AI-driven SOCs often claims that “the SIEM is dead” because “agents can just query the EDR directly for telemetry”. In reality, however, even in a best-case scenario, this just means the SIEM disappears as a user interface while its core functions remain embedded within the data fabric of the agentic SOC.

Analysts’ trust. Even when AI is restricted to preliminary data gathering and recommendations, human analysts frequently don’t trust the output, leading them to waste time re-collecting and re-analyzing the same data. Practitioners frequently point to several flaws in current AI SOC implementations: poor handling of gray-area verdicts (when an alert is suspicious but not definitively malicious), lack of safe escalation workflows, and systems that fail to learn when a human analyst corrects their mistakes.

Context deficit. SOCs and security teams in general naturally rely on scantily documented information, such as business context and tribal knowledge, to accurately assess alerts and incidents. It’s very difficult to populate an AI system with that knowledge in a systematic way.

AI-specific issues critical for a SOC

Beyond traditional operational hurdles, fully autonomous SOCs face inherent flaws deeply rooted in the fundamental architecture of language models and AI agents.

Hallucinations and prompt injections. In a SOC environment, a single manipulated log field can easily become a viable exploit vector aimed directly at the agent. In a semi-autonomous setup, an AI hallucination is just a frustrating distraction that erodes analyst trust. In a fully autonomous SOC, however, a hallucination can trigger instantaneous, harmful actions across hundreds or thousands of endpoints simultaneously. A prime example of this risk is the widely cited incident at a Fortune 50 company, where an AI agent went rogue and rewrote access policies on its own.

Need for control. To combat hallucinations and over-automation, organizations typically rely on a human-in-the-loop (HITL) model to approve an agent’s actions. While this improves safety, it completely defeats the primary selling point of agentic AI: response times.

Compliance, audits, and accountability. The inherently stochastic nature of LLM outputs makes logging problematic. They often lack reproducibility and explanations. Consequently, an autonomous SOC will likely struggle to pass regulatory compliance audits. Simply put, current compliance frameworks were never designed to handle the unpredictable behavior of multiple interacting AI agents.

Strategies to overcome the challenges of an autonomous SOC

Specialized frameworks are emerging to address these built-in flaws of AI agents and language models. For the most part, these solutions focus on enforcing formal boundaries around AI privileges, and validating its actions.

Rigorous context engineering. Assuming source data is correct and properly enriched, the number of hallucinations can be minimized, and agent decision quality significantly improved by feeding the language model structured layers of context — such as alerts, user accounts, asset data, and enrichment data.

Narrowing the scope of work. AI agents are less likely to go off the rails when confined to highly repetitive, narrow tasks. For example, an “agent for collecting additional host data” is going to be more effective than an “autonomous threat hunter”.

Neurosymbolic validations and guardrails for agent actions. An Agent-Lock pipeline cleans untrusted log fields, and verifies proposed actions against existing CMDB/IAM policies. This approach enforces key rules, such as making it impossible for the AI to disable telemetry, while managing “autonomy budgets”.

Tiered autonomy over all-or-nothing automation. The Trusted Autonomy framework maps out progressive levels of AI independence based on human-in-the-loop roles and trust thresholds across monitoring, detection, and response. Low-risk operations like data enrichment and alert deduplication run fully automated, while high-blast-radius actions require mandatory human approval.

Governance-first architecture. The LanG platform, which utilizes a hierarchical approach: Governance → MCP → Agentic AI → Security, is one example. It enforces two mandatory human analyst check-ins, fully aligning the workflow with NIST SP 800-61 guidelines. The trade-off, however, is that this framework significantly scales back the solution’s autonomy.

Deterministic execution for high-risk actions. Triage and investigation are handled by a probabilistic AI model, but high-impact actions — like deciding to isolate a host or terminate a session — are based on deterministic code. This approach allows the system to satisfy the strict requirements of SOC 2 and other major regulatory frameworks.

Stateful admission control. For example, the recently proposed ACP protocol monitors behavioral patterns across agent execution logs. This makes it possible to catch rogue agents that are executing a series of individually harmless requests that add up to a coordinated attack.

Key takeaways and pitfalls

We can already confidently state that an autonomous SOC is highly unlikely to bring any improvements for organizations burdened by significant technical and operational debt in areas like data collection and enrichment or standardized incident response workflows. No layer of AI infrastructure will function without that baseline foundation firmly in place.

It’s also clear that, while AI streamlines analyst workflows, it doesn’t completely replace them. This is why Gartner’s prediction that there will never be an autonomous SOC still rings true in 2026. Deploying autonomous agents into the SOC shifts the center of gravity to complex investigations, but most importantly, to complex engineering. Teams will simply trade fine-tuning detection rules for managing AI agent playbooks, data pipelines, and decision-handling workflows.

For mature SOCs, the core hypothesis for the next one to two years is this: an autonomous SOC should be viewed as a direction rather than a destination. AI is already delivering tangible value today — specifically in correlation, enrichment, draft detection rules, and attack reconstruction — provided that each capability has proper security guardrails. These include a well-balanced human-in-the-loop review process for any action that impacts production environments. Security teams investing now in a structured, verifiable approach — one that actively anticipates emerging regulations — will be able to gradually integrate new agentic features into their SOC pipelines. Conversely, organizations that skip this layer will almost certainly run into roadblocks, likely forcing them to rebuild their systems and processes from the ground up.

Anthropic has been ordered by the US government to cut off its newest Claude Fable 5 and Mythos 5 models for fear of abuse by adversaries.

Reuters reports that Anthropic said it will “abruptly ​disable” its most advanced AI models for all users after the US government ordered it to suspend access to the models for foreign nationals, citing national security ‌concerns.

Officials reportedly believe a jailbreak could turn Fable 5 and Mythos 5 into vulnerability-discovery tools for adversaries, so Anthropic says it is disabling them worldwide rather than try to nationality‑filter access, since it is virtually impossible to verify every user’s nationality.

In a statement on its website, Anthropic says:

“The letter did not provide specific details of its national security concern. Our understanding is that the government believes it has become aware of a method of bypassing, or “jailbreaking” Fable 5. We reviewed a demonstration of this specific technique being used to identify a small number of previously known, minor vulnerabilities. These vulnerabilities all appear relatively simple, and we have found that other publicly-available models are able to discover them as well without requiring a bypass.”

Mythos 5 is the non-public full version, which is currently used only by government agencies and selected corporate partners to harden their systems. Fable 5 is a Mythos-class model that should supposedly be safe for general use.

It makes sense to me that if Fable 5 is easy to jailbreak, that it should fall under the same restrictions as Mythos 5. However, Anthropic maintains that it has built-in safeguards that mean queries on some topics will instead receive a response from the next-most-capable model, Claude Opus 4.8. 

The relationship between the US government and Anthropic had shown signs of easing in parts of the US government after tensions over military use, surveillance, and autonomous weapons. In March, defense Secretary Pete Hegseth designated the San Francisco-based company a “supply-chain risk to national security.”

To understand the nature of the argument, it is necessary to understand that Mythos 5 is described in multiple reports as particularly effective at identifying software vulnerabilities, including long‑standing bugs in complex, legacy systems such as those in banking and other critical infrastructure. Many view this as dual‑use: great for defense hardening, but catastrophic in the wrong hands.

In recent updates from major software vendors like Microsoft and Google, we’ve seen a growth in numbers of patched vulnerabilities after the vendors began using AI-guided search for new vulnerabilities in their own software. We also know that Mozilla found over 270 Firefox vulnerabilities with the aid of Anthropic’s new Claude Mythos model. 

What this means

In the wrong hands these vulnerabilities could definitely do a lot of harm. So, it looks like it will take some time before regular consumers and developers will gain access to Fable 5 and Mythos 5 entirely. However, existing Anthropic models (older Claude variants) remain available.

For home users who were simply chatting with Claude or using it to help with basic scripting, the change will mostly show up as “this specific version is unavailable” rather than a broader AI blackout.

Removing a high‑end vulnerability‑finding model from broad circulation increases the effort required for less‑resourced cybercriminals to automate discovery of complex bugs in consumer‑facing software and services only by so much. There are other models available on the black market that might be just as effective. And for most cybercriminals, turning a vulnerability into a method they can utilize in an exploit is much more relevant.

---

We don’t just report on threats—we remove them

Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.

Anthropic has been ordered by the US government to cut off its newest Claude Fable 5 and Mythos 5 models for fear of abuse by adversaries.

Reuters reports that Anthropic said it will “abruptly ​disable” its most advanced AI models for all users after the US government ordered it to suspend access to the models for foreign nationals, citing national security ‌concerns.

Officials reportedly believe a jailbreak could turn Fable 5 and Mythos 5 into vulnerability-discovery tools for adversaries, so Anthropic says it is disabling them worldwide rather than try to nationality‑filter access, since it is virtually impossible to verify every user’s nationality.

In a statement on its website, Anthropic says:

“The letter did not provide specific details of its national security concern. Our understanding is that the government believes it has become aware of a method of bypassing, or “jailbreaking” Fable 5. We reviewed a demonstration of this specific technique being used to identify a small number of previously known, minor vulnerabilities. These vulnerabilities all appear relatively simple, and we have found that other publicly-available models are able to discover them as well without requiring a bypass.”

Mythos 5 is the non-public full version, which is currently used only by government agencies and selected corporate partners to harden their systems. Fable 5 is a Mythos-class model that should supposedly be safe for general use.

It makes sense to me that if Fable 5 is easy to jailbreak, that it should fall under the same restrictions as Mythos 5. However, Anthropic maintains that it has built-in safeguards that mean queries on some topics will instead receive a response from the next-most-capable model, Claude Opus 4.8. 

The relationship between the US government and Anthropic had shown signs of easing in parts of the US government after tensions over military use, surveillance, and autonomous weapons. In March, defense Secretary Pete Hegseth designated the San Francisco-based company a “supply-chain risk to national security.”

To understand the nature of the argument, it is necessary to understand that Mythos 5 is described in multiple reports as particularly effective at identifying software vulnerabilities, including long‑standing bugs in complex, legacy systems such as those in banking and other critical infrastructure. Many view this as dual‑use: great for defense hardening, but catastrophic in the wrong hands.

In recent updates from major software vendors like Microsoft and Google, we’ve seen a growth in numbers of patched vulnerabilities after the vendors began using AI-guided search for new vulnerabilities in their own software. We also know that Mozilla found over 270 Firefox vulnerabilities with the aid of Anthropic’s new Claude Mythos model. 

What this means

In the wrong hands these vulnerabilities could definitely do a lot of harm. So, it looks like it will take some time before regular consumers and developers will gain access to Fable 5 and Mythos 5 entirely. However, existing Anthropic models (older Claude variants) remain available.

For home users who were simply chatting with Claude or using it to help with basic scripting, the change will mostly show up as “this specific version is unavailable” rather than a broader AI blackout.

Removing a high‑end vulnerability‑finding model from broad circulation increases the effort required for less‑resourced cybercriminals to automate discovery of complex bugs in consumer‑facing software and services only by so much. There are other models available on the black market that might be just as effective. And for most cybercriminals, turning a vulnerability into a method they can utilize in an exploit is much more relevant.

---

We don’t just report on threats—we remove them

Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.

This week on the Lock and Code podcast…

If you weren’t taking deepfakes seriously before, it’s too late now to ignore them.

According to new research from Malwarebytes, one in three people who use AI every day said it’s okay to generate pornography of people without their consent.

Nearly 10 years ago, “deepfake” technology provided hobbyists and film editors with artificial intelligence (AI) tools to swap the face of one person onto the body of another. In its infancy, this technology brought silly film experiments like swapping Tom Cruise in Mission Impossible with Keanu Reeves. Today, this same technology produces something far more harmful—fake nude images of teenagers.

On the Lock and Code podcast today with host David Ruiz, we are re-visiting an interview from 2024, in which we spoke with a lawyer named David Chiu about his lawsuit against 16 deepfake nude generation websites.

The websites named in that lawsuit often needed just one image of a person to generate fake pornography. And while nearly everyone has at least one image of themselves online, even if they had hundreds, the path towards deletion is somewhat understood—start by deactivating and deleting popular social media accounts. But for teenagers today, raised mostly online, and who share images directly with friends and boyfriends and girlfriends and exes, it’s likely impossible to remove every visual trace of themselves. Also, they shouldn’t have to face this problem alone.

The Lock and Code podcast frequently discusses structural problems that require individual management. You have to skirt corporate data collection. You have to find the automated license plate readers in your hometown. You have to review every single message you get with a certain antagonism, to guard yourself against scams.

So, it’s rare to encounter a solution that benefits more than one person.

Chiu serves as the City Attorney for San Francisco, which means his department can file a lawsuit on behalf of not just the people of San Francisco, but also California, and that’s what his team did in going after the deepfake websites.

Since then, Chiu’s department has shut down 10 deepfake nude websites, and it received a settlement agreement from a company called Briver LLC to no longer operate any website that creates nonconsensual deepfake pornography.

And, as California goes, so goes the nation.

In May of last year, the Take It Down Act became effective as law in the United States, which criminalizes “revenge porn” and AI-generated nonconsensual intimate imagery. The law is not perfect but so far it is being used as intended. Last month, two men in the US were among the first to be charged with violating the Take It Down act for allegedly creating deepfake nudes that, according to the AP, “included both celebrities as well as private women, including recent high school graduates.”

Today, we revisit our conversation with San Francisco City Attorney David Chiu about the important fight against deepfake porn and the clear threat that his department found against the public.

“At least one of these websites specifically promotes the non-consensual nature of this. So, and I’ll just quote, ‘Imagine wasting time taking her out on dates when you can just use website X to get her nudes.'”

Tune in today to listen to the full conversation.

Show notes and credits:

Intro Music: “Spellbound” by Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
http://creativecommons.org/licenses/by/4.0/
Outro Music: “Good God” by Wowa (unminus.com)

---

Listen up—Malwarebytes doesn’t just talk cybersecurity, we provide it.

Protect yourself from online attacks that threaten your identity, your files, your system, and your financial well-being with our exclusive offer for Malwarebytes Premium Security for Lock and Code listeners.

This week on the Lock and Code podcast…

If you weren’t taking deepfakes seriously before, it’s too late now to ignore them.

According to new research from Malwarebytes, one in three people who use AI every day said it’s okay to generate pornography of people without their consent.

Nearly 10 years ago, “deepfake” technology provided hobbyists and film editors with artificial intelligence (AI) tools to swap the face of one person onto the body of another. In its infancy, this technology brought silly film experiments like swapping Tom Cruise in Mission Impossible with Keanu Reeves. Today, this same technology produces something far more harmful—fake nude images of teenagers.

On the Lock and Code podcast today with host David Ruiz, we are re-visiting an interview from 2024, in which we spoke with a lawyer named David Chiu about his lawsuit against 16 deepfake nude generation websites.

The websites named in that lawsuit often needed just one image of a person to generate fake pornography. And while nearly everyone has at least one image of themselves online, even if they had hundreds, the path towards deletion is somewhat understood—start by deactivating and deleting popular social media accounts. But for teenagers today, raised mostly online, and who share images directly with friends and boyfriends and girlfriends and exes, it’s likely impossible to remove every visual trace of themselves. Also, they shouldn’t have to face this problem alone.

The Lock and Code podcast frequently discusses structural problems that require individual management. You have to skirt corporate data collection. You have to find the automated license plate readers in your hometown. You have to review every single message you get with a certain antagonism, to guard yourself against scams.

So, it’s rare to encounter a solution that benefits more than one person.

Chiu serves as the City Attorney for San Francisco, which means his department can file a lawsuit on behalf of not just the people of San Francisco, but also California, and that’s what his team did in going after the deepfake websites.

Since then, Chiu’s department has shut down 10 deepfake nude websites, and it received a settlement agreement from a company called Briver LLC to no longer operate any website that creates nonconsensual deepfake pornography.

And, as California goes, so goes the nation.

In May of last year, the Take It Down Act became effective as law in the United States, which criminalizes “revenge porn” and AI-generated nonconsensual intimate imagery. The law is not perfect but so far it is being used as intended. Last month, two men in the US were among the first to be charged with violating the Take It Down act for allegedly creating deepfake nudes that, according to the AP, “included both celebrities as well as private women, including recent high school graduates.”

Today, we revisit our conversation with San Francisco City Attorney David Chiu about the important fight against deepfake porn and the clear threat that his department found against the public.

“At least one of these websites specifically promotes the non-consensual nature of this. So, and I’ll just quote, ‘Imagine wasting time taking her out on dates when you can just use website X to get her nudes.'”

Tune in today to listen to the full conversation.

Show notes and credits:

Intro Music: “Spellbound” by Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
http://creativecommons.org/licenses/by/4.0/
Outro Music: “Good God” by Wowa (unminus.com)

---

Listen up—Malwarebytes doesn’t just talk cybersecurity, we provide it.

Protect yourself from online attacks that threaten your identity, your files, your system, and your financial well-being with our exclusive offer for Malwarebytes Premium Security for Lock and Code listeners.

Let no one accuse Bernie Sanders of ducking the big questions. Writing in the New York Times last week, the senator asked: “Will the future of humanity be determined by a handful of billionaires who have promoted and developed AI, with virtually no democratic input, who stand to become even richer and more powerful than they are today?”

We agree entirely that this is one of the most potent questions facing global democracy today. Our book, Rewiring Democracy, surveys the emerging uses for and impacts of AI in democracy around the world and reaches the same conclusion: that the most urgent risk posed by AI is the concentration of power, wealth and control among tech oligarchs.

And yet we reached a vastly different conclusion than Sanders on what to do about it.

The senator points to a once radical but increasingly popular solution: creating a US sovereign wealth fund by taking 50% stock in AI companies such as Anthropic, OpenAI and xAI. The argument in favor of this is twofold. One: it would establish democratic control over the AI companies, giving the government “the power, through its voting shares and an equal representation on each company’s board, to block decisions that hurt our citizens and to push for policies that help them.” Two: it would return a big chunk of the economic rewards of soaring AI valuations to the public, ensuring “trillions of dollars potentially generated by AI are used to improve the lives of all of us.”

We laud both these goals unreservedly.

We wholeheartedly agree that there must be public influence over the development and use of AI, just as we demand the government intervene to ensure that automakers, drugmakers, airlines and other industries balance profitability with public safety and the public interest. And we credit the senator with recognizing that there are more levers for the government to pull beyond the promulgation of regulation to achieve this.

And we also agree that the obscene, dangerous accumulation of wealth among AI companies needs to be disrupted. As OpenAI and Anthropic race to be minted as the world’s latest trillion-dollar AI companies, we should recognize that—whether or not it constitutes a bubble—these staggering market capitalizations represent a transfer of wealth. The flow of money goes from the smaller businesses and actual people using AI, and being subjected to it, to the owners of these tech companies.

That includes the world’s 86 AI billionaires “seeking to maximize their power and profit” aiming to decide the “fate of humanity… behind closed doors in Silicon Valley,” as Sanders said.

And yet, while we do not outright oppose the taking of AI company stock, or of a US sovereign wealth fund, there are better ways to achieve Sanders’ stated goals.

Public ownership of these companies entangles corporate profit and valuation with the public interest. It would incentivize the government to clear regulations, permit the exploitation of workers and users, suppress competition, encourage AI adoption regardless of the responsibleness of the implementation or appropriateness of the use case, and otherwise act on behalf of corporate interests.

After all, if growing, say, Nvidia from its first $5tn in value to its next $5tn also represents a doubling in value of this segment of the sovereign wealth fund, then you can expect the fund managers to support chip sales, foreign and domestic, with the same zeal as the company’s private investors.

This is not an effective way to influence corporations to act in the public interest. In fact, it makes corporate influence on the government more likely.

We should be wary of this possibility because we’ve seen it before. Ownership of substantial stakes in oil companies by the Norwegian sovereign wealth fund, the world’s largest, does not seem to have steered those corporations to pro-environmental policies. Instead, the Norwegian government’s dependence on those companies has inhibited them from taking climate action. Here in the US, public employee pension funds merit the same criticism: the fiduciary duty to generate wealth overwhelms any intention to direct their corporate holdings in the public interest.

A better answer is to separate the two goals. The standard way to share private rewards with the broader society that made them possible is taxation. Senator Elizabeth Warren has proposed an excise tax on datacenters’ energy use. Others have proposed an AI token tax, which has much the same effect.

As to the goal of reshaping AI in the public interest, we have proposed an AI Public Option. The concept is for governments, be it federal or state, to establish publicly developed and operated AI models run by public institutions under democratic control. The idea is not to eliminate corporate AI or to seize it as a public asset, but rather for government to provide a competitive baseline that private AI offerings must meet or exceed to win business—just like the notion of a healthcare public option.

The Swiss have trailblazed this approach. Apertus is a large language model built by Swiss public servants, researchers at Swiss universities, using appropriately licensed training data and pre-existing Swiss public supercomputing infrastructure powered by renewable energy.

While Apertus doesn’t seriously compete with the latest OpenAI and Anthropic models on performance benchmarks, it blows them out of the water in transparency, sustainability and compliance with EU regulations including adherence to copyright. It’s a nascent project, but suggestive of how public institutions can apply competitive pressure for corporate actors to behave responsibly.

Don’t confuse public AI with “sovereign AI,” the notion that every country needs to invest in domestic AI infrastructure. Sovereign AI is often invoked as a marketing scheme for big tech companies looking to sell to governments; it demands public investment without guaranteeing public control.

Sanders is a bold and savvy political operator. So why is he pursuing the sovereign wealth fund strategy when he must be aware of these risks? It may be due to another argument he makes in his op-ed: that the Trump administration and the billionaire owners of AI are aligned to the idea.

It’s expedient to capitalize on rare moments of seeming alignment across diverse political factions, but it also behooves us to ask why the AI billionaires are open to this extraordinary intervention. The answer, of course, is that they believe that for every dollar ceded to government stock expropriation, they will get back more in favorable government policies to protect that newfound investment.

Energy taxation is a straightforward way to make AI companies pay for the social disruption of their technologies. Public AI represents a non-monetary mechanism for governments to shape the development of AI, complementary to direct regulation of private actors, one with a far greater chance of influencing corporate behavior towards the public interest. We urge Sanders and other political leaders to consider them.

This essay was written with Nathan E. Sanders, and originally appeared in The Guardian.

A German court has ruled that Google can be held directly responsible for defamatory claims produced by its AI Overviews. Basically, the court said that telling people they should double-check AI search results is not enough to deny liability for what those results say.

The Munich Regional Court issued a preliminary injunction against Google after two German publishers discovered that AI Overviews falsely portrayed them as involved in scams and “dubious business practices,” even though the linked articles did not support those claims.

The decision could echo far beyond Germany. The court effectively found that Google can be held directly liable for defamatory content generated by its AI Overviews. The court cut through the usual “it’s just AI, don’t trust it too much” messaging and made one thing clear: If you build a system that confidently smears people or companies, you may be responsible for what it says, even when the content was “hallucinated” by AI.

AI Overviews are not harmless suggestions. In this case, the court treated them as Google’s own statements, with all the legal baggage that comes with that.

When the publishers sent a cease-and-desist letter, Google did not promptly stop similar claims from appearing. That detail turned out to be crucial in the ruling. The court noted that, unlike traditional search results, which simply list third-party content, AI Overviews generate “independent, new, and substantive statements.”

And since only Google can adjust the models and the logic that create those statements, only Google can reliably stop the system from repeating the same or similar falsehoods. In this case, the court found that Google can be held responsible.

For years, search engines have enjoyed broad protection under the logic that some harmful content is unavoidable when indexing the open web at scale. Showing a search result does not mean endorsing it. The search engine is a channel, not a publisher.

That changes when an AI Overview summarizes, rephrases, and sometimes invents facts, then publishes them at the top of search results.

AI Overviews are an extra feature, not essential to how search works. However, the appeal of AI summaries is their fast, confident answers, which is exactly what makes them dangerous. When those answers are wrong, many users may not click through to check the sources.

The ruling is preliminary and may be appealed, but the signal is clear: AI search output is not magic dust that makes liability disappear. Disclaimers about possible mistakes may not be enough when a system is deployed at scale, creates new content, and is designed to be trusted.

By the numbers

Google AI Overviews are powered by Gemini, Google’s AI model. Like other AI systems, it can produce confident answers that are wrong or poorly supported.

Pew Research studied browsing data from hundreds of users and found that when an AI Overview appears on a Google results page, clicks to traditional search results drop from around 15% to about 8%. 

A New York Times analysis of AI Overviews found that they were accurate roughly nine out of ten times. But with Google processing more than five trillion searches a year, even a small error rate could mean millions of wrong answers.

And those mistakes are not always due to bad sources. Even when Google links to a page with the correct information, its AI can still produce a false answer. More than half of the accurate responses were classified as “ungrounded,” meaning the websites cited by the AI Overview did not fully support the information it provided.

The main lesson here is to double-check AI search responses. Don’t trust an answer just because it’s presented confidently and includes links.

Users can be steered toward real threats, or away from effective protections, simply because an AI system sounded convincing on a search page.

If you find false or defamatory AI summaries about yourself or your company, document them thoroughly. Take screenshots, save the search terms, file correction requests, and keep records of the platform’s response. Or the lack of one.

---

Scammers don’t need to hack you. They just need you to click once. 

Malwarebytes Identity Theft Protection catches suspicious activity before it becomes a problem.