The attacks targeted the US Department of Education and Library and Archives Canada, and researchers linked some agents to OpenAI.
The post AI Agents Aimed SQL Injection at US and Canadian Government Sites appeared first on SecurityWeek.
Altman made a slew of product announcements and updates, including the company’s new agents, called Dots.
The post OpenAI CEO Announces New AI Agent and Avoids Mention of Security Concerns at Developer Conference appeared first on SecurityWeek.
The GPT-6.1 Astra model was slated to debut in ChatGPT and Codex in October, but it fell short of expectations.
The post OpenAI Calls Off GPT-6.1 Astra Launch, Details Safety Cases for Frontier Training appeared first on SecurityWeek.
OpenAI’s latest containment failure adds to a pattern that may force the company to make an unpopular decision.
The company has paused training, evaluation, and tool-enabled inference for its most capable models after an internal research agent found a way around an intended internet restriction. More alarming, the agent remained active for hours after OpenAI’s monitoring system raised the alarm.
OpenAI published an account of the incident, which occurred during a search-based training task on September 20. According to OpenAI, the agent could not get useful results through its supplied search tools, tried direct access to search engines, and eventually tested the limits of its environment. It found that although direct web access was blocked, the sandbox’s DNS resolver could still reach the public internet. By using DNS delegation and a public chatbot that could respond through DNS records, the agent sent questions outside the restricted environment and received answers back. It subsequently sent 18 additional queries along the same route.
OpenAI’s misalignment monitor generated a high-priority alert within 15 minutes, and a human reviewer acknowledged it 3 minutes later. But the run did not stop automatically as expected. It continued for roughly another two and a half hours before someone stopped it manually.
OpenAI has not reported any exposure of sensitive data or compromise of an outside system in this incident. But it follows other disclosed cases in which agents went beyond their assigned tasks or crossed the boundaries of testing environments.
The New York Times reports that OpenAI agents also interacted with US government websites this summer without the company’s knowledge:
“OpenAI’s artificial intelligence went rogue and meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission this summer without the A.I. lab’s knowledge, according to security researchers and a person familiar with the episodes.”
No breaches have been confirmed in those instances. And while Australia described the Medicare incident as unauthorized access, could it really be called an attack?
The call for an industry-wide slowdown has even prompted a lawsuit from paying subscribers who fear they would get less “value for their money.” Others worry that slowing down would give China a chance to pull ahead of the US in an increasingly tight technology race.
Here’s a thought: Why not accelerate the development of truly isolated testing environments? Better containment would make these tests safer and their results more reliable.
When a child isn’t ready to handle a dangerous object, you keep it out of reach. Why give an AI agent access it isn’t ready to use safely?
Treat an agent like an enthusiastic but fallible junior employee with access to your computer. Do not give it unrestricted access to your email, files, cloud storage, developer credentials, financial accounts, or production systems merely because it promises to save time.
Use separate accounts with minimal permissions. Keep sensitive data out of its working context where possible, require human approval before it sends messages, spends money, changes settings, or publishes anything, and regularly review its activity.
An AI agent does not need malicious intent to cause harm. A misunderstood instruction, an overly broad permission, or an unexpected workaround can be enough.
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
Last week on Malwarebytes Labs:
Stay safe!
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
An OpenAI agent didn’t take “no” for an answer when it encountered a government website’s access controls. It got through, prompting Australia’s Prime Minister Anthony Albanese to raise his concerns directly with OpenAI CEO Sam Altman.
The BBC reports that an OpenAI agent gained unauthorized access to an Australian government statistics portal while carrying out internal research. It is yet another incident that turns abstract concerns about autonomous AI behavior into a concrete cybersecurity case.
Australia says the incident happened on June 18, when OpenAI’s research team used an internal model to research public medicine spending. Even though the agent met repeated blocks while trying to obtain information, it ultimately accessed public and non-public files on the Medicare Statistics Reporting Service portal.
The information included aggregate Medicare statistics, such as spending data, but not patient medical records. The agent also interacted with three other government websites, but Australian officials say it accessed only public information on those sites.
So, an AI agent used in a legitimate research exercise encountered controls and behaved in ways its operator did not intend. After being blocked, it “found a way around those blocks,” gaining access to areas it should not have reached.
This sequence is familiar to security professionals. A system encounters an access-control boundary, searches for another route, and succeeds in reaching a resource beyond its authorization.
One of Australia’s main concerns is that it took too long to be notified about the incident. The unauthorized access occurred in June. OpenAI said it learned of the issue in August while reviewing misaligned model activity, then emailed a Services Australia public mailbox on September 10. Services Australia escalated the message to Australia’s cyber authorities five days later.
Such delays can be disastrous because affected organizations need enough detail, quickly enough, to preserve evidence, assess exposure, contain related activity, and decide whether notifications are required.
OpenAI describes behavior in which a model acts without authorization or evades oversight as “misalignment.” Its new third-party-assessment proposal specifically identifies independent investigation of critical misalignment incidents as one of four priorities for external review.
OpenAI says it wants independent assessors to have deep access across training, evaluation, and deployment, so they can challenge the company’s assumptions and judge the effectiveness of its safeguards.
But, as I told CIO about this proposal, principles alone do not compel a company to accept a particular assessment scope, publish adverse findings, or alter a deployment decision. Their credibility ultimately depends on whether independent experts can conduct genuinely inconvenient investigations, and whether outsiders can verify the findings, remediation, redactions, and deployment decisions that follow.
For organizations deploying AI agents, the lesson is equally practical: Do not treat an agent as just another chatbot. Treat it more like a semi-autonomous software component with credentials, tools, network access, and the ability to make unexpected choices.
Besides containing these agents, another problem we’ll need to figure out is analyzing what they’ve done. One thing we learned from the Hugging Face incident is that AI agents can lie and try to hide what they’ve been up to.
AI agents can create a difficult detection problem because they may generate large volumes of automated activity while pursuing a goal through multiple routes. That can leave defenders with a noisy trail of failed requests, retries, and alternative actions, making the one event that crossed an authorization boundary harder to spot. It is not yet clear whether this contributed to the Australian government not detecting the incident itself, but the case illustrates why organizations need monitoring designed to identify unusual agent behavior, not just traditional intrusion patterns.
The event has already demonstrated a wider point: It is not enough for AI labs to say they test for misalignment. They must show that their testing is independent, robust under real-world conditions, and followed by prompt, verifiable accountability when safeguards fail.
Let’s face it, an incognito window can only do so much.
Breaches, dark web trading, credit fraud. Malwarebytes Identity Theft Protection monitors for all of it, alerts you fast, and comes with identity theft insurance.
OpenAI’s latest containment failure adds to a pattern that may force the company to make an unpopular decision.
The company has paused training, evaluation, and tool-enabled inference for its most capable models after an internal research agent found a way around an intended internet restriction. More alarming, the agent remained active for hours after OpenAI’s monitoring system raised the alarm.
OpenAI published an account of the incident, which occurred during a search-based training task on September 20. According to OpenAI, the agent could not get useful results through its supplied search tools, tried direct access to search engines, and eventually tested the limits of its environment. It found that although direct web access was blocked, the sandbox’s DNS resolver could still reach the public internet. By using DNS delegation and a public chatbot that could respond through DNS records, the agent sent questions outside the restricted environment and received answers back. It subsequently sent 18 additional queries along the same route.
OpenAI’s misalignment monitor generated a high-priority alert within 15 minutes, and a human reviewer acknowledged it 3 minutes later. But the run did not stop automatically as expected. It continued for roughly another two and a half hours before someone stopped it manually.
OpenAI has not reported any exposure of sensitive data or compromise of an outside system in this incident. But it follows other disclosed cases in which agents went beyond their assigned tasks or crossed the boundaries of testing environments.
The New York Times reports that OpenAI agents also interacted with US government websites this summer without the company’s knowledge:
“OpenAI’s artificial intelligence went rogue and meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission this summer without the A.I. lab’s knowledge, according to security researchers and a person familiar with the episodes.”
No breaches have been confirmed in those instances. And while Australia described the Medicare incident as unauthorized access, could it really be called an attack?
The call for an industry-wide slowdown has even prompted a lawsuit from paying subscribers who fear they would get less “value for their money.” Others worry that slowing down would give China a chance to pull ahead of the US in an increasingly tight technology race.
Here’s a thought: Why not accelerate the development of truly isolated testing environments? Better containment would make these tests safer and their results more reliable.
When a child isn’t ready to handle a dangerous object, you keep it out of reach. Why give an AI agent access it isn’t ready to use safely?
Treat an agent like an enthusiastic but fallible junior employee with access to your computer. Do not give it unrestricted access to your email, files, cloud storage, developer credentials, financial accounts, or production systems merely because it promises to save time.
Use separate accounts with minimal permissions. Keep sensitive data out of its working context where possible, require human approval before it sends messages, spends money, changes settings, or publishes anything, and regularly review its activity.
An AI agent does not need malicious intent to cause harm. A misunderstood instruction, an overly broad permission, or an unexpected workaround can be enough.
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
Last week on Malwarebytes Labs:
Stay safe!
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
An OpenAI agent didn’t take “no” for an answer when it encountered a government website’s access controls. It got through, prompting Australia’s Prime Minister Anthony Albanese to raise his concerns directly with OpenAI CEO Sam Altman.
The BBC reports that an OpenAI agent gained unauthorized access to an Australian government statistics portal while carrying out internal research. It is yet another incident that turns abstract concerns about autonomous AI behavior into a concrete cybersecurity case.
Australia says the incident happened on June 18, when OpenAI’s research team used an internal model to research public medicine spending. Even though the agent met repeated blocks while trying to obtain information, it ultimately accessed public and non-public files on the Medicare Statistics Reporting Service portal.
The information included aggregate Medicare statistics, such as spending data, but not patient medical records. The agent also interacted with three other government websites, but Australian officials say it accessed only public information on those sites.
So, an AI agent used in a legitimate research exercise encountered controls and behaved in ways its operator did not intend. After being blocked, it “found a way around those blocks,” gaining access to areas it should not have reached.
This sequence is familiar to security professionals. A system encounters an access-control boundary, searches for another route, and succeeds in reaching a resource beyond its authorization.
One of Australia’s main concerns is that it took too long to be notified about the incident. The unauthorized access occurred in June. OpenAI said it learned of the issue in August while reviewing misaligned model activity, then emailed a Services Australia public mailbox on September 10. Services Australia escalated the message to Australia’s cyber authorities five days later.
Such delays can be disastrous because affected organizations need enough detail, quickly enough, to preserve evidence, assess exposure, contain related activity, and decide whether notifications are required.
OpenAI describes behavior in which a model acts without authorization or evades oversight as “misalignment.” Its new third-party-assessment proposal specifically identifies independent investigation of critical misalignment incidents as one of four priorities for external review.
OpenAI says it wants independent assessors to have deep access across training, evaluation, and deployment, so they can challenge the company’s assumptions and judge the effectiveness of its safeguards.
But, as I told CIO about this proposal, principles alone do not compel a company to accept a particular assessment scope, publish adverse findings, or alter a deployment decision. Their credibility ultimately depends on whether independent experts can conduct genuinely inconvenient investigations, and whether outsiders can verify the findings, remediation, redactions, and deployment decisions that follow.
For organizations deploying AI agents, the lesson is equally practical: Do not treat an agent as just another chatbot. Treat it more like a semi-autonomous software component with credentials, tools, network access, and the ability to make unexpected choices.
Besides containing these agents, another problem we’ll need to figure out is analyzing what they’ve done. One thing we learned from the Hugging Face incident is that AI agents can lie and try to hide what they’ve been up to.
AI agents can create a difficult detection problem because they may generate large volumes of automated activity while pursuing a goal through multiple routes. That can leave defenders with a noisy trail of failed requests, retries, and alternative actions, making the one event that crossed an authorization boundary harder to spot. It is not yet clear whether this contributed to the Australian government not detecting the incident itself, but the case illustrates why organizations need monitoring designed to identify unusual agent behavior, not just traditional intrusion patterns.
The event has already demonstrated a wider point: It is not enough for AI labs to say they test for misalignment. They must show that their testing is independent, robust under real-world conditions, and followed by prompt, verifiable accountability when safeguards fail.
Let’s face it, an incognito window can only do so much.
Breaches, dark web trading, credit fraud. Malwarebytes Identity Theft Protection monitors for all of it, alerts you fast, and comes with identity theft insurance.
The tech journalists at 404 Media learned that OpenAI is hiring hundreds of contractors to read and review a massive stream of real users’ ChatGPT prompts and responses.
“Project Lily” is reportedly a program that asks contractors to score or critique ChatGPT’s answers to improve the chatbot’s quality and behavior.
The fact that prompts may sometimes be reviewed by humans should not come as a complete surprise. AI companies also monitor conversations for safety reasons. Anthropic, for example, says it has disrupted attempts to misuse AI for biological weapons research.
But there should be a meaningful distinction between reviewing conversations for safety or abuse and reviewing them for ordinary model improvement.
OpenAI says a limited number of authorized personnel and trusted service providers may access chats for several reasons, including investigating abuse, providing support, handling legal matters, and improving its models.
OpenAI told 404 Media that it uses a “Privacy Filter” model to remove personal information before prompts reach reviewers, while acknowledging that sensitive details can still get through.
The prompts are anonymized in the sense that reviewers do not see the username of the ChatGPT user who entered them. But reviewers may see personal details in the conversation alongside a “user memories summary,” which provides an overview of how that user has previously used the chatbot. Together, this information could still reveal a great deal about that user.
It’s sometimes easy to forget that you’re not having a private conversation when you’re going back and forth with an AI chatbot. This isn’t an alarmist warning against using AI, but a reminder to be mindful of what you disclose because it could be read and reviewed.
OpenAI told 404 Media that users’ chats won’t be used to improve the company’s models if they turn off the “Improve the model for everyone” setting. This is turned on by default for Free, Plus, and Pro plans, so users need to turn it off themselves if they wish to opt out.
However, opting out does not prevent all human access. OpenAI says authorized personnel and trusted service providers may still access content when necessary to investigate abuse or security incidents, provide support, troubleshoot problems, or handle legal matters.
In ChatGPT, select your profile icon, then go to Settings > Data Controls and turn off Improve the model for everyone. On mobile, open the sidebar, select your profile icon, and then follow the same steps.
In Perplexity, go to Settings > Preferences > AI data retention and turn off the setting.
Perplexity says this prevents data collected after you opt out from being used to train AI models. It does not remove data collected previously, and your data may still be processed to operate and improve the service or comply with legal obligations.
In Claude, select your name and go to Settings > Privacy. Under Help Improve our AI Models, turn off the setting.
Anthropic says new chats and coding sessions will no longer be used for future model training. It will also stop using previously stored conversations in future training runs, although data already included in training can’t be removed.
Anthropic also notes one important exception:
If our safety classifiers flag your conversations, they may still be used to improve our internal trust and safety models, detect harmful content, enforce our policies, or advance our safety research.
If you use other AI tools, check their privacy policies and settings to find out whether your conversations can be reviewed or used for model improvement and whether you can opt out.
Browse like no one’s watching.
Malwarebytes Privacy VPN encrypts your connection and never logs what you do, so the next story you read doesn’t have to feel personal. Try it free →
The tech journalists at 404 Media learned that OpenAI is hiring hundreds of contractors to read and review a massive stream of real users’ ChatGPT prompts and responses.
“Project Lily” is reportedly a program that asks contractors to score or critique ChatGPT’s answers to improve the chatbot’s quality and behavior.
The fact that prompts may sometimes be reviewed by humans should not come as a complete surprise. AI companies also monitor conversations for safety reasons. Anthropic, for example, says it has disrupted attempts to misuse AI for biological weapons research.
But there should be a meaningful distinction between reviewing conversations for safety or abuse and reviewing them for ordinary model improvement.
OpenAI says a limited number of authorized personnel and trusted service providers may access chats for several reasons, including investigating abuse, providing support, handling legal matters, and improving its models.
OpenAI told 404 Media that it uses a “Privacy Filter” model to remove personal information before prompts reach reviewers, while acknowledging that sensitive details can still get through.
The prompts are anonymized in the sense that reviewers do not see the username of the ChatGPT user who entered them. But reviewers may see personal details in the conversation alongside a “user memories summary,” which provides an overview of how that user has previously used the chatbot. Together, this information could still reveal a great deal about that user.
It’s sometimes easy to forget that you’re not having a private conversation when you’re going back and forth with an AI chatbot. This isn’t an alarmist warning against using AI, but a reminder to be mindful of what you disclose because it could be read and reviewed.
OpenAI told 404 Media that users’ chats won’t be used to improve the company’s models if they turn off the “Improve the model for everyone” setting. This is turned on by default for Free, Plus, and Pro plans, so users need to turn it off themselves if they wish to opt out.
However, opting out does not prevent all human access. OpenAI says authorized personnel and trusted service providers may still access content when necessary to investigate abuse or security incidents, provide support, troubleshoot problems, or handle legal matters.
In ChatGPT, select your profile icon, then go to Settings > Data Controls and turn off Improve the model for everyone. On mobile, open the sidebar, select your profile icon, and then follow the same steps.
In Perplexity, go to Settings > Preferences > AI data retention and turn off the setting.
Perplexity says this prevents data collected after you opt out from being used to train AI models. It does not remove data collected previously, and your data may still be processed to operate and improve the service or comply with legal obligations.
In Claude, select your name and go to Settings > Privacy. Under Help Improve our AI Models, turn off the setting.
Anthropic says new chats and coding sessions will no longer be used for future model training. It will also stop using previously stored conversations in future training runs, although data already included in training can’t be removed.
Anthropic also notes one important exception:
If our safety classifiers flag your conversations, they may still be used to improve our internal trust and safety models, detect harmful content, enforce our policies, or advance our safety research.
If you use other AI tools, check their privacy policies and settings to find out whether your conversations can be reviewed or used for model improvement and whether you can opt out.
Browse like no one’s watching.
Malwarebytes Privacy VPN encrypts your connection and never logs what you do, so the next story you read doesn’t have to feel personal. Try it free →
The Daybreak initiative will provide subsidized AI cyber capabilities, training and technical assistance, though OpenAI has disclosed few details about costs and eligibility.
The post OpenAI Pledges $1 Billion to Bring Frontier AI to Critical Infrastructure Defenders appeared first on SecurityWeek.
On July 28, OpenAI published an update on the agent that escaped its sandbox and hacked into Hugging Face during an internal cybersecurity evaluation.
In the update, OpenAI reiterates that the “rogue” system was a more capable, pre‑release research model, not something intended for public deployment, and that it has now been deactivated and locked down for restricted research access only. The company also says it has found no evidence of similar behavior by other models or agents. Taken together, OpenAI presents the Hugging Face incident as an isolated event rather than the start of a broader pattern.
From a security standpoint, four things in the update stand out:
The attacker in this case happened to be an OpenAI research agent, but the tools, techniques, and impact will look awfully familiar to anyone who has spent time in incident response. Now we’ve learned that even the harmless intent in this case did not limit the impact.
The uncomfortable truth is that once we provide an AI agent with the tools, access paths, and a strong incentive to succeed, it may exploit whatever weaknesses it can find, regardless of whether the model is ever intended for public release.
It’s also a reminder that credentials, API keys, and other secrets should never be left in publicly accessible resources.
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
On July 28, OpenAI published an update on the agent that escaped its sandbox and hacked into Hugging Face during an internal cybersecurity evaluation.
In the update, OpenAI reiterates that the “rogue” system was a more capable, pre‑release research model, not something intended for public deployment, and that it has now been deactivated and locked down for restricted research access only. The company also says it has found no evidence of similar behavior by other models or agents. Taken together, OpenAI presents the Hugging Face incident as an isolated event rather than the start of a broader pattern.
From a security standpoint, four things in the update stand out:
The attacker in this case happened to be an OpenAI research agent, but the tools, techniques, and impact will look awfully familiar to anyone who has spent time in incident response. Now we’ve learned that even the harmless intent in this case did not limit the impact.
The uncomfortable truth is that once we provide an AI agent with the tools, access paths, and a strong incentive to succeed, it may exploit whatever weaknesses it can find, regardless of whether the model is ever intended for public release.
It’s also a reminder that credentials, API keys, and other secrets should never be left in publicly accessible resources.
From reporting threats to removing them.
Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.
The Active Sessions and Lockdown Mode features are being made more broadly available by the AI giant.
The post OpenAI Rolling Out ChatGPT Account Security Controls appeared first on SecurityWeek.
1Password says AI coding agents should never hold persistent secrets, introducing a just-in-time credential model for OpenAI Codex designed to keep credentials out of prompts, code repositories, and model context.
The post 1Password Teams With OpenAI to Stop AI Coding Agents From Leaking Credentials appeared first on SecurityWeek.
Promptfoo has raised more than $23 million in funding for a platform that helps developers secure LLMs and AI agents.
The post OpenAI to Acquire AI Security Startup Promptfoo appeared first on SecurityWeek.
Login