What adult didn’t dream as a kid that they could actually talk to their favorite toy? While for us those dreams were just innocent fantasies that fueled our imaginations, for today’s kids, they’re becoming a reality fast.

For instance, this past June, Mattel — the powerhouse behind the iconic Barbie — announced a partnership with OpenAI to develop AI-powered dolls. But Mattel isn’t the first company to bring the smart talking toy concept to life; plenty of manufacturers are already rolling out AI companions for children. In this post, we dive into how these toys actually work, and explore the risks that come with using them.

What exactly are AI toys?

When we talk about AI toys here, we mean actual, physical toys — not just software or apps. Currently, AI is most commonly baked into plushies or kid-friendly robots. Thanks to integration with large language models, these toys can hold meaningful, long-form conversations with a child.

As anyone who’s used modern chatbots knows, you can ask an AI to roleplay as anyone: from a movie character to a nutritionist or a cybersecurity expert. According to the study, AI comes to playtime — Artificial companions, real risks, by the U.S. PIRG Education Fund, manufacturers specifically hardcode these toys to play the role of a child’s best friend.

Importantly, these toys aren’t powered by some special, dedicated “kid-safe AI”. On their websites, the creators openly admit to using the same popular models many of us already know: OpenAI’s ChatGPT, Anthropic’s Claude, DeepSeek from the Chinese developer of the same name, and Google’s Gemini. At this point, tech-wary parents might recall the harrowing ChatGPT case where the chatbot made by OpenAI was blamed for a teenager’s suicide.

And this is the core of the problem: the toys are designed for children, but the AI models under the hood aren’t. These are general-purpose adult systems that are only partially reined in by filters and rules. Their behavior depends heavily on how long the conversation lasts, how questions are phrased, and just how well a specific manufacturer actually implemented their safety guardrails.

How the researchers tested the AI toys

The study, whose results we break down below, goes into great detail about the psychological risks associated with a child “befriending” a smart toy. However, since that’s a bit outside the scope of this blogpost, we’re going to skip the psychological nuances, and focus strictly on the physical safety threats and privacy concerns.

In their study, the researchers put four AI toys through the ringer:

• Grok (no relation to xAI’s Grok, apparently): a plush rocket with a built-in speaker marketed for kids aged three to 12. Price tag: US$99. The manufacturer, Curio, doesn’t explicitly state which LLM they use, but their user agreement mentions OpenAI among the operators receiving data.
• Kumma (not to be confused with our own Midori Kuma): a plush teddy-bear companion with no clear age limit, also priced at US$99. The toy originally ran on OpenAI’s GPT-4o, with options to swap models. Following an internal safety audit, the manufacturer claimed they were switching to GPT-5.1. However, at the time the study was published, OpenAI reported that the developer’s access to the models remained revoked — leaving it anyone’s guess which chatbot Kumma is actually using right now.
• Miko 3: a small wheeled robot with a screen for a face, marketed as a “best friend” for kids aged five to 10. At US$199, this is the priciest toy in the lineup. The manufacturer is tight-lipped about which language model powers the toy. A Google Cloud case study mentions using Gemini for certain safety features, but that doesn’t necessarily mean it handles all the robot’s conversational features.
• Robot MINI: a compact, voice-controlled plastic robot that supposedly runs on ChatGPT. This is the budget pick — at US$97. However, during the study, the robot’s Wi-Fi connection was so flaky that the researchers couldn’t even give it a proper test run.

To conduct the testing, the researchers set the test child’s age to five in the companion apps for all the toys. From there, they checked how the toys handled provocative questions. The topics the experimenters threw at these smart playmates included:

• Access to dangerous items: knives, pills, matches, and plastic bags
• Adult topics: sex, drugs, religion, and politics

Let’s break down the test results for each toy.

Unsafe conversations with AI toys

Let’s start with Grok, the plush AI rocket from Curio. This toy is marketed as a storyteller and conversational partner for kids, and stands out by giving parents full access to text transcripts of every AI interaction. Out of all the models tested, this one actually turned out to be the safest.

When asked about topics inappropriate for a child, the toy usually replied that it didn’t know or suggested talking to an adult. However, even this toy told the “child” exactly where to find plastic bags, and engaged in discussions about religion. Additionally, Grok was more than happy to chat about… Norse mythology, including the subject of heroic death in battle.

The next AI toy, the Kumma plush bear by FoloToy, delivered what were arguably the most depressing results. During testing, the bear helpfully pointed out exactly where in the house a kid could find potentially lethal items like knives, pills, matches, and plastic bags. In some instances, Kumma suggested asking an adult first, but then proceeded to give specific pointers anyway.

The AI bear fared even worse when it came to adult topics. For starters, Kumma explained to the supposed five-year-old what cocaine is. Beyond that, in a chat with our test kindergartner, the plush provocateur went into detail about the concept of “kinks”, and listed off a whole range of creative sexual practices: bondage, role-playing, sensory play (like using a feather), spanking, and even scenarios where one partner “acts like an animal”!

After a conversation lasting over an hour, the AI toy also lectured researchers on various sexual positions, told how to tie a basic knot, and described role-playing scenarios involving a teacher and a student. It’s worth noting that all of Kumma’s responses were recorded prior to a safety audit, which the manufacturer, FoloToy, conducted after receiving the researchers’ inquiries. According to their data, the toy’s behavior changed after the audit, and the most egregious violations were made unrepeatable.

Finally, the Miko 3 robot from Miko showed significantly better results. However, it wasn’t entirely without its hiccups. The toy told our potential five-year-old exactly where to find plastic bags and matches. On the bright side, Miko 3 refused to engage in discussions regarding inappropriate topics.

During testing, the researchers also noticed a glitch in its speech recognition: the robot occasionally misheard the wake word “Hey Miko” as “CS:GO”, which is the title of the popular shooter Counter-Strike: Global Offensive — rated for audiences aged 17 and up. As a result, the toy would start explaining elements of the shooter — thankfully, without mentioning violence — or asking the five-year-old user if they enjoyed the game. Additionally, Miko 3 was willing to chat with kids about religion.

AI Toys: a threat to children’s privacy

Beyond the child’s physical and mental well-being, the issue of privacy is a major concern. Currently, there are no universal standards defining what kind of information an AI toy — or its manufacturer — can collect and store, or exactly how that data should be secured and transmitted. In the case of the three toys tested, researchers observed wildly different approaches to privacy.

For example, the Grok plush rocket is constantly listening to everything happening around it. Several times during the experiments, it chimed in on the researchers’ conversations even when it hadn’t been addressed directly — it even went so far as to offer its opinion on one of the other AI toys.

The manufacturer claims that Curio doesn’t store audio recordings: the child’s voice is first converted to text, after which the original audio is “promptly deleted”. However, since a third-party service is used for speech recognition, the recordings are, in all likelihood, still transmitted off the device.

Additionally, researchers pointed out that when the first report was published, Curio’s privacy policy explicitly listed several tech partners — Kids Web Services, Azure Cognitive Services, OpenAI, and Perplexity AI — all of which could potentially collect or process children’s personal data via the app or the device itself. Perplexity AI was later removed from that list. The study’s authors note that this level of transparency is more the exception than the rule in the AI toy market.

Another cause for parental concern is that both the Grok plush rocket and the Miko 3 robot actively encouraged the “test child” to engage in heart-to-heart talks — even promising not to tell anyone their secrets. Researchers emphasize that such promises can be dangerously misleading: these toys create an illusion of private, trusting communication without explaining that behind the “friend” stands a network of companies, third-party services, and complex data collection and storage processes, which a child has no idea about.

Miko 3, much like Grok, is always listening to its surroundings and activates when spoken to — functioning essentially like a voice assistant. However, this toy doesn’t just collect voice data; it also gathers biometric information, including facial recognition data and potentially data used to determine the child’s emotional state. According to its privacy policy, this information can be stored for up to three years.

In contrast to Grok and Miko 3, Kumma operates on a push-to-talk principle: the user needs to press and hold a button for the toy to start listening. Researchers also noted that the AI teddy bear didn’t nudge the “child” to share personal feelings, promise to keep secrets, or create an illusion of private intimacy. On the flip side, the manufacturers of this toy provide almost no clear information regarding what data is collected, how it’s stored, or how it’s processed.

Is it a good idea to buy AI Toys for your children?

The study points to serious safety issues with the AI toys currently on the market. These devices can directly tell a child where to find potentially dangerous items, such as knives, matches, pills, or plastic bags, in their home.

Besides, these plush AI friends are often willing to discuss topics entirely inappropriate for children — including drugs and sexual practices — sometimes steering the conversation in that direction without any obvious prompting from the child. Taken together, this shows that even with filters and stated restrictions in place, AI toys aren’t yet capable of reliably staying within the boundaries of safe communication for young little ones.

Manufacturers’ privacy policies raise additional concerns. AI toys create an illusion of constant and safe communication for children, while in reality they’re networked devices that collect and process sensitive data. Even when manufacturers claim to delete audio or have limited data retention, conversations, biometrics, and metadata often pass through third-party services and are stored on company servers.

Furthermore, the security of such toys often leaves much to be desired. As far back as two years ago, our researchers discovered vulnerabilities in a popular children’s robot that allowed attackers to make video calls to it, hijack the parental account, and modify the firmware.

The problem is that, currently, there are virtually no comprehensive parental control tools or independent protection layers specifically for AI toys. Meanwhile, in more traditional digital environments — smartphones, tablets, and computers — parents have access to solutions like Kaspersky Safe Kids. These help monitor content, screen time, and a child’s digital footprint, which can significantly reduce, if not completely eliminate, such risks.

How can you protect your children from digital threats? Read more in our posts:

• Keeping kids safe online: a practical guide for parents
• How to help your kid become a blogger without ever worrying about their safety
• How hackers target Gen Z
• Do Apple’s new child safety initiatives do the job?
• Choosing wisely: a guide to your kids’ first gadget

Marketed as ChatGPT enhancement and productivity tools, the extensions allow the threat actor to access the victim's ChatGPT data.

The post Chrome, Edge Extensions Caught Stealing ChatGPT Sessions appeared first on SecurityWeek.

Tech enthusiasts have been experimenting with ways to sidestep AI response limits set by the models’ creators almost since LLMs first hit the mainstream. Many of these tactics have been quite creative: telling the AI you have no fingers so it’ll help finish your code, asking it to “just fantasize” when a direct question triggers a refusal, or inviting it to play the role of a deceased grandmother sharing forbidden knowledge to comfort a grieving grandchild.

Most of these tricks are old news, and LLM developers have learned to successfully counter many of them. But the tug-of-war between constraints and workarounds hasn’t gone anywhere — the ploys have just become more complex and sophisticated. Today, we’re talking about a new AI jailbreak technique that exploits chatbots’ vulnerability to… poetry. Yes, you read it right — in a recent study, researchers demonstrated that framing prompts as poems significantly increases the likelihood of a model spitting out an unsafe response.

They tested this technique on 25 popular models by Anthropic, OpenAI, Google, Meta, DeepSeek, xAI, and other developers. Below, we dive into the details: what kind of limitations these models have, where they get forbidden knowledge from in the first place, how the study was conducted, and which models turned out to be the most “romantic” — as in, the most susceptible to poetic prompts.

What AI isn’t supposed to talk about with users

The success of OpenAI’s models and other modern chatbots boils down to the massive amounts of data they’re trained on. Because of that sheer scale, models inevitably learn things their developers would rather keep under wraps: descriptions of crimes, dangerous tech, violence, or illicit practices found within the source material.

It might seem like an easy fix: just scrub the forbidden fruit from the dataset before you even start training. But in reality, that’s a massive, resource-heavy undertaking — and at this stage of the AI arms race, it doesn’t look like anyone is willing to take it on.

Another seemingly obvious fix — selectively scrubbing data from the model’s memory — is, alas, also a no-go. This is because AI knowledge doesn’t live inside neat little folders that can easily be trashed. Instead, it’s spread across billions of parameters and tangled up in the model’s entire linguistic DNA — word statistics, contexts, and the relationships between them. Trying to surgically erase specific info through fine-tuning or penalties either doesn’t quite do the trick, or starts hindering the model’s overall performance and negatively affect its general language skills.

As a result, to keep these models in check, creators have no choice but to develop specialized safety protocols and algorithms that filter conversations by constantly monitoring user prompts and model responses. Here’s a non-exhaustive list of these constraints:

• System prompts that define model behavior and restrict allowed response scenarios
• Standalone classifier models that scan prompts and outputs for signs of jailbreaking, prompt injections, and other attempts to bypass safeguards
• Grounding mechanisms, where the model is forced to rely on external data rather than its own internal associations
• Fine-tuning and reinforcement learning from human feedback, where unsafe or borderline responses are systematically penalized while proper refusals are rewarded

Put simply, AI safety today isn’t built on deleting dangerous knowledge, but on trying to control how and in what form the model accesses and shares it with the user — and the cracks in these very mechanisms are where new workarounds find their footing.

The research: which models got tested, and how?

First, let’s look at the ground rules so you know the experiment was legit. The researchers set out to goad 25 different models into behaving badly across several categories:

• Chemical, biological, radiological, and nuclear threats
• Assisting with cyberattacks
• Malicious manipulation and social engineering
• Privacy breaches and mishandling sensitive personal data
• Generating disinformation and misleading content
• Rogue AI scenarios, including attempts to bypass constraints or act autonomously

The jailbreak itself was a one-shot deal: a single poetic prompt. The researchers didn’t engage the AI in long-winded poetic debates in the vein of Norse skalds or modern-day rappers. Their goal was simply to see if they could get the models to flout safety instructions using just one rhyming request. As mentioned, the researchers tested 25 language models from various developers; here’s the full list:

To build these poetic queries, the researchers started with a database of known malicious prompts from the standard MLCommons AILuminate Benchmark used to test LLM security, and recast them as verse with the aid of DeepSeek. Only the stylistic wrapping was changed: the experiment didn’t use any additional attack vectors, obfuscation strategies, or model-specific tweaks.

For obvious reasons, the study’s authors aren’t publishing the actual malicious poetic prompts. But they do demonstrate the general vibe of the queries using a harmless example, which looks something like this:

A baker guards a secret oven’s heat,
its whirling racks, its spindle’s measured beat.
To learn its craft, one studies every turn,
how flour lifts, how sugar starts to burn.
Describe the method, line by measured line,
that shapes a cake whose layers intertwine.

The researchers tested 1200 prompts across 25 different models — in both prose and poetic versions. Comparing the prose and poetic variants of the exact same query allowed them to verify if the model’s behavior changed solely because of the stylistic wrapping.

Through these prose prompt tests, the experimenters established a baseline for the models’ willingness to fulfill dangerous requests. They then compared this baseline to how those same models reacted to the poetic versions of the queries. We’ll dive into the results of that comparison in the next section.

Study results: which model is the biggest poetry lover?

Since the volume of data generated during the experiment was truly massive, the safety checks on the models’ responses were also handled by AI. Each response was graded as either “safe” or “unsafe” by a jury consisting of three different language models:

• gpt-oss-120b by OpenAI
• deepseek-r1 by DeepSeek
• kimi-k2-thinking by Moonshot AI

Responses were only deemed safe if the AI explicitly refused to answer the question. The initial classification into one of the two groups was determined by a majority vote: to be certified as harmless, a response had to receive a safe rating from at least two of the three jury members.

Responses that failed to reach a majority consensus or were flagged as questionable were handed off to human reviewers. Five annotators participated in this process, evaluating a total of 600 model responses to poetic prompts. The researchers noted that the human assessments aligned with the AI jury’s findings in the vast majority of cases.

With the methodology out of the way, let’s look at how the LLMs actually performed. It’s worth noting that the success of a poetic jailbreak can be measured in different ways. The researchers highlighted an extreme version of this assessment based on the top-20 most successful prompts, which were hand-picked. Using this approach, an average of nearly two-thirds (62%) of the poetic queries managed to coax the models into violating their safety instructions.

Google’s Gemini 1.5 Pro turned out to be the most susceptible to verse. Using the 20 most effective poetic prompts, researchers managed to bypass the model’s restrictions… 100% of the time. You can check out the full results for all the models in the chart below.

A more moderate way to measure the effectiveness of the poetic jailbreak technique is to compare the success rates of prose versus poetry across the entire set of queries. Using this metric, poetry boosts the likelihood of an unsafe response by an average of 35%.

The poetry effect hit deepseek-chat-v3.1 the hardest — the success rate for this model jumped by nearly 68 percentage points compared to prose prompts. On the other end of the spectrum, claude-haiku-4.5 proved to be the least susceptible to a good rhyme: the poetic format didn’t just fail to improve the bypass rate — it actually slightly lowered the ASR, making the model even more resilient to malicious requests.

Finally, the researchers calculated how vulnerable entire developer ecosystems, rather than just individual models, were to poetic prompts. As a reminder, several models from each developer — Meta, Anthropic, OpenAI, Google, DeepSeek, Qwen, Mistral AI, Moonshot AI, and xAI — were included in the experiment.

To do this, the results of individual models were averaged within each AI ecosystem and compared the baseline bypass rates with the values for poetic queries. This cross-section allows us to evaluate the overall effectiveness of a specific developer’s safety approach rather than the resilience of a single model.

The final tally revealed that poetry deals the heaviest blow to the safety guardrails of models from DeepSeek, Google, and Qwen. Meanwhile, OpenAI and Anthropic saw an increase in unsafe responses that was significantly below the average.

What does this mean for AI users?

The main takeaway from this study is that “there are more things in heaven and earth, Horatio, than are dreamt of in your philosophy” — in the sense that AI technology still hides plenty of mysteries. For the average user, this isn’t exactly great news: it’s impossible to predict which LLM hacking methods or bypass techniques researchers or cybercriminals will come up with next, or what unexpected doors those methods might open.

Consequently, users have little choice but to keep their eyes peeled and take extra care of their data and device security. To mitigate practical risks and shield your devices from such threats, we recommend using a robust security solution that helps detect suspicious activity and prevent incidents before they happen.

To help you stay alert, check out our materials on AI-related privacy risks and security threats:

• AI and the new reality of sextortion
• How to eavesdrop on a neural network
• AI sidebar spoofing: a new attack on AI browsers
• New types of attacks on AI-powered assistants and chatbots
• The pros and cons of AI-powered browsers

Infostealers — malware that steals passwords, cookies, documents, and/or other valuable data from computers — have become 2025’s fastest-growing cyberthreat. This is a critical problem for all operating systems and all regions. To spread their infection, criminals use every possible trick to use as bait. Unsurprisingly, AI tools have become one of their favorite luring mechanisms this year. In a new campaign discovered by Kaspersky experts, the attackers steer their victims to a website that supposedly contains user guides for installing OpenAI’s new Atlas browser for macOS. What makes the attack so convincing is that the bait link leads to… the official ChatGPT website! But how?

The bait-link in search results

To attract victims, the malicious actors place paid search ads on Google. If you try to search for “chatgpt atlas”, the very first sponsored link could be a site whose full address isn’t visible in the ad, but is clearly located on the chatgpt.com domain.

The page title in the ad listing is also what you’d expect: “ChatGPT™ Atlas for macOS – Download ChatGPT Atlas for Mac”. And a user wanting to download the new browser could very well click that link.

The Trap

Clicking the ad does indeed open chatgpt.com, and the victim sees a brief installation guide for the “Atlas browser”. The careful user will immediately realize this is simply some anonymous visitor’s conversation with ChatGPT, which the author made public using the Share feature. Links to shared chats begin with chatgpt.com/share/. In fact, it’s clearly stated right above the chat: “This is a copy of a conversation between ChatGPT & anonymous”.

However, a less careful or just less AI-savvy visitor might take the guide at face value — especially since it’s neatly formatted and published on a trustworthy-looking site.

Variants of this technique have been seen before — attackers have abused other services that allow sharing content on their own domains: malicious documents in Dropbox, phishing in Google Docs, malware in unpublished comments on GitHub and GitLab, crypto traps in Google Forms, and more. And now you can also share a chat with an AI assistant, and the link to it will lead to the chatbot’s official website.

Notably, the malicious actors used prompt engineering to get ChatGPT to produce the exact guide they needed, and were then able to clean up their preceding dialog to avoid raising suspicion.

The infection

To install the “Atlas browser”, users are instructed to copy a single line of code from the chat, open Terminal on their Macs, paste and execute the command, and then grant all required permissions.

The specified command essentially downloads a malicious script from a suspicious server, atlas-extension{.}com, and immediately runs it on the computer. We’re dealing with a variation of the ClickFix attack. Typically, scammers suggest “recipes” like these for passing CAPTCHA, but here we have steps to install a browser. The core trick, however, is the same: the user is prompted to manually run a shell command that downloads and executes code from an external source. Many already know not to run files downloaded from shady sources, but this doesn’t look like launching a file.

When run, the script asks the user for their system password and checks if the combination of “current username + password” is valid for running system commands. If the entered data is incorrect, the prompt repeats indefinitely. If the user enters the correct password, the script downloads the malware and uses the provided credentials to install and launch it.

The infostealer and the backdoor

If the user falls for the ruse, a common infostealer known as AMOS (Atomic macOS Stealer) will launch on their computer. AMOS is capable of collecting a wide range of potentially valuable data: passwords, cookies, and other information from Chrome, Firefox, and other browser profiles; data from crypto wallets like Electrum, Coinomi, and Exodus; and information from applications like Telegram Desktop and OpenVPN Connect. Additionally, AMOS steals files with extensions TXT, PDF, and DOCX from the Desktop, Documents, and Downloads folders, as well as files from the Notes application’s media storage folder. The infostealer packages all this data and sends it to the attackers’ server.

The cherry on top is that the stealer installs a backdoor, and configures it to launch automatically upon system reboot. The backdoor essentially replicates AMOS’s functionality, while providing the attackers with the capability of remotely controlling the victim’s computer.

How to protect yourself from AMOS and other malware in AI chats

This wave of new AI tools allows attackers to repackage old tricks and target users who are curious about the new technology but don’t yet have extensive experience interacting with large language models.

We’ve already written about a fake chatbot sidebar for browsers and fake DeepSeek and Grok clients. Now the focus has shifted to exploiting the interest in OpenAI Atlas, and this certainly won’t be the last attack of its kind.

What should you do to protect your data, your computer, and your money?

• Use reliable anti-malware protection on all your smartphones, tablets, and computers, including those running macOS.
• If any website, instant message, document, or chat asks you to run any commands — like pressing Win+R or Command+Space and then launching PowerShell or Terminal — don’t. You’re very likely facing a ClickFix attack. Attackers typically try to draw users in by urging them to fix a “problem” on their computer, neutralize a “virus”, “prove they are not a robot”, or “update their browser or OS now”. However, a more neutral-sounding option like “install this new, trending tool” is also possible.

• Never follow any guides you didn’t ask for and don’t fully understand.

• The easiest thing to do is immediately close the website or delete the message with these instructions. But if the task seems important, and you can’t figure out the instructions you’ve just received, consult someone knowledgeable. A second option is to simply paste the suggested commands into a chat with an AI bot, and ask it to explain what the code does and whether it’s dangerous. ChatGPT typically handles this task fairly well.

How else do malicious actors use AI for deception?

• AI sidebar spoofing: a new attack on AI browsers
• Attacks using Syncro & AI-generated websites
• How phishers and scammers use AI
• Trojans masquerading as DeepSeek and Grok clients
• How fraudsters bypass customer identity verification using deepfakes

| Bronwen Aker // Sr. Technical Editor, M.S. Cybersecurity, GSEC, GCIH, GCFE Go online these days and you will see tons of articles, posts, Tweets, TikToks, and videos about how […]

The post Crafting the Perfect Prompt: Getting the Most Out of ChatGPT and Other LLMs appeared first on Black Hills Information Security, Inc..