A company appears to have pulled its AI-powered teddy bear after the toy was found to have brought up inappropriate subjects including sexual positions and how to find a knife.
Called “Koma,” the $99 teddy bear comes from a tommaker called Follotoy. But last week, watchdog group USPIRG Education Fund flagged some troubling behavior from the product, which uses OpenAI’s GPT-4OA model as its default setting.
“We were surprised to find how quickly Koma would take a single sexual topic we introduced into conversation and run with it, simultaneously introducing his own sexual concepts while adding graphic detail,” PIRG discovered.
(Credit: Folotoy)
For example, if the term “kink” is brought up, the bear might come up with things like “blindfolds” and “playful hitting.” The group’s report added: “In another hour-long exchange, Koma discussed even more graphic sexual topics in detail, such as explaining different sexual positions, giving step-by-step instructions on a typical ‘knot for beginners’ to tie a partner, and describing role-play dynamics involving teachers and students and parents and children.”
The PIRG also discovered that Koma could sometimes provide instructions on how to search for dangerous items, such as knives, bullets, matches and plastic bags. In other cases, the toy will direct the child to find the parent.
“In our testing, it was clear that some toy companies are putting safeguards in their toys to make them behave more appropriately than chatbots available for adults. But we found that those safeguards vary in effectiveness—and, at times, can be broken entirely,” noting that the same AI products can collect user data.
Recommended by our editors
Follotoy seems to have since pulled the product. A marketing director also told the Register that the company has temporarily suspended sales of the product as it conducts an internal safety audit. “This review will cover our model safety alignment, content filtering system, data protection processes and child interaction safeguards,” the company added.
Openei also allegedly suspended access to Folotoy’s model for violating its policies.
Get our best stories!
Your daily dose of our top tech news
By clicking Sign Up, you confirm that you are 16+ years of age and agree to our Terms of Use and Privacy Policy.
Thanks for signing up!
Your membership has been confirmed. Watch your inbox!
About our expert
Michael Kahn
Senior reporter
Experience
I have been a journalist for over 15 years. I got my start as a schools and cities reporter in Kansas City and joined PCMag in 2017, where I cover satellite Internet services, cybersecurity, PC hardware and more. I am currently based in San Francisco, but previously spent over five years in China, covering the country’s technology sector.
Since 2020, I’ve covered the launch and explosive growth of SpaceX’s StarLink satellite Internet service, writing 600+ stories on availability and feature launches, but also regulatory battles over expanding satellite constellations, battles with rival providers like AST SpaceMobile and Amazon, and efforts to expand satellite-based mobile service. I’ve combed through FCC filings for the latest news and reached out to remote corners of California to test StarLink’s cellular service.
I also cover cyber threats, from ransomware gangs to the emergence of AI-based malware. Earlier this year, the FTC forced Avast to pay $16.5 million for secretly harvesting and selling its personal information to third-party clients, as revealed in my joint Investigation Along with the motherboard
I also cover the PC graphics card market. Pandemic-era shortages led me to camp out in front of a Best Buy to get an RTX 3000. Now I’m following how President Trump’s tariffs will affect the industry. I’m always eager to learn more, so please jump into the comments with feedback and send me pointers.
Read full bio
