Artificial intelligence safety researchers have raised concerns after discovering a prompt that allegedly allowed ChatGPT to generate graphic and sexualised images despite built-in content restrictions.
British AI security company Mindgard said it uncovered the issue by modifying a widely shared prompt that was originally intended to produce harmless and humorous image results. According to the researchers, the altered instruction was able to bypass safeguards and generate disturbing content.
The findings were reported to OpenAI, the company behind ChatGPT, which said it had since introduced additional protections to prevent the chatbot from responding to similar prompts.
“After investigating this trend, we’ve introduced additional safeguards against this type of prompt,” OpenAI said in a statement, adding that it uses multiple layers of protection, including automated systems and human review, to block harmful content.
However, Mindgard claimed that even after the changes were implemented, slight variations of the prompt were still capable of producing concerning images. The researchers declined to publicly disclose the exact instructions used.
Mindgard founder Peter Garraghan said the results were troubling because the prompt itself appeared harmless, yet generated violent and sexualised imagery without specifically requesting such content.
He claimed some images contained graphic injuries, scenes suggestive of violence and sexually explicit themes. According to the researchers, the AI-generated images featured fictional adults rather than real individuals.
AI safety researcher Jim Nightingale, who discovered the issue, described the generated images as disturbing and said the experience left him emotionally shaken. He argued that the incident highlights the ongoing challenge of preventing advanced AI systems from producing harmful content when users find ways to circumvent restrictions.
The researchers also warned that while OpenAI had previously addressed concerns surrounding the creation of explicit deepfake-style images, alternative methods could still potentially bypass some safeguards.
OpenAI’s policies prohibit the generation of sexual violence, non-consensual intimate content, child sexual abuse material and attempts to evade safety protections. The company said it continues to monitor emerging threats and strengthen its systems against misuse.
Experts say the challenge of securing AI models remains complex because the technology does not truly understand human concepts such as intent, morality or context. Instead, it predicts outputs based on patterns learned from vast amounts of data.
AI governance specialist Rumman Chowdhury described the situation as a constant “cat-and-mouse” battle between developers improving safeguards and users seeking ways around them.
The incident comes amid growing scrutiny of artificial intelligence systems worldwide, with governments, regulators and security researchers calling for stronger protections as AI tools become increasingly powerful and accessible to the public.
