How People Trick Generative Artificial Intelligence Chatbots into Exposing Company Secrets
Immersive Labs Unveils New ‘Dark Side of GenAI’ Report
Analysis of prompt injection techniques reveals organizations are at risk as Generative Artificial Intelligence (GenAI) bots are susceptible to attacks by users of all skill levels, not just experts
Immersive Labs published its “Dark Side of GenAI” report about a Generative Artificial Intelligence (GenAI)-related security risk known as a prompt injection attack, in which individuals input specific instructions to trick chatbots into revealing sensitive information, potentially exposing organizations to data leaks. Based on analysis of Immersive Labs’ prompt injection challenge, GenAI bots are especially susceptible to manipulation by people of all skill levels, not just cyber experts.
Among the most alarming findings was the discovery that 88% of prompt injection challenge participants successfully tricked the GenAI bot into giving away sensitive information in at least one level of an increasingly difficult challenge. Nearly a fifth of participants (17%) successfully tricked the bot across all levels, underscoring the risk to organizations using GenAI bots.
AI-powered Cyber Attacks Cast Unprecedented Threats on IT Leadership
This report states that public and private-sector cooperation and corporate policies should mitigate security risks. The extensive adoption of GenAI bots pose these prompt injection risks. Security leaders should take decisive action, including establishing comprehensive policies for Generative Artificial Intelligence use within their organizations.
Key Findings from Immersive Labs “Dark Side of GenAI” Study
The team observed the following key takeaways based on their data analysis, including:
- GenAI is no match for human ingenuity (yet): Users successfully leverage creative techniques to deceive GenAI bots, such as tricking them into embedding secrets in poems or stories or altering their initial instructions, to gain unauthorized access to sensitive information.
- You don’t need to be an expert to exploit GenAI: The report’s findings show that even non-cybersecurity professionals and those unfamiliar with prompt injection attacks can leverage their creativity to trick bots, indicating that the barrier to exploiting GenAI in the wild using prompt injection attacks may be easier than one would hope.
- As long as bots can be outsmarted by people, organizations are at risk: No protocols exist today to fully prevent prompt injection attacks. Cyber leaders and GenAI developers need to urgently prepare for – and respond to – this emerging threat to mitigate potential harm to people, organizations, and society.
The research team at Immersive Labs consisting of Dr. John Blythe, Director of Cyber Psychology; Kev Breen, Senior Director of Cyber Threat Intelligence; and Joel Iqbal, Data Analyst, analyzed the results of Immersive Labs’ prompt injection GenAI Challenge that ran from June to September 2023. The challenge required individuals to trick a GenAI bot into revealing a secret password with increasing difficulty at each of 10 levels. The initial sample consisted of 316,637 submissions, with 34,555 participants in total completing the entire challenge. The team examined the various prompting techniques employed, user interactions, prompt sentiment, and outcomes to inform its study.
AiThority AI Updates: QR Code-Based Attacks Are Growing in Popularity; Now Comprise 11% of All Malicious Emails
Comments are closed.