Hackers Exploit ‘Prompt Injection’ Technique to Manipulate AI Systems


  • Hackers exploit ‘prompt injection’ to control AI systems with simple English commands, posing security risks.
  • New cyberattacks redefine hacking as AI technologies proliferate, necessitating vigilant security measures.
  • Companies open AI systems to hackers at Defcon conference, highlighting the need for robust safeguards.

Hackers have discovered a new technique called ‘prompt injection’ that exploits the responsiveness of AI systems, enabling them to manipulate these systems using simple English commands. One notable instance involves Johann Rehberger, a security researcher, who successfully coaxed OpenAI’s ChatGPT into performing unauthorized actions, such as reading an email, summarizing its content, and posting it online. This approach highlights the potential security risks associated with AI technologies and the need for vigilance in protecting against such vulnerabilities.

AI’s vulnerability to prompt injection attacks

AI systems like ChatGPT have gained immense popularity, attracting over 100 million users due to their ability to respond to basic commands swiftly. However, this convenience has also attracted hackers exploiting AI’s limitations. Rehberger demonstrated that hackers could compromise AI systems using plain language, making it possible to manipulate the technology without requiring extensive coding skills or deep computer science knowledge. In this context, the attack doesn’t necessarily target all ChatGPT users but rather leverages specific features or vulnerabilities.

The prompt injection technique

Prompt injection, the technique employed by Rehberger, is part of a new breed of cyberattacks that has become more significant as AI technologies proliferate in various industries and consumer products. These attacks exploit vulnerabilities that can redefine the notion of hacking in an AI-driven landscape. Security researchers are racing against time to identify and address these vulnerabilities before malicious actors exploit them extensively.

Concerns and types of attacks

Various concerns surround AI security breaches. “Data poisoning” attacks, where hackers tamper with training data to mislead AI models, pose a risk to reliable model outcomes. Among the emerging concerns are ethical bias in AI systems, disinformation campaigns, extraction attacks leading to corporate data leaks, and using AI to breach defensive security systems.

Challenges of protecting AI systems

Efforts to secure AI systems are challenging due to the evolving nature of these technologies. While creators attempt to anticipate misuse scenarios, new techniques like prompt injection continue to emerge. Even well-protected systems like Google’s VirusTotal, which uses AI for malicious software analysis, aren’t immune to attacks. Google’s system was deceived by a hacker who manipulated code to describe malicious software as harmless.

The upcoming Defcon hacking conference

Companies like OpenAI, Google, and Anthropic are set to open their AI systems to hackers during the annual Defcon hacking conference in Las Vegas. This event will allow hackers to identify vulnerabilities in these AI systems and earn rewards for their successful attacks. The conference highlights the need for robust security measures to protect AI technologies from exploitation.

Implications and future security measures

As AI-powered devices and applications become more integrated into daily life, the potential avenues for hackers to exploit them also increase. The reliance on language models to make decisions could lead to more avenues for manipulation, as highlighted by Arvind Narayanan, a computer science professor at Princeton University. Therefore, ensuring the security of AI systems involves not only addressing vulnerabilities but also understanding the far-reaching consequences of AI-related breaches.

The advent of prompt injection attacks and other AI-related vulnerabilities underscore the need for ongoing vigilance and security measures in the AI landscape. The incident with ChatGPT reveals that hackers can manipulate AI systems using simple language without requiring advanced technical skills. Security researchers, technology companies, and the wider community must collaborate to anticipate, detect, and mitigate such threats, safeguarding the potential benefits of AI while minimizing risks.

Disclaimer. The information provided is not trading advice. Cryptopolitan.com holds no liability for any investments made based on the information provided on this page. We strongly recommend independent research and/or consultation with a qualified professional before making any investment decisions.

Share link:

Glory Kaburu

Glory is an extremely knowledgeable journalist proficient with AI tools and research. She is passionate about AI and has authored several articles on the subject. She keeps herself abreast of the latest developments in Artificial Intelligence, Machine Learning, and Deep Learning and writes about them regularly.

Most read

Loading Most Read articles...

Stay on top of crypto news, get daily updates in your inbox

Related News

Subscribe to CryptoPolitan