Artificial intelligence with generative capabilities (gen AI) is at the forefront of cybersecurity, with red teams playing a crucial role in pinpointing vulnerabilities that others may miss.
Considering that the mean price of a data breach has surged to a record $4.88 million in 2024, organizations must be aware of their vulnerabilities. Given the rapid adoption of gen AI, some vulnerabilities may reside within AI models themselves or in the data used to train them.
This is where AI-specific red teaming becomes important. It involves testing the resilience of AI systems against dynamic threats by simulating real-world attack scenarios. This testing is essential to ensure that organizations can leverage the advantages of gen AI without introducing additional risks.
IBM’s X-Force Red Offensive Security service employs an iterative approach with ongoing testing to address vulnerabilities in four key areas:
- Testing of model safety and security
- Testing of gen AI applications
- Security testing of AI platforms
- Security testing of MLSecOps pipelines
In this piece, we will delve into three kinds of adversarial attacks that focus on AI models and training data.
Injection Prompt
Most mainstream gen AI models are designed with protective measures to mitigate the risk of generating harmful content. However, methods like prompt injection attacks and jailbreaking can bypass these safeguards.
One objective of AI red teaming is to purposely induce misbehavior in AI systems, akin to how attackers operate. Jailbreaking is a method that involves clever prompting to subvert a model’s safety filters. Though jailbreaking could aid in committing actual crimes theoretically, malicious actors typically use more effective attack vectors.
Injection prompt attacks are considerably more severe as they target the entire software supply chain by hiding malicious instructions in seemingly harmless prompts. For instance, an attacker could exploit prompt injection to extract sensitive details like an API key from an AI model, potentially granting unauthorized access to connected systems.
Red teams can also mimic evasion attacks, a form of adversarial attack where modifications are made to inputs to deceive a model into misinterpreting instructions. Although these modifications are often imperceptible to humans, they can trick an AI model into taking unwanted actions.
Discover X-Force Red Offensive Security Services
Data Contamination
Attackers also focus on AI models during the training and development stages, making it essential for red teams to replicate these attacks to identify potential risks that could compromise the entire project.
A data poisoning attack occurs when a malicious actor introduces corrupt data to the training set, thereby compromising the learning process and introducing vulnerabilities into the model. If the training data is compromised, it often necessitates complete retraining of the model, a resource-intensive and time-consuming task.
Early red team involvement is crucial in mitigating data poisoning risks during AI model development. Red teams execute real-world data poisoning attacks in a secure sandbox environment isolated from operational systems to assess the model’s susceptibility to such attacks and how actual threat actors might infiltrate the training process.
AI red teams can also identify weaknesses in data collection pipelines proactively. Large language models (LLMs) often draw data from diverse sources, making it vital for organizations to scrutinize the origin and quality of training data thoroughly.
Model Reversal
Proprietary AI models are frequently trained, at least partially, on an organization’s internal data. Even when trained on anonymized data, privacy breaches can still occur due to model inversion attacks and membership inference attacks.
Post-deployment, gen AI models can retain traces of their training data. Model inversion attacks can enable attackers to reconstruct training data, potentially exposing confidential information in the process.
Membership inference attacks function similarly, involving predicting if a particular data point was part of the model training data. Red teams can evaluate AI models for their potential to unintentionally leak sensitive information directly or indirectly through inference, aiding in identifying vulnerabilities in the training data workflows.
Establishing Confidence in AI
Establishing trust in AI necessitates a proactive approach, with AI red teaming playing a pivotal role. By employing tactics like adversarial training and simulated model inversion attacks, red teams can uncover vulnerabilities that standard security assessments might overlook.
These insights enable AI developers to prioritize and implement preemptive measures to thwart real adversaries from exploiting these vulnerabilities. Ultimately, this leads to reduced security risks and enhances confidence in AI models, which are now integral to numerous critical business systems.