Considerations When Including AI Implementations in Penetration Testing

Originally published by Schellman.

Did you recently implement a new artificial intelligence (AI) feature within your application and now your customers are starting to ask for AI-specific penetration tests? Are you curious as to how an assessment like that would work? As with all these exercises, it starts with scoping.

Scoping goes beyond just deciding the boundaries of your test—there are other considerations you must make as part of your penetration test—which can also be referred to as “AI Red Teaming”—planning process.

In this blog post, we’re going to detail six questions you’ll need to ask and answer as part of scoping out a penetration test that includes AI systems. Whether you use our team to perform this exercise or not, you’ll be ready when it comes time to engage your testers in evaluating your AI.

AI-Focused Penetration Test Scoping FAQ

When scoping your penetration test engagement, here are some questions to ask and answer when considering including your AI systems for testing.

Are You Using an API from Established Providers?

You should know that when using an Application Program Interface (API) from proven Large Language Model (LLM) providers like OpenAI, Anthropic, etc., they take on the task of securing their API endpoints, external network, and—at some level—address data privacy concerns.

Despite that, you’ll still want to make sure you protect your API keys, take into consideration output handling—or how the response text is going to be displayed within your application—and have logging and monitoring enabled to make sure you don’t end up with a large bill from your chosen provider.

Are You Self-Hosting Large Language Models (LLMs)?

If you host your own open-source model from services link AWS Bedrock or Google Cloud’s Model Garden, a penetration test can still be valuable to you, but you’ll also need to need to ensure the security measures that are in place to protect the model from unauthorized access and even potential exploitation are correctly configured.

Did You Fine-Tune and Train A Model?

If you did in fact decide to fine-tune or train your own model(s) with services like Google Cloud’s Generative AI Studio, we advise that you make sure your penetration testers thoroughly evaluate:

• The data used for training;
• The potential for data leakage; and
• The robustness of the model against targeted attacks that exploit model-specific weaknesses.

What are Your Goals for an AI Systems Penetration Test?

Most people’s first answer will be that they need to satisfy a request for a pen test report, but when considering this kind of exercise and how to maximize the returns, you should take a moment to really think about the potential negative impact on your organization if your AI implementation was exploited—ask, “What’s the worst that could happen in your specific implementation of AI?”

Once you know that, work with your tester to find ways for them to successfully simulate creating that fallout as your penetration test should be a collaborative effort to make your organization—and AI systems—more resilient to adversarial attacks.

Moreover, a penetration test shouldn’t stop with just the mere identification of a vulnerability—encourage your testers to fully show the impact of a finding (within the agreed-upon scope). Once you understand what’s possible, you can start to remediate the root cause.

How Will AI Penetration Testing Fit Into Your Broader Security Strategy?

If you’ve already established quarterly or annual penetration testing that’s aligned with your organization’s overall cybersecurity posture or compliance requirements, your AI could be tested in tandem.

But an AI engagement also pairs nicely with the recently published ISO 42001 standard and the related certification, which you can get started with through a gap assessment. Getting certified together with having an AI penetration test performed can go a long way to show your customers that you take security—and their data—seriously.

What Testing Methodology Will Be Used?

While not a question you’ll need to ask yourself, you may be curious about how your pen testers will approach your assessment. And though guidance for penetration testers to use when examining AI continues to evolve rapidly, documentation does exist to help them right now, including:

• OWASP Top 10 for Large Language Model Applications
• MITRE ATLAS
• NIST AI 100-2 - Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations

While these publications help testers use correct terminology and focus on the weaknesses that apply to your AI use case, it’s also worth noting that their attention will not be not solely on the AI solution itself—when you add AI as another feature within an existing application, vulnerabilities highlighted within the OWASP Web Security Testing Guide will still be very much in play.

Next Steps

While these questions represent some of the aspects you’re likely wondering about regarding a potential penetration test of your AI systems, the only way to get a complete picture of what yours will look like is to speak to a tester and undergo a scoping exercise.

In the meantime, here are some answers to other questions pertaining to your potential AI penetration test:

• How to Set Your Penetration Test Scope
• What Does a Penetration Test Cost? Scope Factors That Matter
• How Long Does a Penetration Test Take?