Introduction

Enterprises are increasingly adopting GenAI and LLM technologies, creating a need for security measures to address the challenges arising from both the models and the training data that supports them. This paper applies ZT principles to secure LLM environments, protecting both the model and data.
The rapid adoption of GenAI services, coupled with the lack of industry-wide standards for securing such technologies and the continuous evolution of the underlying models and tools, presents unique security challenges.

The National Security Telecommunications Advisory Committee (NSTAC) Report to the President on Zero Trust and Trusted Identity Management defines Zero Trust as “a cybersecurity strategy premised on the idea that no user or asset is to be implicitly trusted. It assumes that a breach has already occurred or is likely to occur. Therefore, an entity should not be granted access to sensitive information based solely on a single verification performed at the enterprise perimeter. Instead, each user, device, application, and transaction must be continually verified.”

Traditional perimeter-based security models are insufficient for dynamic, data-driven environments, as they struggle to keep pace with the innovativeness of malicious actors. For example, perimeter-based security trusts the authorization tokens once the identity has been authenticated. However, breaches may be caused by stolen tokens, which are not well factored into perimeter-based security models. ZT assumes no implicit inherent trust, assumes breach, and enforces strict verification and authentication before granting access to data and devices, as well as data exchanges. By adopting a ZT approach, organizations can mitigate these risks. LLM environments also face these risks, as they have datasets that may be part of the organization’s protect surface(s). Hence, it is essential to secure enterprise information in LLM environments, and Zero Trust principles can be employed to achieve this.

Scope

This document explores the integration of Zero Trust principles to secure enterprise information within LLM environments, concentrating on the following key areas:

• Application of Zero Trust to LLM Environments: Examining how Zero Trust can enhance the security of LLM systems, with a focus on access controls, applications, data validation, and real-time monitoring

• Securing Model and Data Integrity: Discussing strategies to ensure the integrity of both LLM models and the data used in training, fine-tuning, and inference, while preventing adversarial attacks and data leakage

• Securing the Application Layer: Securing the application layer to detect and prevent compromises, such as data exfiltration

• Securing the Integration Layer: Securing any integration layer when LLM environments utilize the “bring data to the model” approach to synchronize datasets with databases in enterprise environments

• Managing Third-Party Dependencies: Outlining how Zero Trust can secure third-party tools, APIs, and external services that interact with LLMs, mitigating associated supply chain risks

• Continuous Monitoring and Verification: Highlighting the role of continuous verification and monitoring within Zero Trust frameworks to safeguard against unauthorized model access and manipulations

• Best Practices for Implementation: Providing actionable guidance for organizations on how to integrate Zero Trust security practices into LLM deployment, with considerations for performance and scalability

• AI Output Quality: Establishing quality assurance for AI-generated outputs and the propagation of inaccurate or harmful content. This is especially critical when proprietary or sensitive datasets are used for training or fine-tuning models

• Content Governance and Moderation: Implementing controls to monitor, validate, and moderate AI outputs in alignment with enterprise policies, regulatory requirements, and ethical guidelines

The guidance provided in this document is applicable across various sectors and industries.
The scope for this document does not extensively cover AutoML, such as AI models that generate code, or deep dives into shadow AI and agentic AI, EdgeAI and IoT, robotics, gaming, explainable AI, AI ethics, AI collaboration tools, AI in biometrics, AI in cybersecurity, quantum, chatbots, hallucination, or bias.

What is an LLM Environment?

An LLM environment is the ecosystem in which large language models are developed, deployed, and maintained. These environments may be operated internally in a self-hosted model or consumed externally as Software-as-a-Service (SaaS) offerings. Because they process both organizational and end-user data, understanding their use cases and security requirements is critical.
From a generic view, an LLM environment includes the following layers:

• Datasets and Databases: Data sources and storage systems that provide training, fine-tuning, and inference inputs

• Integration Layer: Integration components that sync data between on-premise databases/data warehouses/data lakes and the datasets hosted in the AI workspace

• Models and Algorithms: LLM architectures, pre-trained or fine-tuned models, and supporting AI/ML variations

• Application Layer: Interfaces and applications that enable interaction with models, supporting both human and machine collaboration

• Hosting Environment: Data centers or cloud platforms, both public and private

• Infrastructure: Compute resources (e.g., CPUs, GPUs, TPUs) and accelerators

• Network Connectivity: Network infrastructure that provides the necessary connectivity between data and models

• End Users/Clients: Internal users, external customers, or client-side processes (e.g., web assemblies, service workers, browser threads) that consume model outputs

Figure 1: Example of Layered Architecture of an LLM Environment

Types of Organizational Data

Enterprise data may be hosted with the organization’s own infrastructure or in a service provider’s environment. Based on this distinction, the following scenarios emerge:

• Enterprise Data Used for Training LLMs: Data drawn from organizational sources is incorporated into training processes. Protecting this data is critical, as it may contain sensitive or proprietary information

• Training, Benchmarking, Testing, and Validation Data: The data sets used to train, benchmark, test, and validate the model comprise text sources from which the model derives insights into language patterns and semantics essential to its quality and effectiveness. Each data element is managed individually to maintain integrity and traceability

• Fine-Tune Training Data: Following initial training, additional data may be applied to fine-tune or further pre-train the model. This process adjusts parameters to align the model more closely with domain-specific requirements, enhancing both adaptability and accuracy

• Retrieval-Augmented Generation (RAG): An approach that integrates external knowledge bases with LLMs. By retrieving relevant information before generating responses, RAG combines model knowledge with external sources. It can retrieve supplementary data from internal systems or public sources, such as the internet, enriching prompts, refining context, and producing higher-quality responses

• Model Parameters (Weights): The internal parameters acquired during training that govern the model’s behavior and determine how effectively it generates contextually relevant responses

• Model Hyperparameters: Training configurations, including learning rate, batch size, or architecture choices, that critically influence the model’s performance and efficiency

• User Session Data: Information collected during user interactions with AI systems, including input queries, model-generated responses, and any additional user-provided context. This data supports personalization:

  • AI (User) Prompts (Queries): Inputs provided by users to an AI system in the form of questions, instructions, or contextual information that guide the model’s response

  • AI Responses—Model Output Data: The outputs generated by the model in response to user prompts, including text, predictions, or other processed results that reflect the model’s comprehension and inference capabilities

• Log Data: Recorded data on events and interactions during the model’s operation, including prompts, responses, performance metrics, and any errors or anomalies. This data is essential for monitoring and refining the model’s functionality and performance

Potential risks associated with these data types are described in theThreat Scenarios section, including control considerations discussed in the Zero Trust for LLM Environments section.

Why Do We Need to Secure LLM Environments?

LLMs are increasingly being used to generate AI use cases. While this aspect brings business benefits, it also introduces security risks, such as whether the right models are being used, the possibility of employing poisoned models (models compromised by malicious data or deliberate manipulation during training), the accuracy of the data, unintended leakage or sharing of data, and whether the organization’s data is being used for training.

The challenge lies in securing models, data, and environments, both for organizations consuming AI services and providers responsible for sourcing models and training data. This is because consumers, without any intent, may unknowingly contribute enterprise data from their endpoints for training through interactions with voice assistants (e.g., Amazon Alexa) or smart home platforms, raising issues of consent, monitoring, and continuous data harvesting.

There are two key reasons to secure data in LLM environments: 1) to secure enterprise data from threat actors that act directly on data, such as data exfiltration, compromising confidentiality, integrity, and availability of data, poisoning the dataset, or unknowingly providing enterprise data to train LLM model; and, 2) an indirect threat in the form of models that do not provide the required functionality, which could be due to poisoning, bias, hallucinations (outputs that appear plausible but are factually incorrect), or other issues. Such models have an indirect impact on the enterprise data and any derived data from the model.

Self-Hosted LLM Environments Versus SaaS Implementation

While using LLM environments, data can be hosted in a self-hosted environment (hosted by organizations themselves and not by a third party) that is on Infrastructure as a Service (IaaS) in a public cloud (e.g., AWS) or on-premises (i.e., private cloud), and the data is completely controlled by the organization.

Data hosted in a supplier’s LLM environment (using a SaaS product), would be under a shared responsibility model, supported by contractual obligations and security assurances, unlike data persisted in self-hosted environments (e.g., public cloud, private cloud).

Figure 2: Securing Data with Zero Trust — SaaS & Self-Hosted Data Environments

Organizations have complete control over enterprise data in a private cloud, as there is little to no access for a service provider. Service providers may have access to data within LLM environments hosted on IaaS. However, when the LLM environments are hosted by SaaS environments, issues that arise include:

• Without proper contractual obligations, a supplier may use an organization’s enterprise data to train its models, potentially exposing that data beyond the supplier’s environment
• In the absence of contractual obligations, the extent to which an organization can control the use of its data for LLM training is uncertain
• Suppliers and their hosting providers or subcontractors must demonstrate compliance with the required standards and legal requirements, including the GDPR, the EU AI Act, the EU Data Act, PCI-DSS, and ISO 27001:2022
• Data sovereignty, including metadata, may pose challenges, particularly in the context of cross-boundary complexities
• Smart home owners that utilize home automation devices, such as Amazon Echo powered by Alexa, may disclose enterprise information through conversations from user/employee endpoints, which in turn may be used by suppliers to train their models
• Data isolation may not be enforced by an organization in the context of multi-tenancy
• Prompt injection may extract unrelated information and present incorrect results, potentially disclosing sensitive information about an organization’s enterprise
• Without appropriate guardrails, denial-of-service attacks may occur at the infrastructure layer or the application layer when uncontrolled requests are sent to the LLM environment

Based on the above, it is essential to understand the threats and the corresponding controls that should be applied to ensure the proper and secure deployment of LLM environments.

Legislative Approaches to AI and Data Access

The information provided reflects legislation that was either pending or enacted at the time this document was drafted. Because laws and regulations are subject to change, readers should verify the most current information.

A key component of AI apps is to leverage information to train and improve their models, which raises an important legal question: Do AI apps need permission from users to utilize their content?

The United States passed the National Artificial Intelligence Initiative Act of 2020.

Several AI bills have been introduced, including the PRO Codes Act, the Transparency and Responsibility for Artificial Intelligence Networks (TRAIN) Act, the Generative AI Copyright Disclosure Act of 2024, and the No AI FRAUD Act. These proposals are under consideration and have not been enacted. Enacted AI legislation in the US has been primarily at the state level.

The European Parliament approved the EU Artificial Intelligence Act (AI Act) in 2024 to ensure AI systems are safe, transparent, traceable, non-discriminatory, and environmentally sustainable. Its primary aim is to regulate AI in Europe while safeguarding copyright, especially against unauthorized use by general-purpose AI (GPAI) providers.

The EU Data Act took effect in September 2025 to enhance the efficiency of data, particularly in the industry sector. The act is not directly associated with AI; however, it does affect AI when data from IoT devices is used to train AI models. Users may request their device data from companies or have it sent to another service, thereby promoting innovation and competition.

Shadow AI

The discovery of shadow AI, inclusive of intentionally bypassed governance and accidental non-compliance, presents a challenge for enterprises. Shadow AI is similar to shadow IT, occurring when users (e.g., employees) either do not recognize the implications of their work or pursue scenarios that the enterprise would not accept. It is challenging to detect due to indicators that are hidden in plain sight.

Typical ways of working with shadow AI include:

• Developers or data scientists using self-hosted AI models connected to live databases may share database connection strings with non-DBA staff, which should be addressed as an educational point. Detection is challenging unless the SOC team actively monitors database connection initiations
• A similar model, where SaaS applications connect to live databases, is also challenging to detect
• Employees uploading files into untracked LLM environments or copying content from a file and using it as a prompt on an untracked LLM environment

More information on this topic can be obtained from:

• Shadow Access and AI | Address Unauthorized Access | CSA

Agentic AI

Agentic AI systems represent a unique security challenge because they combine LLM capabilities with direct enterprise tool integrations. When implementing these systems:

• LLM Security Boundaries: Unlike standard LLM deployments, agentic AI extends the LLM’s reach through direct API connections to enterprise systems, expanding potential data exposure beyond traditional boundaries

• Model Integrity Concerns: The trustworthiness of the underlying LLM becomes critical when it can execute actions across enterprise tools. A poisoned or biased model may make harmful decisions that propagate through connected systems

• Prompt Injection Amplification: In agentic systems, malicious prompt injection attacks against LLMs have a greater impact, potentially allowing attackers to manipulate the agent into performing unauthorized actions across connected enterprise tools

• Data Flow Security: Sensitive data processed by the LLM may move through multiple components in the agentic workflow (e.g., knowledge bases, enterprise tools), requiring comprehensive security controls at each transfer point

• Authorization and Least Privilege Enforcement: Agentic AI must operate under tightly scoped permissions. Access to enterprise systems should be role-based and action-specific to prevent misuse and privilege escalation

• Continuous Monitoring and Auditability: All agent actions and cross-system data flows must be logged, monitored in real-time, and correlated to detect anomalies

More information about agentic AI can be obtained from the following CSA publications:

• Agentic AI Red Teaming Guide | CSA
• Agentic AI Identity and Access Management: A New Approach | CSA
• Secure Agentic System Design - A Trait-Based Approach | CSA

Model Context Protocol

The Model Context Protocol (MCP) is an emerging standard to enable interoperability between LLMs and external systems. It provides a structured way for LLMs to access context, integrate with tools, and trigger automated actions reliably and securely.

As organizations move toward LLM-driven automation, the MCP is increasingly being adopted. It allows LLM-based tools to:

• Ingest contextual data from multiple sources
• Integrate seamlessly with enterprise applications, APIs, and knowledge bases
• Automate workflows and actions with traceability and governance

This aligns closely with agentic AI, where LLMs function not just as conversational assistants but as autonomous agents capable of reasoning, decision-making, and executing tasks across interconnected systems.

More information about MCP can be obtained from the following publications:

• Model Context Protocol Security
• Architecture Overview - Model Context Protocol

AI Deployment Model Use Cases

LLM use cases can be implemented in different deployment models. Some operate within SaaS environments, others are deployed in self-hosted scenarios, and some adopt a hybrid approach. The following table illustrates how LLMs are applied across different AI use cases:

Table 1: LLM Deployment Models, Use Cases, and Zero Trust Security Considerations

SaaS deployments may raise issues of shared responsibility and contractual safeguards, while self-hosted models require stronger controls over access, monitoring, and system hardening. Hybrid models combine aspects of both, creating unique risks and challenges. Additionally, emerging domains like agentic AI introduce additional challenges, as LLMs gain the ability to reason, make decisions, and act autonomously across enterprise systems. These use cases require closer examination, including an analysis of examples such as the DeepSeek model and the unique risks it presents.

LLM Deployment Risks by Type

• Location/Hosting Vendor

  • On-Premises: Strong compliance/audit control, minimal data exposure
  • Cloud/IaaS: Vendor dependency, cross-border residency, segmentation risks

• Integration with Other Data Sources

  • Data Risks: Leakage, misconfiguration, and unintended usage of data

• Solution Type

  • Single-Tenant SaaS: Isolated by contract, risk of the vendor using the model training
  • Multi-Tenant SaaS: Tenant isolation and inference risks

• Management Model

  • Organization-Managed: Internal teams own compliance/security
  • Vendor-Managed: Privacy, governance, and vendor data usage concerns

Threat Scenarios

Threats to Enterprise Data

Data Poisoning

Malicious actors may manipulate or corrupt training datasets to compromise the behavior, accuracy, or integrity of AI/ML models.

Data Exfiltration

Inadequate monitoring and weak access controls can facilitate unauthorized data extraction. For example, having read access on Azure Storage Accounts can allow the user to download data to their device. Although information labeling can be implemented with tools like Microsoft Purview Data Loss Prevention (DLP) in Azure, tools like Information Rights Management (IRM) may not function effectively.

Capabilities of Solutions Missed in Solution Architecture

Solutions are realized using tools. It is essential to verify the access control capabilities of the tools and the preventive security controls that can be applied to them to ensure that prompts do not attempt to exfiltrate unauthorized data and use an authorization process that is independent to the LLM models. For example, an employee is trying to exfiltrate the performance records of other employees using an AI prompt. Although this threat can be addressed by applying keyword filtering on an outbound proxy to filter the output, it is equally essential to ensure that access control is applied at the dataset layer to ensure that extracting data with queries is a fail-safe process. One way to start addressing this threat is to understand the type of data (e.g., structured, unstructured, semi-structured) and the access controls or preventive security controls that can be applied. For example, one preventive security control for structured data typically is query parameterization, which may not work well with unstructured data. (Testing for NoSQL injection - OWASP).

This should be seen with the background of any denial of information to authorized users, because too much filtering may lead to presentation of undesirable output. Any adoption of LLM environments for sensitive data must be analyzed thoroughly and any prompt injections should be considered. For example, if HR data is used in LLM environments and an employee seeks to understand their manager’s salary, a malicious prompt injection may lead to the threat materializing. In such a scenario, authentication and authorization would mitigate such risk. An employee asking for their manager’s salary should not be authorized to view the information. But it depends on how well authorization is designed and implemented. With very sensitive data, it may be beneficial not to use it in an LLM environment.

Supply Chain Vulnerabilities

Extend beyond conventional software dependencies to include pretrained models and the data sources used to train them. Exploitation of these elements can undermine model integrity and introduce systemic security failures.

Additional Threats

Data in LLM environments may also be exposed to risks from malware, a lack of shared security responsibility execution in SaaS environments, inadequate system hardening, and prompt injection attacks that manipulate model inputs to override safeguards, bypass policies, or extract sensitive enterprise data.

The following attack vectors illustrate how the threats described above may materialize:

• A malicious, unauthorized actor gains access to the network segment hosting the data or to the database itself

• A malicious, unauthorized actor gains access to the database and attempts to exfiltrate data to either a personal laptop or an external storage device. While exfiltration to an external storage can be prevented by adding network rules to prevent outbound internet connectivity, the same cannot be said of actors downloading data to their local devices

• A malicious actor can utilize a prompt injection to manipulate model inputs, causing the LLM to reveal sensitive data or execute unauthorized API operations. These attacks may bypass system prompts using “jailbreaking” techniques or by embedding malicious instructions in external sources. The impact can include data leakage, unauthorized system actions, or proxies that enable attacks on other systems or users within the enterprise network

• Common attack vectors for supply chain vulnerabilities include the injection of poisoned crowd-sourced data, reliance on insecure pre-trained models during fine-tuning, and the use of outdated components that lack necessary security updates

• A malicious actor could include but is not limited to services, humans, autonomous AI agents, and agentic AI

Threats from Models

Threats from models can occur in several ways. Since models may be either acquired externally or developed in-house, supply chain risk is a reality for both scenarios.

Supply chain risk vectors for models include:

• Poisoned models may be obtained from unvetted or untrusted sources

• Poisoned models can be introduced through typosquatting, where attackers publish malicious models or libraries with names that closely resemble legitimate ones, tricking developers into downloading and using the compromised version

• When developers create models, they typically import libraries provided by programming language frameworks. For example, TensorFlow, PyTorch, Scikit-Learn (including commonly used classes such as LinearRegression). If these libraries are compromised or poisoned, the resulting models inherit the vulnerability, creating a supply chain risk. This type of threat is often referred to as a dependency or supply chain attack

• The compromised or poisoned models may lead to various impacts, including, but not limited to, incorrect processing and results, data exfiltration if malicious code is embedded in libraries, data poisoning, and hallucinations

• Prompt injections are specialized threat vectors for LLM environments. Malicious prompts can manipulate a model into sharing information not intended for the end-user

• Failure to comply with frameworks such as the EU’s AI Act, GDPR, or US state-level laws like the CCPA results in governance gaps that heighten the risk of sensitive data exposure

Threats from the Application and API Layer

The results from the prompts or any operations conducted using models are rendered on a web page that is a part of a web application. In this scenario, the following threat vectors are applicable:

• Injecting malicious content in prompts leading to unauthorized disclosure of information

• Injecting malicious prompts that may compromise the back-end systems with excess consumption of resources, such as setting the back-end resources in an endless loop

• Phishing of AI/AI-based applications and products leading to incorrect information being served to end-users. It could also lead to consuming information added as prompts and using the data to train their models. This threat vector is fairly theoretical, but may materialize in edge cases

• Lack of security hygiene by SaaS product owners can lead to the compromise of credentials through brute-force attacks, session-related vulnerabilities such as session hijacking or CSRF, and other security issues

• The reliance on third-party plugins, datasets, models, and libraries can compromise the AI system, causing a supply chain vulnerability

• Malicious actors can exploit embedded pipeline and vector database flaws

More information about SaaS can be obtained from the following CSA publications:

• SaaS Challenges, Solutions, and Best Practices for 2024 | CSA
• Cloud Security for SaaS Startups | CSA

Threats from the Infrastructure Layer

Regardless of whether the enterprise or a supplier hosts the environment, threat vectors exist for the underlying infrastructure, typically including:

• Lack of Infrastructure Hardening: Can expose devices to risk, including but not limited to:

  • Unintentionally exposed services or open ports
  • Vulnerabilities in devices exposed to the public network

• Lack of Network Segmentation: This increases the risk of lateral movement, allowing attackers to expand their access once inside the environment

• Insufficient Monitoring of File Uploads: May enable malware injection or the introduction of malicious code into the environment

Threats from the Integration Layer

Integration layers are used by enterprises where real-time data is not used as a dataset, but delta changes are synced with the datasets. Examples include Snowflake and Databricks, with integration layers provided by solution providers such as Confluent or PaaS solutions like Event Grid.

Threat vectors for the integration layer include:

• Misconfigured Connection String: Incorrectly configured connection strings in the integration layer can lead to the leakage of credentials and sensitive enterprise data

• Inherent Security Weaknesses: Vulnerabilities within the integration layer may lead to lateral movement or other downstream threats

Real-Life Incidents

While threat categories can seem abstract, real-world incidents show how these risks materialize in practice. The following are real-world examples of AI excesses, data leaks, and other compromises in LLM environments:

• Information Leaked via Prompt Injection:

  • Slack AI can leak private data via prompt injection (The Register)

• Organizations Exercising Caution About Sharing Confidential Information in SaaS LLM Environment:

  • Amazon warns employees not to share confidential information with ChatGPT after seeing cases where its answer ‘closely matches existing material’ from inside the company (Business Insider)

  • Samsung Bans ChatGPT, Google Bard, Other Generative AI Use by Staff After Leak (Bloomberg)

• Manipulation of Response from an AI Model by a Malicious Actor:

  • Hacker tricks chatbot into selling him a car for $1 (Upworthy)

  • Air Canada’s chatbot promised a discount. Now the airline has to pay it (Washington Post)

• Impact of Compromises of LLM Environments:

  • Alphabet shares dive after Google AI chatbot Bard flubs answer in ad (Reuters)

Zero Trust for LLM Environments

There are various interpretations of Zero Trust principles across industry and government guidance. For this document, we adopt the foundational principles as outlined in the NSTAC Report to the President on Communications Resiliency, emphasizing comprehensive visibility, least privilege access, continuous risk-based evaluation and authentication, and micro-segmentation.

In this section, we examine how these principles can be applied to securing enterprise information in LLM environments.

Figure 3: Reproduced from NSTAC. (2022). NSTAC Report to the President on Zero Trust and Trusted Identity Management. Retrieved from CISA.

Zero Trust Guiding Principles for LLM Environments

One of the essential steps in applying Zero Trust principles to LLM environments is understanding the protect surfaces (e.g., critical data, applications, assets, vector database, API endpoints, prompt interfaces, services), their risk rating, and their security status. Guidance on the identification of protect surfaces is available in the CSA publication Defining the Zero Trust Protect Surface. For example, enterprise information that is public-facing would have a lower risk rating than personally identifiable information (PII), which demands stronger safeguards. In the context of LLMs, protect surfaces may include components such as vector databases that store, index, and query high-dimensional embeddings, API endpoints that serve as access points for client applications, and prompt interfaces that allow users to interact with the model.

A Zero Trust strategy centers on the principle of assuming that a breach has occurred or will occur. This assumption encompasses a range of scenarios, including the potential exfiltration of data and unauthorized access, even if such exfiltration has not yet occurred. This also extends to unauthorized alteration, deletion, and encryption of data.

Breaches occur when unauthorized entities, including both human and non-human, gain access to sensitive information. This risk can be mitigated with various security controls.

Security Controls

Access Control, Least Privilege, and Authentication

Any access to information should be granted to a named identity, rather than allowing anonymous access. Access control, when combined with the principle of least privilege, reduces the attack surface that a malicious actor may have. For example, a lack of database access for a compromised identity would limit the malicious actor’s ability to access the database containing critical enterprise information. Any suspected compromises of an identity or detected lateral movement should be re-authenticated before granting access to the database. The re-authentication should include MFA. In fact, such compromises can be deterred, detected, and the probability of a breach can be reduced with CBAC. This can help prevent the materialization of prompt injections that may attempt to exploit the lack of least privilege. Access controls, coupled with the principle of least privilege, prevent malware attacks and unauthorized outbound internet connectivity. If there is no need to enable outbound access, don’t do so.

Network Layer

Network layer controls, such as reverse proxies or forward proxies, can be employed to filter return data before it reaches the user or the application interface.

Zero Trust at the network layer could:

• Remove inbound exposure by enforcing private egress only
• Bind connectivity to verified identities (human or non-human) rather than IPs or ports
• Apply policy-as-code for every flow (who/what/where/when)
• Instrument per-flow telemetry to provide audit and anomaly detection for model and data access

Together, these controls create a software-defined micro-perimeter for each agent, model API, or dataset—far more adaptive than static network access control lists (ACLs).

Micro-Segmentation (Full Stack)

Micro-segmentation is a preventive control that reduces the attack surface by restricting access to the compromised stack, which includes:

• Network Segmentation: Partitioning a network into smaller sections to minimize the risk of lateral movement

• Identity Segmentation: Also known as credential partitioning or purpose-based identities; for example:

  • A dedicated identity for desktop administration
  • A separate identity for Google Cloud Platform (GCP) administration
  • Additional identities for other administrative domains

By applying segmentation across the full stack, a compromise of one identity (e.g., desktop administration) does not impact other environments (e.g., GCP), thereby reducing the attack surface.

Comprehensive Visibility at Application Layer

While least privilege and micro-segmentation limit lateral movement, risks remain if models are maliciously acquired, insiders act with ill intent, or an attacker gains a valid token. Continuous monitoring is essential to detect threats such as data exfiltration in real time or near real time.

Key practices include:

• Monitoring for poisoned libraries
• Scanning third-party code or models with Software Composition Analysis (SCA) and Static Application Security Testing (SAST) tools to assess dependencies and vulnerabilities in the source code

• Strengthening security of application code and pipelines by following a Secure Software Development Lifecycle (S-SDLC), including:

  • Vetting libraries from a security perspective
  • Applying secure coding practices during algorithm and model development

Comprehensive Visibility in the Supply Chain

Supply chain security is critical when managed services are procured to manage the environment where models are deployed or to provide SaaS. Enterprises can reduce these risks by implementing:

• Contractual Obligations: Include audit clauses and security requirements to enforce safeguards where the enterprise security team has limited oversight

• Data Protection Measures: Ensure any requirements for anonymization or pseudonymization are explicitly defined in contracts to protect sensitive enterprise information

Comprehensive Visibility with Analytics

Data loss prevention (DLP) tools may apply this principle if they can detect and block key threat vectors, including:

• Data Exfiltration from Within Cloud Environments: Monitor for data leaving private or public clouds (e.g., Azure, AWS, GCP)

• Unauthorized Uploads to Third-Party Services: Detect and block attempts to move data to file-sharing platforms or unmanaged SaaS collaboration tools

However, threats like data exfiltration to local devices, which materialise with risks such as clipboard access on remote desktops or transfers to unmanaged/insecure endpoints used by end-users and employees, may not be addressed with DLP tools. Also, using DLP tools comes with a requirement for the classification of data that is being monitored. If the tool is not configured correctly, it may not serve any useful purpose from a security perspective.

Logging and Monitoring

One of the most essential security controls for achieving comprehensive visibility is logging. Zero Trust guidance recommends recording all security events across every component of the protect surface. These logs should be collected and monitored through a Security Information and Event Management (SIEM) tool to support timely detection and responses.

Governance of Secure Adoption of LLMs

Another critical control to govern the secure adoption of LLMs is the deployment of a Cloud Access Security Broker (CASB). This enables organizations to:

• Discover and Monitor Shadow AI: Detect unsanctioned LLM tools accessed by employees

• Control Access to LLM Platforms: Enforce restrictions based on user identity, role, device, or location

• Prevent Data Leakage: Inspect prompts and responses to ensure sensitive or regulated data is not shared with external LLMs

Output Validation and LLM Gateways

The risk of threat materialization for the output of LLM models can be mitigated by applying the Zero Trust principle of “never trust, always verify,” which is implemented through human validation of the output.

For programmatic interfaces with APIs and agentic AI, outputs should be validated through checks, including:

• Boundary conditions
• Schema validation
• Type validation
• Input validation to block malicious characters
• Authorization bypass prevention
• Server-side request forgery (SSRF) prevention using controls like micro-segmentation

Since enterprise data is predominantly accessed through organization-provided devices, network-level controls such as proxies can inspect and filter data entering or leaving the environment to enforce organizational policies. For example, detecting and blocking sensitive or prohibited queries.

Threat Vectors and Real-Life Incidents

To illustrate how these threat categories materialize in practice, the following table maps real-life incidents in LLM environments to the corresponding threat categories, highlighting their practical impact:

Table 2: Threats Mapped to Impacted Layer and Addressed with Zero Trust Principles

Zero Trust IAM Guidance for AI

Because breaches often stem from implicit trust, governing identity and access management (IAM) effectively is critical. The CSA provides Zero Trust guidance for IAM in its publication Zero Trust Principles and Guidance for Identity and Access.

Managing and Authenticating Human User Identities While Using AI Products

Identity governance centers on the secure creation of identities, verifying them to prevent spoofing attempts, and the ongoing management of the identity lifecycle. This guidance is suitable for all identities, human and non-human, across hosted environments, non-hosted environments, and SaaS applications offering AI solutions. As with any other environment, identity governance is essential in LLM environments.

The governance program must manage the lifecycle of identities, including but not limited to:

• Creation of identities
• Adding and maintaining attributes
• Adding and managing permissions
• Managing the movers-and-leavers process
• Conducting regular access control review
• Implementing and managing single sign-on (SSO) with MFA, preferably phishing-resistant MFA where possible

Managing and Authenticating Agentic AI and Non-Human Identities

It is not always possible to intercept authentication flows for agentic AI and other non-human identities to enforce MFA¹. However, any compromise of the authentication token can have an impact similar to that of human identity compromises, including data breaches and unauthorized access to sensitive information.

While intercepting service calls is challenging, credential compromise is not. Keys, certificates, username/password combinations, secrets, and other sensitive information can be exploited.

In these scenarios, it is imperative to secure the credentials against compromise through strong access controls and encryption, and, where feasible, to secure them out-of-band from their point of usage.

Access Control and Authorization for AI Agents

Identities are required to run services in any environment, including LLMs. Because AI agents require access to data and other services, access control and authorization must ensure that only authorized agents have access to data. To prevent implicit trust and to address risks such as stolen tokens, CBAC should be implemented. This approach evaluates factors like user identity, device, and location before granting access. In addition, CBAC should govern scenarios where human identities interact with agentic identities, ensuring that implicit trust is not assumed when both operate in the same environment.

LLM environments have several roles, including IaaS administrators and architects, data administrators, data scientists, data architects, LLM model owners, DevOps engineers, application developers, and data integration architects. Each role requires properly scoped access aligned with the principle of least privilege.

More information about access management can be obtained from the following CSA publications:

• Agentic AI Identity and Access Management: A New Approach | CSA
• Secure Agentic System Design - A Trait-Based Approach | CSA

The Human Element

There are scenarios where implementing security controls is particularly difficult. For example, employees using enterprise data with AI services like ChatGPT can enter sensitive information into prompts. This risk can be mitigated with education and training, ensuring staff understand the consequences of exposing enterprise data. Technical measures, such as CASB, Secure Web Gateways (SWG), or simple proxies, may also be used to block access to SaaS-based LLM services. However, these controls have limitations, as services like ChatGPT can be accessed from personal devices outside the enterprise network.

Ultimately, education remains the most effective safeguard, reinforcing employee awareness and responsible data use.

More information about the human element can be obtained from the following CSA publication:

• Addressing Employee Resistance to AI Adoption | CSA

Conclusion

LLMs present significant opportunities for enterprises, offering tangible benefits in terms of efficiency, innovation, and business value. Yet, their adoption must be carefully considered, as the risks outlined in this paper can significantly affect security, compliance, and trust. To address these challenges, organizations should adopt a Zero Trust model from the beginning, as outlined throughout this document. This approach ensures that LLM integration is not only effective but also secure, resilient, and aligned with enterprise governance requirements.

Future Outlook

Looking ahead, many trends will shape the evolution of LLM environments and their role in enterprises:

• Greater Efficiency with Smaller Models: Optimized architectures will deliver high performance while reducing resource requirements

• Faster Deployment and Improved Compliance: Streamlined adoption will enhance return on AI investments while aligning with regulatory expectations

• Shift from Automation to Augmentation: The future is man-plus-machine, where AI enhances rather than replaces human capabilities

• Emergence of AI Agents: LLMs will move beyond content generation to become agents capable of performing tasks, making decisions, and automating a wide range of digital processes

• AI as a Force Multiplier: Systems will become more versatile, efficient, personalized, and increasingly autonomous

• AI and Zero Trust Convergence: Security will remain central, with AI itself enhancing Zero Trust architectures by improving anomaly detection, adaptive access controls, and data protection

AI will expand in capability and impact, and its secure adoption, particularly through Zero Trust principles, will remain essential for protecting enterprise data.

Useful References

Cloud Security Alliance (CSA). (2025, January 15). Context-Based Access Control for Zero Trust. Context-Based Access Control for Zero Trust | CSA

National Security Telecommunications Advisory Committee (NSTAC). (2022).
Final Draft NSTAC Report to the President on Zero Trust and Trusted Identity Management

Cybersecurity and Infrastructure Security Agency (CISA). (2025). Best Practices for Securing Data Used to Train and Operate AI Systems. Joint Cybersecurity Information AI Data Security

Regulatory References

Artificial Intelligence Act (EU). (2024, February 7). High-level Summary of the AI Act. Future-of-Life-InstituteAI-Act-overview-30-May-2024.pdf

Latham & Watkins LLP. (2025, September 15). EU Data Act — What Businesses Need to Know. EU Data Act — What Businesses Need to Know

U.S. Senate. (2023, March 16). Senators Coons, Cornyn, Tillis, Whitehouse introduce legislation to ensure copyright protection and public access to safety standards (PRO Codes Act). Senators Coons, Cornyn, Tillis, Whitehouse introduce legislation to ensure copyright protection and public access to safety standards

Transparency Coalition. (2025, August 5). Senators unveil the TRAIN Act, bipartisan bill to protect creators from unauthorized AI training — Legislation for Transparency in AI Now. Senators unveil the TRAIN Act, bipartisan bill to protect creators from unauthorized AI training — Transparency Coalition. Legislation for Transparency in AI Now

IP IQ. (2024, May 13). The Generative AI Copyright Disclosure Act of 2024: Balancing Innovation and IP Rights. The Generative AI Copyright Disclosure Act of 2024: Balancing Innovation and IP Rights | IP IQ

U.S. House of Representatives. (2024, January 10). Salazar introduces No AI FRAUD Act to protect Americans from deceptive AI impersonations. Salazar Introduces the No AI Fraud Act

AI Security References

Cloud Security Alliance (CSA). (2025, July 30). Secure Agentic System Design – A Trait-Based Approach. Secure Agentic System Design - A Trait-Based Approach | CSA

Adversa AI. (2025, March 31). NIST AI 100-2 E2025: Adversarial Machine Learning — A Taxonomy and Terminology of Attacks and Mitigations. NIST AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations | Adversa AI

ScienceDirect. (n.d.). Training Data – An Overview. Training Data - an overview | ScienceDirect Topics

National Institute of Standards and Technology (NIST). (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST.AI.600-1). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

Cloud Security Alliance (CSA). (2024, August 13). Securing LLM-Backed Systems: Essential Authorization Practices. Securing LLM Backed Systems: Essential Authorization Practices

Cloud Security Alliance (CSA). (2025, May 28). Agentic AI Red Teaming Guide. Agentic AI Red Teaming Guide | CSA

Cloud Security Alliance (CSA). (2024, September 25). AI in Medical Research: Applications and Considerations. AI in Medical Research: Applications & Considerations

Cloud Security Alliance (CSA). (2025, January 28). AI Organizational Responsibilities: AI Tools and Applications. Organizational Responsibilities for AI Applications | CSA

Cloud Security Alliance (CSA). (2024, October 21). AI Organizational Responsibilities – Governance, Risk Management, Compliance, and Cultural Aspects. AI Organizational Responsibilities - Governance, Risk Management, Compliance, and Cultural Aspects

Cloud Security Alliance (CSA). (2024, November 11). AI Risk Management: Thinking Beyond Regulatory Boundaries. AI Risk Management: Thinking Beyond Regulations | CSA

Cloud Security Alliance (CSA). (2024, August 6). Using AI for Offensive Security. Using AI for Offensive Security | CSA

Cloud Security Alliance (CSA). (2024, July 23). AI Model Risk Management Framework. AI Model Risk Management Framework | CSA

Cloud Security Alliance (CSA). (2024, May 5). AI Organizational Responsibilities – Core Security Responsibilities. AI Core Security Responsibilities | CSA

Cloud Security Alliance (CSA). (2024, May 5). AI Resilience: A Revolutionary Benchmarking Model for AI Safety. AI Resilience: Benchmarking AI Governance & Compliance | CSA

Cloud Security Alliance (CSA). (2024, May 5). Principles to Practice: Responsible AI in a Dynamic Regulatory Environment. Responsible AI in a Dynamic Regulatory Environment | CSA

OWASP. (2025, January 5). LLM and Generative AI Security Solutions Landscape – Q1, 2025. LLM and Generative AI Security Solutions Landscape - Q1,2025

Cloud Security Alliance - Taxonomy for Large Language Models
LLM Threats Taxonomy | AI Terms Glossary | CSA

SaaS Application Security References

Cloud Security Alliance (CSA). (2024, September 12). SaaS Challenges, Solutions, and Best Practices for 2024. SaaS Challenges, Solutions, and Best Practices for 2024 | CSA

Cloud Security Alliance (CSA). (2024, December 18). Cloud Security for Startups 2024. Cloud Security for SaaS Startups | CSA

Blended ZT and AI References

BankInfoSecurity. (2025, May 9). Bringing Zero Trust Into the AI Era. Bringing Zero Trust Into the AI Era - BankInfoSecurity

Amazon. (2025, June 10.). Rise of the Machines: A Project Zero Trust Story. Rise of the Machines: A Project Zero Trust Story

Cloud Security Alliance (CSA). (2023, August 24). Zero Trust and AI: Better Together. Zero Trust and AI: Better Together | CSA

Cloud Security Alliance (CSA). (2024, May 6). Shadow Access: IAM Considerations for Zero Trust and AI Deployments. Shadow Access: IAM Considerations for Zero Trust & AI | CSA

Cloud Security Alliance (CSA). (2025, February 27). How is AI Strengthening Zero Trust? How is AI Strengthening Zero Trust? | CSA

Cloud Security Alliance (CSA). (2025, September 15). Analyzing Log Data with AI Models to Meet Zero Trust Principles. Analyzing Log Data with AI Models to Meet Zero Trust Principles | CSA

Cloud Security Alliance (CSA). (2025, December 3). Data Security within AI Environments. Data Security within AI Environments | CSA

Glossary

Poisoned Models: Model poisoning attacks are attacks in which an adversary attempts to directly modify the trained ML model by injecting malicious functionality into it.

Training Data: A collection of training samples used to develop and optimize a neural network model.

Confabulation/Hallucination: Refers to a phenomenon in which GenAI systems generate and confidently present erroneous or false content in response to prompts. Confabulations also include generated outputs that diverge from the prompts or other input or that contradict previously generated statements in the same context.

Need-to-Know Information Leakage: Many organizations are legally bound by non-disclosure and confidentiality agreements. These agreements can include need-to-know clauses whereby confidential information may only be shared with a limited set of people within an organization. A second example is where specific parts of an organization develop business plans or intellectual property. This type of information is of high value to the organization while it remains confidential. Need-to-know information can also be regulated by federal laws (e.g., International Traffic in Arms Regulations (ITAR)). Access to this information has traditionally been managed by, say, Document Management Systems. These systems have strict access controls limiting to named users or business groups. A new threat has emerged, whereby a well-meaning employee who has access to need-to-know information adds that information to an AI model in a SaaS . If the model has no awareness that the information is provided on a need-to-know basis, that information can be leaked to other users who have access to the model. Without controls in place, the organization will have breached its need-to-know responsibilities.