Introduction

The practice of DevSecOps has evolved significantly since the start of the CSA DevSecOps Working Group in 2019. We have taken the time to formulate industry guidance as security practices and technology matured in our Six Pillars of DevSecOps. The working group principles—intended to provide guidance for new and emerging industry challenges—will now focus on how we apply lessons learned in DevSecOps to generative-AI-developed services and machine learning operations.

We have established a plan to release the following four whitepapers in 2025:

• MLOps Overview: A high-level view of the MLOps landscape and threat vectors
• DevSecOps Practices in MLOps: A deep manifesto that pulls DevSecOps practices that have high relevance to MLOps
• Threat Model of an MLOps Solution: Implementing a key principle of DevSecOps to threat model an MLOps high-level design
• MLSecOps Reference Architecture: A security reference architecture that provides end-to-end guidance on securing products that follow MLOps practices

Goals

The Cloud Security Alliance’s DevSecOps working group is outlining the practice of MLOps. This builds the foundation for future papers that explore MLSecOps and justifies why this is an area of high priority.

Audience

The target audience of this document includes those involved in the management and operational functions of information security and information technology. This includes decision makers, architects, and builders.

Executive Summary

As machine learning becomes increasingly central to business operations, securing the ML pipeline is no longer optional but essential. This whitepaper introduces MLSecOps—the intersection of machine learning operations and security—and outlines the unique challenges organizations face when protecting ML assets. Our analysis reveals that traditional security approaches are insufficient for ML systems due to novel threats such as data poisoning, model inversion, and adversarial attacks. By implementing a comprehensive MLSecOps framework, organizations can protect their ML investments while maintaining regulatory compliance. The Cloud Security Alliance will release a series of whitepapers throughout 2025 to provide practical guidance on securing ML operations across industries.

What is MLOps?

For many, Machine Learning Operations (MLOps) may sound related to DevOps and DevSecOps, albeit they follow the same principles but with different focus areas—DevOps focuses on software development and MLOps focuses on the lifecycle of the machine learning model. MLOps is primarily a set of repeatable processes to build, deploy, and continuously monitor machine learning models. Incorporating operations to machine learning allows for development and testing in a reliable, incremental, and repeatable way.[^1]

In MLOps, there are three main areas of focus: the data, the software, and the model.

• Data includes information used to train a machine learning model. The data used to train models may have different levels of sensitivity, and the data used for training can impact the results and the quality of the machine learning model itself[^2].
• The software or code includes the deployment pipelines, the Continuous Delivery for Machine Learning (CD4ML) of the ML model to a production environment, any monitoring that needs to occur to observe model performance, predictions, deviation or signals, machine learning applications and agents consuming the ML model, and software logging of model performance.
• The model itself is the core of MLOps and a key differentiator between traditional DevOps and MLOps. The model includes the writing and execution of machine learning algorithms to create an ML model. (from https://arxiv.org/abs/2205.02302)

Figure 1: MLOps Areas of Focus
Machine Learning Security Operations (MLSecOps) extends MLOps by embedding security principles throughout the machine learning lifecycle, from data ingestion to model deployment and monitoring. When focusing on MLSecOps, security practices should be top of mind when considering the machine learning model development and operations.

Taking into consideration the three main areas within MLOps, the additional security perspective for MLSecOps would entail protecting the confidentiality, integrity, availability, and traceability of the data, software, and model. These three areas are at risk from several threats specific to machine learning, such as:

• Data poisoning (deliberately corrupting training data to manipulate model outcomes)
• Data pollution (unintentional introduction of corrupt or biased data into training sets)
• Label flipping (altering the values of the model’s training labels)
• Extraction attacks (querying the model to reverse engineer the parameters)
• Inversion attacks (reconstructing the data that was used to train the model)
• Model transferring (using the model for a different purpose than the one it was intended for)
• Member inference attack (privacy attack against machine learning models where an adversary attempts to determine whether a specific data sample was part of the training dataset used to build the model)
• Unauthorized model modification (security attack where an adversary tampers with a machine learning model without proper authorization)
• Data leakage (confidentiality breaches by exposing private or proprietary data)
• Backdoor attacks (embedding hidden triggers into a model during the training or inference phase)
• Insufficient logging and monitoring (undetected malicious activity or errors)
• Weighting attack (altering the weight of values within the model to cause misclassification)

LLMOps vs. AgentOps vs. Traditional MLOps Convergence

Over the last few years, the traditional MLOps concept evolved in the industry as a set of practices and workflows to operationalize classical machine learning models. With the recent evolution of Large Language Models (LLMs) and LLM-powered AI agents, the industry has coined new terms like LLMOps and AgentOps to describe a set of practices and workflows to address challenges that are unique to operationalizing these technologies.

LLMs introduce significantly higher computational complexity compared to traditional machine learning models, requiring specialized infrastructure and optimization techniques. LLMOps has evolved to train, deploy, and manage these LLM models in production. The table below shows the differences between classical MLOps and LLMOps.

Table 1: MLOps and LLMOps Differences

AI agents have been around since the dawn of AI. With the recent evolution and breakthroughs in LLMs, AI agents have evolved to use natural language for reasoning, planning, and executing complex tasks in a multi-agentic setting. These AI agents are powered by LLMs and other LLM-based techniques like Retrieval-Augmented Generation (RAG) that can process and generate natural language. The industry has coined the term “AgentOps,” which is a set of practices and workflows to operationalize, deploy, and manage these LLM-powered AI agents in production. The table below shows the differences between classical MLOps and AgentOps.

Table 2: MLOps and AgentOps Differences

Even though the industry differentiates between MLOps, LLMOps, and AgentOps, we believe these terms should converge and the term “MLOps” should encompass all operational practices, workflows, and technology architectures related to anything machine learning. By definition, the universe of machine learning is the superset that encompasses all machine learning techniques, including learning techniques and modeling architectures like transformers that LLMs presently use, other generative AI learning techniques like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTMs) networks, and more classical machine learning approaches like Support Vector Machines (SVMs), regression models, Artificial Neural Networks (ANNs), and so on. Therefore, in this and the following whitepapers, our use of the term MLOps differs from industry standard in that it encapsulates LLMOps and AgentOps.

MLOps Stakeholders

MLOps tends to be slightly less agile than traditional DevOps, as a significant portion of time is spent on the machine learning model research, data collection, training, and development. Adopting MLOps practices allows the introduction of automation into the machine learning model lifecycle. There are several key players involved in the MLOps pipeline, all of whom have a different role to play in building the relationship between machine learning and operations that creates MLOps.

Note: Cybersecurity roles have not been included in the stakeholders below. That will be the topic of a future paper, MLSecOps Reference Architecture.

Data Scientists

Data scientists often manage the model engineering pipeline, including the machine learning algorithms and data used to create and train a machine learning model. They perform roles such as model creation, training, evaluation, model testing, and model packaging. Data scientists are often not experienced software engineers who build production-ready software services.

Machine Learning Engineers

Machine learning engineers are most often handed over a model from the data science team to turn into a callable artifact (e.g., API integration). The training of machine learning models at scale, storage, upload of the model into a code repository or model registry, deploying the model and making it available for use, and building scalable inference pipelines and applications to consume the machine learning model is often the duty of a machine learning engineer. [^3]

Operations Teams

Operations teams manage the continuous integration and continuous delivery pipelines. In ML deployment, pipelines are more complex than traditional software CI/CD flows (e.g., Dev → QA → Prod), as they must accommodate multiple workflows for data, models, and training artifacts. There tends to be a pipeline for data and another pipeline for models, as models tend not to be updated, but entire models are released in iteration. More details will be provided in the MLOps pipeline stages section.

Platform Teams

Platform teams are responsible for managing infrastructure, automation, scalability, and system monitoring. In the context of machine learning, these teams play a crucial role in designing and maintaining frameworks for designing scalable, reliable ways to quickly deploy ML models.

Security Team/Security Engineers

Security teams play a critical role in the MLOps lifecycle by ensuring that security controls are integrated at every stage, ideally by design. They are responsible for conducting the threat modeling of ML systems, identifying potential vulnerabilities in both data and models, implementing security guardrails, monitoring for security incidents, and ensuring compliance with security policies and standards. Security engineers, on the other end, will work closely with data scientists and ML engineers to design secure model architectures, implement proper access controls, protect sensitive training data, prevent model tampering, and defend against ML-specific attacks like adversarial examples and data poisoning. Their expertise is essential for conducting security assessments of ML systems and developing incident response procedures specific to ML security incidents.

Privacy Team

In the MLOps lifecycle, the privacy team is responsible for ensuring that the data used in model development and training adheres to applicable privacy regulations as well as organizational standards. They are responsible for enforcing data minimization principles, identifying Personally Identifiable Information (PII), and implementing techniques like data anonymization, pseudonymization, and differential privacy to protect user data. The privacy team ensures that privacy considerations are embedded throughout the MLOps lifecycle, supporting both ethical AI practices and regulatory compliance.

Data and Compliance Teams

Data and compliance teams are often responsible for managing the data lifecycle, ensuring that ML
models are trained on high-quality, relevant, and unbiased data. This includes the introduction or collection of new data from various sources, data management, analysis and data security. Data security is a prime consideration in the MLOps pipeline, as compliance teams will need to implement measures to protect and govern sensitive information.

C Level Positions/Leadership

Leadership and C-level teams are interested in introducing ML capabilities to optimize internal processes and business offerings to customers. Leadership is often interested in the time reduction that would come from MLOps, ensuring models are reliable, up to date, meet cost considerations and uptime requirements, and move the needle significantly on business key performance indicators (KPIs) and metrics.

Project/Product Teams

Product teams are essential to guiding the vision for ML products and are often responsible for ML use case prioritization. Project teams are critical to driving the delivery of product vision to meet target dates.

Customers

Customers are interested in how their data is leveraged with regard to the MLOps pipeline. Customers are interested in whether their data is used for model training purposes, that their sensitive data is well-protected and is not leaked to other tenants in a multi-tenant environment, and that their data is being properly cleaned up and machine learning models won’t retain their sensitive data after they have off-boarded. Customers are also interested in the reliability and explainability of the machine learning models.

The Stages of MLOps

This section introduces the stages of the MLOps pipeline and overlays key players in MLOps with relation to the overall stages. We have provided a view of MLOps stages and also referenced industry insight below:

• Google MLOps ([What is MLOps?

  Google Cloud](https://cloud.google.com/discover/what-is-mlops))

• Amazon MLOps ([What is MLOps?

  AWS](https://aws.amazon.com/what-is/mlops/))

• Microsoft MLOps ([What is MLOps?

  Azure](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2))

• Oracle MLOps (MLOps simplified: Deploying MLflow on OCI)

Design Stage

In the design stage, the focus is on gathering requirements for the machine learning project from key stakeholders and making sure there is alignment on the scope of the ML system. Use cases are prioritized, architectural blueprints are fleshed out for the ML system, cost benefit analysis is performed by the business, and the project is greenlighted if it aligns with the business strategy.

Model Development Stage

The model development stage is critical to the creation of the model. Model development includes the following activities:

• Data Preparation/Sourcing Data for Model: Relevant data is identified, testing for data quality is done, statistical analysis on data is done to understand potential data features of interest for machine learning models, data labeling is performed, and feature learning code is developed.

• Model Experimentation and Testing: In this stage, which is also often called “model selection,” various machine learning algorithms and model architectures are evaluated to identify the approach that delivers the best performance for the given task.

Operations Stage

In the operations stage, the model is trained on production datasets, and it is deployed to production if it meets the metrics for model-quality criteria. The production deployment of the model is continuously monitored to ensure the model remains reliable and accurate over time. Manual or automatic retraining is triggered if there is a performance degradation or data drift in model quality. Operations comprises of the following activities:

• Model Training: Features are extracted from production datasets and data is labeled, model is trained and validated on the labeled datasets, and the trained model is versioned and pushed through the model approval workflow.

• Deployment: The model is deployed to production if it meets the model-quality criteria. The old version of the model is replaced with the latest version of the model. The model most often sits behind a REST API that can be invoked by downstream streaming and batch applications.

• Logging: Logging is essential for machine learning models as it captures detailed records of model inputs, predictions, and decision paths, helping track model behavior over time. These logs are invaluable for auditing, debugging, detecting bias or drift, and ensuring the model operates as intended, especially in regulated environments.

• Monitoring: Monitoring is an important aspect of operations, as it allows the identification and remediation of issues, ideally before they become issues for end-users. Monitoring also permits a continuous validation that the ML model is operating as expected by checking various metrics like data drift, model concept drift, response quality, model bias, and so on.

Continuous Feedback Stage

There are two important MLOps stages, different from the above, as they are not essential to the MLOps pipeline, though they often occur as part of the natural process of MLOps.

• Consuming Results of the Model: Customers are an integral part of consumption of the data from the ML model, depending on the use case. The output from an ML model may be made available to customers in various ways.

• New Model Update: New models may go through all of the stages from Design > Development > Operations, or they may leverage some existing datasets and iterate through testing and deployment.

The following table summarizes the key stages and activities in the MLOps workflow, as discussed above, and the key players involved in each of the stages:

Table 3: Key Players and Activities for MLOps Stages

The Challenges of MLOps

If you have already started with MLOps, there may be some challenges keeping you awake at night. These are outlined in the table below.

Conclusion

Machine learning was introduced in 1959 and the term MLOps was coined in 2015, before the introduction of generative-AI, LLM-based AI agents and continuous AI innovations. Our view is that MLOps is all encompassing and provides us with a viewpoint to dive deeper into domains that cover the advancement in AI that we see today.

This paper sets the foundation for how we can apply DevSecOps to MLOps—without actually providing any recommendations, we explain its importance, relevance, and concerns. Throughout 2025, the CSA DevSecOps Working Group will create and publish guidance on securing MLOps. We will achieve this by:

• Threat modeling machine learning solution designs
• Reviewing DevSecOps practices that can be carried over to MLOps
• Creating a security reference architecture for MLSecOps

We are excited and passionate about creating new guidance in this space and would like to invite contributors to work with us to achieve MLSecOps consensus.