The Catastrophic Risk Annex: Next Gen AI Security Controls

AI technologies are entering a new phase defined by their growing systemic impact. Organizations today are already managing familiar AI risks: data leakage, bias, model drift, and misuse. But as AI systems become more capable, autonomous, and embedded in critical infrastructure, possible catastrophic risks are emerging.

These are not traditional enterprise incidents. They represent the potential for large-scale, irreversible, and society-wide consequences. Think loss of human oversight and uncontrolled system behavior.

The CSAI Foundation aims to move beyond abstract discussions of these catastrophic AI risks and toward something far more concrete: Assurance. Controls that professionals can test, validate, and independently audit.

 

Introducing the Catastrophic Risk Annex

To address this challenge, CSAI is launching the STAR for AI Catastrophic Risk Annex. This project will translate concerns about catastrophic AI risk into practical, measurable safeguards.

The Catastrophic Risk Annex builds on the AI Controls Matrix (AICM) and the broader STAR for AI program. It extends them to address the most extreme risk scenarios. Specifically, the Catastrophic Risk Annex will:

• Identify which existing AICM controls are most relevant to catastrophic risk
• Introduce new controls where critical gaps exist
• Define evidence requirements and testing criteria suitable for independent assessment

The result is a system where organizations can demonstrate control over advanced AI risks.

 

Built for a Multi-Stakeholder Future

Catastrophic AI risk is a shared challenge across the global ecosystem.

The Catastrophic Risk Annex supports:

• AI developers demonstrating safe system design
• Enterprises building audit-ready AI programs
• Cloud providers embedding safety into infrastructure
• Regulators seeking consistent AI assurance and governance models

Through this project, these stakeholders can align around common controls, shared evidence, and comparable outcomes.

 

Why Catastrophic Risk Requires a New Approach

Traditional risk frameworks don't consider systems that act autonomously, interact dynamically with tools, and operate at scale across cloud and critical infrastructure. As a result, many existing controls are:

• Too abstract to validate
• Too static to capture runtime behavior
• Too narrow to address systemic failure modes

The Catastrophic Risk Annex addresses this gap by focusing on what you can actually test in real environments. Examples include:

• Verifying that human-in-the-loop controls cannot be bypassed
• Testing whether action gating prevents unsafe escalation
• Demonstrating that kill-switches and rollback mechanisms function under pressure
• Validating telemetry and detection capabilities for emergent behavior

 

About the CSAI Foundation

CSAI is a new 501(c)(3) nonprofit dedicated to advancing secure, trustworthy, and transparent AI.

CSAI brings together a global, cross-disciplinary community of:

• Cloud providers and AI developers
• Enterprises adopting AI at scale
• Cybersecurity and risk professionals
• Policymakers, regulators, and researchers

Its mission is to advance AI security and assurance through:

• Research and open standards
• Practical frameworks and controls
• Industry collaboration and consensus-building
• Education and guidance for real-world implementation

Unlike purely academic or policy-driven efforts, CSAI focuses on operationalizing trust, turning complex AI risks into actionable controls and measurable outcomes.

A core principle of the CSAI mission is that AI tools must earn trust through verifiable mechanisms. The Catastrophic Risk Annex directly advances this vision by converting high-level AI safety concerns into auditable controls. It enables independent validation of AI system behavior and supports a shared ecosystem of accountability across developers, providers, and users. It reflects CSAI’s broader goal to create a world where organizations can innovate with AI confidently.

 

About STAR for AI

The STAR for AI program extends CSA’s Security, Trust, Assurance, and Risk (STAR) framework into the AI domain. STAR has long served as a global benchmark for cloud assurance, enabling organizations to document their security postures. STAR for AI builds on that legacy by bringing the same principles to AI systems.

At its core, STAR for AI provides:

• A structured framework for AI assurance
• Standardized control sets and assessment models
• Mechanisms for independent validation and transparency
• A public registry where organizations can demonstrate their AI risk posture

This allows organizations to move beyond internal governance and toward externally verifiable trust.

The Catastrophic Risk Annex is a natural extension of this model. It expands STAR for AI to address the highest-impact, lowest-probability risks associated with advanced AI systems.

 

A Roadmap for Measurable AI Safety

The Catastrophic Risk Annex will roll out over a 15–18 month period, beginning in late Q2 2026. The project will follow a structured, multi-phase approach.

 

Phase 1: Turning Risk into Controls

June – September 2026

This phase focuses on translating catastrophic risk scenarios into clear, auditable control language, including:

• Control families such as autonomy limits, tool governance, and containment
• A catalog of high risk scenarios
• Evidence requirements like runtime logs, red-team outputs, and incident drills

 

Phase 2: Making Controls Testable

October – December 2026

This phase focuses on developing validation protocols to ensure users can consistently assess controls, including:

• Resistance to jailbreaks and escalation
• Enforcement of tool restrictions
• Reliability of rollback and containment mechanisms

These will align with global standards such as the NIST AI RMF, EU AI Act, and ISO/IEC 42001.

 

Phase 3: Proving It in Practice

January – June 2027

This phase brings the Catastrophic Risk Annex into real-world environments through:

• Pilot assessments with AI labs, enterprises, and cloud providers
• Training assessors to evaluate agentic systems and runtime behavior
• Development of reusable reference implementations

 

Phase 4: Scaling Assurance Across the Ecosystem

July – December 2027

The Catastrophic Risk Annex becomes a standardized, scalable AI risk management framework with:

• Public STAR for AI registry entries
• Benchmarking and transparency across organizations
• A published State of Catastrophic AI Risk Controls Report

 

From Awareness to Assurance

The conversation around AI risk is evolving. Awareness is no longer enough. Principles are no longer enough. The next phase of AI governance demands something stronger: Proof.

The STAR for AI Catastrophic Risk Annex reflects CSAI’s commitment to making that proof possible. It turns the hardest AI risks into something professionals can measure, test, and trust.

• Learn more about STAR for AI and how your organization can participate in advancing secure, trustworthy AI.
• Learn more about CSAI and our complete slate of integrated programs spanning the full lifecycle of agentic AI security.