IA-06 (Safety and Security )

3 sub-dimensions · 78 risk scenarios

Ensures that the AI system, under defined conditions, does not cause harm or physical threat to human life, health, well-being, property, or the environment, while guarding against malicious threats such as adversarial attacks, data poisoning, and model theft.

🎯 BenefitsSafeguards physical and psychological safety in automated environments while constructing high-strength cybersecurity defenses against adversarial attacks and data poisoning to protect life and property.

⚠️ Harms / RisksSystem malfunctions or prediction errors directly cause physical injury and property damage, or attacks such as prompt injection and backdoor poisoning breach safety defenses.

Operational Safety12 risk scenarios · ISO 23894 A.10

DefinitionEvaluates whether the AI system, under specified conditions, avoids endangering human life, health, well-being, property, or the environment.

🎯 BenefitsSafeguards the security of physical and digital automated environments, guarding against unauthorized autonomous agent actions, resource exhaustion, cascading collapse, and physical harm.

⚠️ Harms / RisksA runaway automated trading system triggers a market flash crash, or AI servers under extreme request attacks suffer inference resource exhaustion (DoS) and astronomical cloud bills.

Representative ScenariosS0132 Inference Resource Exhaustion Attack S0133 Hijacking of Computing Resources S0134 Crafted Inputs Amplifying Computational Cost S0137 Cost Amplification Attack S0140 Autonomous System Accessing Files Beyond Authorization S0141 Unauthorized Outbound Connections by an Autonomous System S0178 Automated Cyber Attack S0182 Insufficient Security in Inter-System Protocols S0183 Tacit Coordination Among Autonomous Systems S0184 Escalation Through System Interaction in Critical Domains S0186 Multiple Systems Converging on Collective Suboptimum S0189 Tacit Coordination Through Shared Pricing Systems

System Security Vulnerabilities60 risk scenarios · ISO 23894 A.10

DefinitionEvaluates the risk that system security vulnerabilities could allow malicious actors to gain control.

🎯 BenefitsBuilds hardened security defenses against prompt injection, adversarial perturbation attacks, data poisoning, and supply-chain vulnerabilities.

⚠️ Harms / RisksAttackers use multi-turn dialogue or images to perform prompt injection, bypassing safety guardrails to hijack agents into unauthorized actions or exfiltrate confidential internal files.

Representative ScenariosS0036 Label Tampering Attack S0037 Backdoor Trigger Sample Implantation S0039 Poisoning of the Retrieval Knowledge Base S0040 Safety Alignment Broken During Fine-Tuning S0041 Gradient Poisoning in Distributed Training S0043 Upstream Contamination of Automated Data Pipelines S0045 Malicious Packages Entering the Development Environment S0046 Backdoored Models on Public Platforms S0047 Vulnerabilities in the Training Framework Itself S0048 Blind Spots from Missing Component Inventory S0049 Exploitation of Deep Learning Framework Vulnerabilities S0050 Supply Chain Vulnerabilities in Preprocessing Tools S0051 Insecure Serialization Format S0070 Adversarial Perturbation of Images S0071 Adversarial Audio and Hidden Commands S0072 Adversarial Text Evading Detection S0073 Physical-World Adversarial Attack S0074 Cross-Modal Misdirection S0075 Universal Adversarial Perturbation S0076 Cross-Model Transfer Attack S0077 Coordinated Multimodal Perturbation S0078 Functional Replication Attack S0088 Unencrypted Model Weights Leaked S0093 Inference Interface Without Access Control S0094 Unpatched Container Image Vulnerabilities S0095 Overly Permissive Cloud Service Configuration S0096 Inference Endpoint as an Internal Pivot S0097 Multi-Tenant Isolation Failure S0118 Direct Prompt Injection Bypassing Safeguards S0119 Indirect Prompt Injection Hijacking the System S0120 System Instruction Disclosure S0121 Multi-Step Inducement Attack Chain S0122 Tool Invocation Manipulated by Injection S0123 Covert Goal Hijacking S0124 Multimodal Hidden Instruction Injection S0125 Cumulative Context Poisoning S0126 Retrieval Injection Hijacking Output S0127 Capability Theft Through Query Distillation S0128 Side-Channel Inference of Model Information S0129 Side-Channel Risk on Shared Computing Platforms S0130 Residual Data in Shared Hardware Memory S0150 Generation of Hate Content Targeting Groups S0151 Generation of Virtual Content Harmful to Minors S0153 Non-Consensual Synthetic Intimate Imagery S0154 Automated Generation of Harassment Content S0155 Production of Extremist Content S0156 Operational Guidance for Illegal Activity S0157 Inappropriate Content and Guidance for Minors S0158 Large-Scale Contamination by False Information S0168 Synthetic Identity Fraud S0169 Mass Production of False Content S0170 Highly Customized Phishing Attack S0171 Lowered Barriers to Dangerous Knowledge S0174 Misuse of Scientific Reasoning Capability S0175 Precision Social Engineering S0176 Manipulation of Elections and Political Processes S0177 Distribution of Non-Consensual Synthetic Intimate Imagery

Safety Impact of System Changes6 risk scenarios · ISO 23894 A.10

DefinitionEvaluates the safety impact on interested parties when the AI system is changed.

🎯 BenefitsStrengthens the security of CI/CD deployment pipelines and update channels, guarding against model-version substitution, leakage of unencrypted weights, and in-transit tampering.

⚠️ Harms / RisksThe tail end of the deployment pipeline is replaced by attackers with a malicious model, or model weights are tampered with and watermarks stripped during transmission and update pushes.

Representative ScenariosS0053 Replacement of Fine-Tuning Layer Files S0055 Systemic Risk from Model Homogeneity S0089 Removal of Model Markers S0090 Weight Tampering During Transmission S0091 Model Substitution at Deployment S0092 Hijacking of the Update Channel