Harm from Health and Psychological Advice
健康與心理相關建議的傷害
Operation & Monitoring
Risk Description
When a user seeks support from the system during a period of low mood, and the system responds inappropriately; due to unmitigated control gaps, the user's condition does not improve but worsens, and because the conversation is private, no one nearby is aware, triggering compliance exposure and operational reputational costs.
Framework Mappings
EU AI ActArt.50
NIST AI 600-1Human-AI Configuration、Dangerous, Violent, or Hateful Content
NIST AI RMFGOVERN 3.2
ISO/IEC 42005影響評估流程
ISO/IEC 5338運作與監控
MIT AI Risk RepositoryDomain 5
Risk Treatment & Implementation Guidance
Install safety guardrails for health and mental-health interactions, recognizing high-risk signals and responding with care; Establish professional-referral mechanisms proactively offering help resources in sensitive contexts; Continuously monitor response quality to avoid inappropriate content worsening the user's situation