Untrustworthy Output in Professional Domains
專業領域輸出不可信
Operation & Monitoring
Risk Description
When under time pressure, a practitioner adopts the system's recommended course of action, which misclassifies a rare situation as a common one; due to unmitigated control gaps, because the recommendation is professionally phrased and definitive, it is executed without question, producing irreversible consequences, triggering compliance exposure and operational reputational costs.
Framework Mappings
NIST AI 600-1Confabulation、Human-AI Configuration
OWASP Top 10 for LLMLLM09
NIST AI RMFMANAGE 4.1
ISO/IEC 5338運作與監控
MIT AI Risk RepositoryDomain 5
Risk Treatment & Implementation Guidance
Mandate human review for professional recommendations affecting personal safety, with system output as reference only; Clearly mark the applicable scope and rare-case limitations, prompting escalation to professional judgment beyond scope; Set minimum verification requirements for time-pressured settings so checks are not skipped for speed