Confidence Miscalibration
信心度校準失準
When an organization designs an automation threshold based on confidence scores, releasing high-confidence cases automatically; due to unmitigated control gaps, the system consistently assigns high confidence to a particular class of errors, so those cases all bypass human review and are discovered only after losses accumulate, triggering compliance exposure and operational reputational costs.
Framework Mappings
Risk Treatment & Implementation Guidance
Validate confidence-score calibration before launch, confirming high confidence corresponds to high accuracy before using it as an automation threshold; Retain sampled human review of high-confidence auto-approved cases to detect systematic overconfident errors; Apply uncertainty quantification, forcing out-of-distribution inputs to human handling