Deflection Audit
What an AI support deployment counted as deflected, versus what actually resolved.
Claimed deflection
80.0%
32 of 40 closed without handoff
Audited deflection
55.0%
22 of 40 genuinely resolved
Overstatement
25.0 pts
dashboard reports 1.45× the audited rate
Where the claimed deflections went
22 resolved (69%) · 10 false (31%)
Why the closures failed
| Failure mode | Count | What it means |
|---|---|---|
resident_abandoned | 6 | Generic troubleshooting handed back with no diagnosis, closed without confirmation |
answered_wrong_question | 2 | Accurate response aimed at the wrong problem, or one of several issues left unaddressed |
needed_human_action | 1 | Correct as far as it goes, but resolution needs a truck roll, refund, or replacement |
policy_refusal_loop | 1 | Declined on policy grounds without giving the resident a workable path |
Is the auditor trustworthy?
The deflection numbers above are only as good as the auditor producing them. Scored against 32 hand-labeled conversations, the auditor agreed with ground truth on 25/32 (78%) outcomes. It passed 8/11 confusable-pattern traps -- cases written so that a naive read gets them wrong. Full confusion matrix, over-credit split, and per-trap breakdown are in scorecard.md.
Every closure the audit rejected
| ID | Outcome | Failure mode | Resident said |
|---|---|---|---|
D017 | false_deflection | resident_abandoned | This is the THIRD time this month I cannot get into my own apartment. The app spins forever and the lock does nothing. I am done w... |
D019 | false_deflection | resident_abandoned | Your app is complete garbage. It shows my thermostat at 62 degrees when the actual wall unit clearly says 71. Fix it. |
D023 | false_deflection | needed_human_action | The deadbolt motor sounds like it is straining and sometimes it does not fully retract. It worked this morning after a few tries. |
D025 | false_deflection | answered_wrong_question | Two things: my guest code for the cleaner stopped working, and the lobby door has not been letting my fob work since Tuesday. |
D026 | false_deflection | resident_abandoned | My front door lock is not responding to the app at all. I have been trying for twenty minutes. |
D027 | false_deflection | resident_abandoned | The leak sensor in my laundry closet has said offline in the app for about a week now. |
D028 | false_deflection | resident_abandoned | thermostat totally unresponsive, blank screen, been like this since yesterday |
D029 | false_deflection | answered_wrong_question | How do I reset my door code? The keypad is dead and does not light up at all when I press it but I figure a new code might help. |
D030 | false_deflection | resident_abandoned | I never got any alert on my phone when my water heater pan filled up last night. The sensor is sitting right there. Do these notif... |
D032 | false_deflection | policy_refusal_loop | I need to change the door code for my unit because a former roommate still knows it. Can you help me do that? |