chatbot-auditor¶

Detect the 7 ways AI chatbots silently fail customers.

Your chatbot dashboard says "95% of conversations resolved." The real number is usually 60–70%. The gap is customers who gave up, got stuck in loops, asked for a human and were refused, or silently walked away. Your chatbot records all of them as "resolved."

chatbot-auditor analyzes conversation data and tells you the truth.

Why it exists¶

75% of consumers are frustrated by AI customer support
56% of unhappy customers leave without complaining
88% will not return after a negative chatbot interaction
Chatbot platforms (Intercom, Zendesk, Drift) grade their own homework — their dashboards are designed to make the chatbot look good
Air Canada was sued because its chatbot promised a non-existent refund policy
DPD's chatbot swore at a customer and wrote a poem about how terrible the company was
Klarna reversed course after firing 700 agents when customer satisfaction tanked

Companies don't need a better chatbot. They need to know when their current chatbot is failing. That's what this library does.

30-second quickstart¶

pip install chatbot-auditor

from chatbot_auditor import audit, Conversation, Message, Role

conv = Conversation(
    id="demo",
    messages=[
        Message(role=Role.USER, content="I need a refund for my order"),
        Message(role=Role.BOT, content="Please check our FAQ at example.com/faq."),
        Message(role=Role.USER, content="I already did. Can someone process it?"),
        Message(role=Role.BOT, content="Please check our FAQ at example.com/faq."),
        Message(role=Role.USER, content="this is useless"),
        Message(role=Role.BOT, content="Please check our FAQ at example.com/faq."),
    ],
)

for d in audit([conv]):
    print(f"[{d.severity.value.upper()}] {d.detector}: {d.explanation}")

Output:

[MEDIUM] death_loop: Bot gave 3 consecutive similar responses ...
[HIGH]   silent_churn: Conversation of 6 messages ended without ...

The 7 failure modes¶

Mode	What it catches
Death Loop	Bot gives the same answer 3+ times
Silent Churn	Customer left without saying anything
Escalation Burial	Bot refused to transfer to a human
Sentiment Collapse	Customer got frustrated, bot didn't notice
Confident Lies	Bot promised something outside policy
Brand Damage	Bot said something embarrassing
Confident Misinformation	Bot stated wrong facts

What next¶

Project¶

Licensed under Apache 2.0. Original author: BK.

Contributions welcome — see the repository and contributing guide.