Skip to content

Detectors

A Detector is a single, focused analyzer: given a Conversation, it returns zero or more Detection objects describing specific failures. Detectors are pure functions over conversations — they don't call networks, mutate state, or depend on each other.

The contract

Every detector inherits from chatbot_auditor.Detector and declares four class attributes:

class MyDetector(Detector):
    name: ClassVar[str] = "my_detector"              # unique identifier
    description: ClassVar[str] = "What it catches..." # shown in reports
    failure_mode: ClassVar[FailureMode] = ...         # which of the 7 modes
    requires_llm: ClassVar[bool] = False              # whether network calls are required

    def detect(self, conversation: Conversation) -> list[Detection]:
        ...

The detect() method returns zero or more detections. Returning an empty list is normal — most conversations aren't failures.

The production-ready detectors

chatbot-auditor ships with seven detectors, one per failure mode. The default_registry() enables five of them — the two that require knowledge bases are opt-in:

Class Requires config In default registry?
DeathLoopDetector no yes
SilentChurnDetector no yes
EscalationBurialDetector no yes
SentimentCollapseDetector no yes
BrandDamageDetector optional competitor list yes
ConfidentLiesDetector optional PolicyBase no
ConfidentMisinformationDetector optional FactBase no

The registry

A DetectorRegistry holds an ordered set of detectors and runs them all on each conversation. The registry is the composition unit — swap detectors in or out without changing audit code:

from chatbot_auditor import (
    DetectorRegistry,
    DeathLoopDetector,
    SilentChurnDetector,
    audit,
)

# Custom subset — maybe you only care about loops and churn
registry = DetectorRegistry([
    DeathLoopDetector(min_repeats=2),  # more aggressive threshold
    SilentChurnDetector(),
])

detections = audit(conversations, detectors=registry)

Configuration vs. injection

Each detector accepts configuration for thresholds and behavior, and also accepts pluggable backends for the non-trivial parts:

Detector Configuration Pluggable backend
DeathLoopDetector similarity_threshold, min_repeats similarity_fn
SilentChurnDetector min_user_messages, min_bot_messages —
EscalationBurialDetector — custom keyword lists
SentimentCollapseDetector min_drop, end_threshold SentimentScorer
BrandDamageDetector competitor_names ContentSafetyChecker
ConfidentLiesDetector — PolicyBase
ConfidentMisinformationDetector — FactBase

The pluggable backends are where production users upgrade from stdlib defaults to richer backends — transformer sentiment, LLM-based safety, RAG-backed policy verification. The detector itself never changes.

Writing your own

See the tutorial: Write a custom detector.

In short:

  1. Subclass Detector and declare the four required class attributes.
  2. Implement detect(conversation) -> list[Detection].
  3. Return Detection objects with conversation_id, detector, failure_mode, severity, confidence, explanation, and evidence.
  4. Register it in a DetectorRegistry or pass it directly to audit().

Performance notes

  • The default five detectors run in pure Python — no network, no model downloads — and handle tens of thousands of conversations per second on a single core.
  • Detectors are embarrassingly parallel at the conversation level. For large audits, run each conversation through the registry in its own worker.
  • Detectors don't share state, so registry instances are safe to reuse across threads.