Skip to main content

AI Duplicate Detection (deep dive)

User-facing walkthrough: Maintenance → Duplicate Detection.

Why AI over keywords​

"Cold showers" vs "no hot water" share zero keywords but describe the same plumbing issue. Likewise "fridge is making noise" vs "kitchen appliance humming loud again". Jaccard-overlap keyword matching misses both. Claude reads the intent.

Prompt structure​

System: "You are comparing a new maintenance request against candidates from the same property in the last 14 days. For each candidate, return whether it likely describes the same issue. Return JSON only."

User:

SUBJECT (new request):
"The shower ran cold again last night"

CANDIDATES:
1. [MR-2189] "No hot water when we showered Saturday" — 8 days ago
2. [MR-2156] "Dishwasher rattling" — 21 days ago
3. [MR-2145] "Front door sticking" — 25 days ago

For each: { "requestId": "...", "isDuplicate": bool, "confidence": 0.0-1.0, "reasoning": "..." }

Claude returns structured output. We filter to isDuplicate: true with confidence ≥ 0.6.

Heuristic fallback​

HeuristicDuplicateDetector — Jaccard overlap on stopword-stripped tokens. Anything ≥ 0.35 flagged.

private static final Set<String> STOP = Set.of(
    "the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for",
    "of", "with", "from", "is", "are", "was", "were", "been", "being",
    "have", "has", "had", "do", "does", "did", "it", "its", "i", "we",
    "my", "our", "they", "them", "this", "that", "these", "those",
    "not", "no", "again", "still", "just", "also", "some"
);

Tokens shorter than 3 chars are ignored. Works reasonably for same-wording repeats ("dishwasher rattling" → "dishwasher still rattling"), fails on paraphrase ("cold shower" → "no hot water"). Which is exactly why we run Claude first.

Window + filters​

  • Same property only. Property ID scopes the candidate query.
  • Last 14 days only. Older matches are unlikely to be duplicates; they're usually the same problem returning, which is a different workflow (recurring issue, not duplicate).
  • Excludes COMPLETED + CANCELLED by default. We can match against closed WOs on request — an extension for recurring-issue detection.

Confidence calibration​

Claude's raw confidence correlates well with our manual labelling in evals but trends slightly optimistic. We threshold at 0.6 for UI display, 0.8 for auto-highlighting the match as "likely".

API​

POST /api/v1/maintenance-requests/{id}/detect-duplicate. Non-mutating. See Duplicate Detection (user-facing).

See also​