A duplicate detection threshold is the confidence score at which an AI model decides two bug reports describe the same issue, triggering a suggestion, a flag for review, or an automatic close. Most teams should start with a conservative auto-suggest setup that routes matches to a human reviewer rather than closing tickets outright, then track Recall@10, precision, and false positive rate before tightening anything.
TL;DR:
- Using a classifier cutoff around 0.7 to 0.85 generally strikes a good balance between recall and precision, but must be validated with recent, representative data.
- Thresholds should be calibrated on a labeled holdout set containing execution context and revalidated regularly to account for dataset drift and evolving reporting habits.
- Setting the detection threshold too low results in reviewer overload with false positives, while setting it too high risks missing true duplicates and increasing redundant tickets.
- Combining retrieval scores with execution context, such as screen replays and stack traces, improves duplicate detection accuracy from around 43% to over 67%.
- Repetitive logging, human oversight, and drift alarms are critical for maintaining threshold effectiveness in production environments.
Table of Contents
- Where thresholds fit in the two-stage retrieval and classification pipeline
- Why threshold choices carry real business consequences
- How to choose, validate, and iterate your thresholds
- Metrics that tell you whether a threshold is actually working
- Operational checklist for running thresholds safely in production
- What Wezard’s data shows about retrieval limits and context
- Keeping thresholds accurate as your data changes
- A practical roadmap for teams just starting out
- How Wezard helps you get ahead of duplicate tickets
- Research and documentation behind this guide
- Sources
- FAQ
Where thresholds fit in the two-stage retrieval and classification pipeline
Most AI duplicate detection systems run in two stages. First, a retrieval step uses semantic search to pull a shortlist of candidate tickets that resemble the new report. Second, a classifier scores each candidate pair and outputs a probability that they are true duplicates. A combined retrieval and classification pipeline that pairs transformer-based embeddings with a classifier like SVM or XGBoost tends to balance recall, precision, and compute cost better than either stage alone.
Thresholds show up at several points in that pipeline:
- Retrieval cutoff: the minimum similarity score needed for a ticket to even enter the candidate set.
- Candidate count (K): how many top matches the retrieval stage passes to the classifier.
- Classifier probability cutoff: the score that separates a non-match from a likely duplicate.
- Credibility threshold: a separate bar for how much weight to give text-only similarity versus execution evidence like stack traces or screen replay.
Each score then maps to an action: low scores get ignored, mid-range scores get suggested to a reviewer, and only very high scores should ever trigger an automatic close.
Why threshold choices carry real business consequences
Set the bar too low and reviewers drown in false positives, spending triage time chasing matches that turn out to be unrelated bugs. Set it too high and real duplicates slip through, quietly inflating the backlog with redundant tickets that developers end up fixing twice. Neither failure is free, and the right balance depends on how much reviewer time your team can spend per week.
Calibration also drifts for reasons that have nothing to do with the model itself:
- Age bias: thresholds tuned on old tickets stop reflecting how current reports are written.
- ITS bias: behavior learned from one issue-tracking system does not always transfer to another.
- Dataset drift: new product areas or UI changes shift the vocabulary and structure of incoming reports.
A benchmarking study on duplicate bug report detection found that these evaluation biases materially affect reported performance, which is why calibration needs to happen on recent, representative data rather than once at setup.
Reported accuracy for duplicate detection systems typically ranges from the high 70%s to the low 90%s depending on project complexity and semantic model choices, according to research on automated duplicate bug report detection. That range is a reminder that no threshold setting eliminates error. It only shifts where the error lands.
How to choose, validate, and iterate your thresholds
Picking a threshold is an experiment, not a guess. Here is a sequence that works for most enterprise UAT teams.
- Build a labeled holdout set from recent tickets, ideally within the last one to three years, and include execution or contextual traces (screen replay, stack traces) where available rather than relying on text alone.
- Define an experiment matrix that varies retrieval K (try 50, 75, 100) and classifier cutoffs (try 0.5, 0.7, 0.85), since research on duplicate detection pipelines suggests K values around 50 to 100 show diminishing returns beyond that range.
- Measure Recall@K and precision at each cutoff so you can see the trade-off directly instead of guessing at it.
- Start conservative in production: use auto-suggest for medium-confidence matches and reserve auto-close for only the highest-confidence band, since simple threshold-based evaluation shows stricter cutoffs generally improve precision at the cost of recall.
- Roll out gradually with a canary group or A/B split before applying a new threshold organization-wide.
- Log every decision and reviewer outcome so confirmed and rejected matches become training signal for the next recalibration round.
Pro Tip: Run your first threshold experiment on the ugliest, most ambiguous tickets in your backlog. If a cutoff holds up there, it will hold up everywhere else.
Metrics that tell you whether a threshold is actually working
A threshold change is only worth shipping if it moves the numbers that matter. The core metrics are Recall@k (the share of true duplicates found within the top k candidates), precision at cutoff (the share of flagged matches that are real duplicates), F1 (the balance between the two), and false positive rate (how often the system wrongly flags unrelated tickets).
- Recall@k: higher is better for catching duplicates, but it says nothing about reviewer workload on its own.
- Precision at cutoff: the number that protects reviewer time and trust in the tool.
- False positive rate: tie this directly to a reviewer-hours budget, for example a cap of a set number of hours per week spent dismissing bad matches.
A combined retrieval plus classification approach gives teams a practical way to balance these metrics against compute cost, according to the MDPI study on AI-based duplicate detection. Acceptance criteria should be set before the experiment runs, not after: define a minimum recall and a maximum false positive rate, then only promote a threshold change that clears both bars on the holdout set.
Operational checklist for running thresholds safely in production
Thresholds are not a one-time setting. They need logging, human oversight, and monitoring to stay accurate as your product and reporting habits change.
- Log everything: similarity scores, classifier probabilities, final decisions, and reviewer outcomes, including an audit trail for any ticket the system suppressed.
- Keep humans in the loop: use a triage UI with clear actions such as suggest, require confirmation, or auto-close with override, rather than silent automation.
- Watch for drift: schedule regular revalidation against fresh holdout data and set alarms for drops in recall or precision.
- Integrate carefully: favor non-blocking suggestions over hard blocks, build in webhook retry logic, and keep a rollback plan for any integration with Jira or Azure DevOps.
- Protect sensitive content: redact personal or confidential information from screen replay and execution traces before they feed the model.
Pro Tip: Treat every reviewer override as a labeled data point. It is the cheapest, most relevant training signal your team will ever get.
What Wezard’s data shows about retrieval limits and context
Retrieval alone can look strong on paper and still miss real duplicates in practice. Wezard’s analysis of why 94.44% retrieval still fails duplicate bug detection found that text-only similarity scores often mask the gap between surface wording and the actual underlying issue, which is exactly where execution context closes the distance.
- Text-only comparisons frequently miss duplicates that use different wording for the same root cause.
- Adding execution information such as stack traces or screen replay shifts the classifier’s confidence distribution, which research on combining natural language and execution data found raised detection effectiveness to 67% to 93%, compared with 43% to 72% for text alone.
- Screen replay and real-time transcription, as described in Wezard’s guide to preventing duplicate Jira tickets at capture, add that execution signal before a ticket ever reaches the backlog, which supports a higher, safer threshold without sacrificing recall.
Keeping thresholds accurate as your data changes
A threshold that works well this quarter will not necessarily hold up next quarter. Product changes, new feature areas, and shifts in how testers describe bugs all move the distribution of incoming reports, and a static cutoff eventually starts missing duplicates or flooding reviewers with false positives.
The fix is a revalidation cadence, not a one-time tuning exercise. Re-run your holdout evaluation on a rolling window of recent tickets, ideally the same one to three year window recommended for avoiding age bias in duplicate detection benchmarks, and compare current Recall@k and precision against your original acceptance criteria. If either metric drifts past an agreed tolerance, that is the signal to retrain or recalibrate rather than wait for reviewer complaints to pile up.
Two practical habits make this manageable. First, keep the labeled holdout set alive by continuously adding confirmed and rejected matches from production, since those reviewer decisions are the most relevant training signal available. Second, set a drift alarm tied to a measurable drop in RR@k or a rise in false positive rate, rather than relying on a calendar reminder alone. Teams that treat threshold calibration as a recurring checkpoint, tied to actual metric movement, catch problems before they show up as backlog noise. Teams that set a threshold once and move on tend to find out it broke only after reviewers stop trusting the suggestions altogether.
A practical roadmap for teams just starting out
Start conservative: auto-suggest, not auto-close, backed by a labeled holdout set that includes execution context wherever you can get it. Run the A/B experiments in small steps, and let logging and reviewer feedback, not intuition, decide when to tighten the cutoff. The decisive signal is always what reviewers actually confirm or reject, not what the model reports in isolation.
— Marketing
How Wezard helps you get ahead of duplicate tickets
Most duplicate detection tools work on text that has already lost the context of what actually happened on screen. Wezard records the tester’s screen and transcribes the issue in real time, so the duplicate detection model has execution context to work with before a ticket ever reaches your backlog.
- Screen recording and real-time transcription capture the full context of an issue, not just a written summary.
- Duplicate detection can run before tickets sync to issue tracking systems such as Jira, Azure DevOps, or ServiceNow.
- Integrations with issue tracking backlogs can allow flagged duplicates to be seen within existing workflows.
If you want to see how these thresholds perform against your own tickets, the pricing page outlines the Starter, Team, Business, and Enterprise plans so you can start validating on real data.
Research and documentation behind this guide
- ICSE paper on execution-based duplicate detection: data on text versus execution-context accuracy.
- MDPI study on retrieval plus classification: pipeline architecture and K-value guidance.
- Duplicate bug report benchmarking study: bias and recalibration guidance.
- Atlassian automation docs: practical rules for issue-tracker automation.
Sources
- Automated Duplicate Bug Report Detection in (preprint)
- Detecting Duplicates in Bug Tracking Systems with Artificial Intelligence: A Combined Retrieval and Classification Approach
- An Approach to Detecting Duplicate Bug Reports using Natural Language and Execution Information (ICSE)
FAQ
What is a duplicate detection threshold in bug tracking?
It is the confidence score an AI model uses to decide whether two bug reports describe the same issue and what action to take next, such as suggesting a match or flagging it for review. Most teams start conservative, routing medium-confidence matches to a human reviewer rather than auto-closing them.
How do I choose the right threshold for my team?
Build a labeled holdout set from recent tickets, test a range of classifier cutoffs such as 0.5, 0.7, and 0.85, and measure Recall@k and precision at each one, as described in research on duplicate detection pipelines. Pick the cutoff that meets your minimum recall and maximum false positive rate before rolling it out widely.
What is the difference between recall and precision in this context?
Recall measures how many true duplicates your system actually finds, while precision measures how many of its flagged matches are correct. A good threshold balances both against how much reviewer time your team can spend checking flagged tickets.
Does Wezard detect duplicates automatically?
Wezard uses AI-powered duplicate detection combined with screen recording and real-time transcription to catch likely duplicates before they reach your Jira or Azure DevOps backlog. The execution context from screen replay helps the model work with more than text alone when scoring a match.
How often should thresholds be recalibrated?
Thresholds should be revalidated on a recurring basis, not left static, since product changes and shifts in reporting habits can cause dataset drift and bias that degrade accuracy over time. A rolling one-to-three-year holdout window with drift alarms on recall and precision is a practical starting cadence.
