Flaky Tests vs Real Bugs: How AI Tells the Difference
A test that fails 3 out of 5 runs might be a flaky test, a broken locator, a timing issue — or a real, intermittent product bug. Here's how VigilQA's FlakyDetector agent categorises each one.
Flaky tests are the silent killers of QA confidence. When a test fails intermittently, the natural first response is "it's probably flaky" — and the test gets muted, skipped, or ignored. Sometimes that's right. Sometimes a real product bug is being written off as infrastructure noise.
The four failure categories
VigilQA's ResultAnalyzerAgent classifies every test failure into one of four categories after each run:
- Product Bug — the application behaved incorrectly. Consistent failure, traceable to a specific assertion, matches known functionality.
- Flaky — Script — the test script has a timing, selector, or sequencing issue. Fails intermittently, often recovers on retry.
- Flaky — Infrastructure — the environment is unreliable. Network timeouts, slow CI runners, external service unavailability.
- Broken Locator — the selector no longer resolves. Consistent failure, no assertion reached, element not found.
How the FlakyDetector agent works
The FlakyDetector runs after the result classification pass and looks specifically at tests that have appeared in multiple recent runs. Its analysis is based on three signals:
- Failure pattern — does this test fail consistently (product bug) or intermittently (flaky)?
- Failure message similarity — are the error messages across failures identical (deterministic) or varied (non-deterministic)?
- Failure timing — do failures correlate with specific conditions (CI load, time of day, run order)?
A test that fails with the same AssertionError on every run is classified differently from one that fails with a TimeoutError on 2 of 7 runs.
Why the distinction matters
Misclassifying a real product bug as flaky means it ships. Misclassifying a flaky script as a product bug means an engineer wastes time investigating something that isn't broken. Both have real costs.
The FlakyDetector output gives you a prioritised list: fix these locators, investigate these product bugs, ignore these infrastructure blips until they reach a threshold. That's a genuinely useful signal — not just a raw list of failing tests.
The escalation threshold
VigilQA lets you configure a flakiness threshold — if a test fails more than N% of runs over a rolling window, it gets promoted from "Flaky — Script" to a full bug card in the Bug Tracker module, with the run history and error log attached. That prevents genuine intermittent bugs from staying invisible in the "probably flaky" pile indefinitely.