ResultAnalyzer
Classifies every test failure into one of four categories so your team always knows whether a failure is a real product bug, a test issue, a flaky test, or an infrastructure problem.
Trigger
ResultAnalyzer runs automatically after every test run completes, regardless of run outcome. It processes all failed tests in the run in parallel.
Classification categories
| Category | Meaning | Next action |
|---|---|---|
| Product Bug | The application's actual behaviour does not match the expected outcome defined in the KB. | Bug Tracker entry created automatically (if Bug Tracker module enabled). Alert sent to team. |
| Script Error | The test script itself has an issue — stale selector, incorrect assertion, missing fixture setup. | Script flagged for regeneration. ScriptGen re-queued for the affected scenario. |
| Flaky | Test passed in at least one of the last 5 runs with no clear application change in between. | Forwarded to FlakyDetector for stability analysis. |
| Environment Issue | Failure caused by infrastructure: timeout, missing dependency, environment misconfiguration. | Alert with environment context. Not counted as a product regression. |
How classification works
For each failed test, ResultAnalyzer examines:
- The assertion failure message and type (e.g. element not found vs. assertion value mismatch)
- The test's run history (last 5 runs)
- Whether a CI/CD context tag was passed with the run (commit SHA, deployment ID)
- Whether other tests in the same domain or run also failed
- Screenshot at point of failure (for Web UI modules)
The classification confidence score is shown in the run results view. Classifications below 0.7 confidence are shown with a "needs review" indicator.
Product Bug entries
For failures classified as Product Bug, ResultAnalyzer produces a structured entry containing:
- Scenario ID and title
- Steps to reproduce (from KB)
- Expected vs. actual outcome
- Screenshot or API response at point of failure
- Run reference (link back to the specific run)
- Affected environment and version/commit
These entries are sent to the Bug Tracker module and, if an external tracker is configured, mirrored to Jira, Linear, or GitHub Issues.
ResultAnalyzer classifications can be manually overridden from the run results view. If a Product Bug classification was incorrect (e.g. it was actually a script error), click the classification label and select the correct category. This feedback improves future classification accuracy.
Configuration
ResultAnalyzer configuration is in Project Settings → Agents → ResultAnalyzer:
- Flaky threshold — minimum number of times a test must have passed in recent history to be classified as Flaky (default: 1 pass in last 5 runs)
- Auto-create bugs — whether to automatically create Bug Tracker entries for Product Bug classifications (default: enabled)
- Notify on classification — whether to send a notification per failure classification or only on summary (default: summary after run)