Reading a Test Failure: Five Artifacts That Accelerate Debugging

The classic debugging loop for a CI test failure looks like this: notice the failure, re-run the suite locally, try to reproduce it, add some prints, run again. Depending on the test, that loop can eat an hour before you've even looked at the actual bug.

VigilQA captures five artifacts during each test run and makes them available per test in the run results — no re-run needed. Here's what each one tells you and how to use it.

1. Traceback

The full Python exception and assertion error text, exactly as pytest captured it. This is the starting point for every failure analysis.

What to look for: assertion errors tell you what the test expected vs what it got. Exception tracebacks tell you where execution stopped. If the traceback points into a framework file rather than your test code, the failure is likely an infrastructure issue (timeout, connection refused, environment problem) rather than a product bug — which is useful context before you start debugging the app.

2. Screenshot (Web UI tests)

A screenshot taken at the exact moment of failure, displayed inline as a thumbnail. Click to see it full size.

Screenshots answer the question that tracebacks can't: what did the page actually look like? An assertion failure on an error message element might be because the message wasn't shown — or because it was shown but in the wrong place, or because the page never finished loading. The screenshot disambiguates all three in one glance.

A common pattern: the traceback says AssertionError: expected element to be visible. The screenshot shows a loading spinner still spinning. That's an environment issue (slow staging server), not a product bug — and without the screenshot you'd spend time hunting for a non-existent code defect.

3. Browser console log

JavaScript console output captured during the test session — errors, warnings, and any application-level logging your frontend emits.

This is the most underused artifact. Frontend applications increasingly log meaningful state changes to the console: authentication events, API response summaries, feature flag states. When a web UI test fails, a JavaScript error in the console is often the root cause even when the Python traceback points somewhere else.

It's also useful for flaky tests: if a test passes 4 out of 5 runs, comparing the console logs across runs often reveals a race condition — a resource load that sometimes completes in time and sometimes doesn't.

4. API request/response log

Every HTTP call made during the test, captured as a curl-equivalent command with full headers, request body, response status, response body, and elapsed milliseconds.

This is the artifact that most often resolves ambiguity about whether a failure is a frontend issue or a backend issue. If the UI shows the wrong data, the API log tells you whether the API returned the wrong data (backend bug) or the API returned correct data that the frontend rendered incorrectly (frontend bug). That distinction saves significant time routing the bug to the right team.

The curl format is also directly reproducible: copy the command, paste it in your terminal, and you have an isolated reproduction of the API call — no test framework, no auth setup, just the request.

5. Playwright trace (Web UI tests)

A downloadable .zip trace file for replay in the Playwright Trace Viewer. The trace records every browser action, network request, DOM snapshot, and console event in sequence.

The trace is most useful when the other four artifacts don't fully explain the failure. It lets you step through the test action by action, see the DOM state at each point, and identify exactly which step produced the unexpected result. Think of it as a video recording of the test with full network and DOM inspection capability.

Download the trace from the test detail panel, open the Playwright Trace Viewer in your browser, and load the file. No installation required beyond a Chromium-based browser.

A triage approach

Not every failure needs all five artifacts. A quick triage order that works for most failures:

1. Read the traceback — is this an assertion failure or an exception? Does it point to test code or framework code?
2. If it's a Web UI test, look at the screenshot — does the page match what the test expected to see?
3. Check the API log — did the backend return what the frontend expected?
4. If the failure is intermittent, compare console logs across passing and failing runs for JS errors or timing signals.
5. If you still can't explain the failure, download the Playwright trace for full step-by-step replay.

In practice, most failures are resolved at step 1 or 2. The later artifacts are there for the harder cases — and having them available without re-running the suite is the difference between a 10-minute bug investigation and an hour-long one.