Why We Kept Humans in the Loop
Fully autonomous AI testing sounds appealing. Here's why we deliberately chose not to build it — and why that makes SentinelFlux more trustworthy, not less.
When we were designing SentinelFlux's approval queue, the most common pushback from early users was: "Why can't it just commit the fix automatically?" It's a fair question. If the AI identifies a broken locator and proposes a correct replacement, what's the value of asking a human to click approve?
The answer is: the AI doesn't always propose the correct replacement.
The confidence problem
Language models are confident even when they're wrong. An AI that identifies "the submit button on the login form" will propose a locator — but on a page with two submit buttons, it might pick the wrong one. Without human review, that bad locator gets committed, the test passes, and you've silently started testing the wrong element.
Worse: because the test is now "passing", it stops generating signal. You've traded a visible failure (broken locator) for an invisible one (test that passes while testing nothing useful). That's a worse outcome than leaving the test broken.
What human review actually costs
The common objection is that approval queues create friction. In practice, reviewing a locator heal takes about 10 seconds: you see the old selector, the new proposed selector, a screenshot of the element, and the page context. You click Approve or Reject. That's the entire review.
Compare that to the alternative: a fully autonomous system that occasionally silently breaks your test suite in ways that take hours to diagnose.
The three categories of approval items
The Approvals queue in SentinelFlux covers three types of AI suggestions:
- Locator heals — new selectors proposed by the LocatorHealerAgent after all stored locators failed
- Script fixes — test script patches proposed after the ScriptReview agent identifies issues in generated code
- Baseline updates — visual regression baseline image replacements when a legitimate UI change is flagged as a diff
All three have the same property: they're cases where the AI is making a judgment call that could be wrong in ways that aren't immediately visible. Human review is the backstop.
The autonomy slider
We're exploring an optional high-confidence auto-approve mode for teams that want it — where approvals with confidence scores above a threshold are committed without review. But it will always be opt-in, always logged, and always easily reversible. The default will remain human-in-the-loop.
Trustworthy AI tooling isn't about removing humans. It's about putting humans in the right places — the decisions that require judgment — and automating everything else.