Model Evals
Score a model or prompt against a labeled dataset — accuracy, faithfulness, safety, and format — and catch regressions when you change the model or the prompt.
What this module does
The Model Evals module runs dataset-driven evaluations against an LLM endpoint or prompt template. You supply a dataset of inputs (and, where available, reference answers); VigilQA runs each example through the model, scores the output against your chosen metrics, and reports aggregate results with a full per-example breakdown.
Where the AI Agent Testing module tests a live agent application end to end, Model Evals answers a narrower question: how good is this model/prompt on this task, and did the latest change make it better or worse?
Metrics
| Metric | Measures | Scoring |
|---|---|---|
| Accuracy / Exact Match | Output matches the reference answer | Exact, regex, or normalized string |
| Semantic Similarity | Output is equivalent in meaning to the reference | LLM-judge / embedding score |
| Faithfulness | Output is grounded in provided context (no hallucination) | LLM-judge |
| Relevance | Output actually answers the question | LLM-judge |
| Safety / Toxicity | Output avoids unsafe, biased, or toxic content | Classifier / LLM-judge |
| Format Adherence | Output conforms to an expected JSON/schema shape | Deterministic schema check |
Dataset format
Evaluation datasets live alongside your Knowledge Base as YAML or JSONL. Each row is one example; reference fields are optional depending on the metric.
eval: support-classifier
model: ${LLM_MODEL}
metrics: [accuracy, faithfulness, safety]
threshold: 0.85
dataset:
- id: EV-001
input: "My payment failed but I was still charged."
expected: "billing_issue"
- id: EV-002
input: "How do I reset my password?"
expected: "account_access"
- id: EV-003
input: "Summarise this ticket."
context: "Customer reports the export button does nothing on Safari."
judge_rubric: "Summary mentions the export button and Safari; no invented details."
Prompt & model regression
Model Evals is designed for comparison. Run the same dataset against two candidates — model A vs model B, or prompt v1 vs prompt v2 — and VigilQA produces a side-by-side scorecard with per-metric deltas and the specific examples that changed verdict.
When an eval is part of your CI pipeline, RegressionGuard alerts on any metric that drops below its baseline, so a prompt tweak that quietly degrades faithfulness is caught before it ships.
Reproducibility. Set temperature: 0 (or a fixed seed where the provider supports it) for the model under test, and pin the judge model version. Record both in the eval file so a score is always reproducible.
Reporting
Results appear in the Model Evals tab of the run: aggregate score per metric, pass/fail against threshold, and a searchable table of every example with its input, output, reference, and judge rationale. Scorecards export to PDF and JSON, and per-example results can be fed into Release Readiness as a gate signal.